How to parse custom JSON log format with GoAccess

[mahesh-kore]

Can someone please help to parse below custom JSON log

Nginx Logformat

log_format logjson '{ "@timestamp": "$time_iso8601", '
'"msec": $msec, '
'"remote_addr": "$remote_addr", '
'"x_forwarded_for": "$proxy_add_x_forwarded_for", '
'"remote_port": "$remote_port", '
'"pipelined": "$pipe", '
'"body_bytes_sent": "$body_bytes_sent", '
'"bytes_sent": $bytes_sent, '
'"request_time": $request_time, '
'"upstream_response_time": "$upstream_response_time", '
'"upstream_response_length": "$upstream_response_length", '
'"upstream_status": "$upstream_status", '
'"kore_route": "$upstream_http_x_route", '
'"koreserver": "$upstream_http_server", '
'"host": "$host", '
'"hostname": "$hostname", '
'"server_name": "$server_name", '
'"request_completion": "$request_completion", '
'"status": $status, '
'"connection_requests": $connection_requests, '
'"request_uri": "$request_uri", '
'"request_method": "$request_method", '
'"request_content_type": "$content_type", '
'"request_content_length": "$content_length", '
'"request_total_length": $request_length, '
'"args": "$args",'
'"is_args": "$is_args", '
'"x-traceid":"$upstream_http_x_traceid", '
'"http_user_agent": "$http_user_agent" }';

Sample Loglines:

{ "@timestamp": "2021-02-23T08:27:47+00:00", "msec": 1614068867.506, "remote_addr": "172.1.0.32", "x_forwarded_for": "172.1.0.32", "remote_port": "2431", "pipelined": ".", "body_bytes_sent": "46", "bytes_sent": 710, "request_time": 0.010, "upstream_response_time": "0.012", "upstream_response_length": "46", "upstream_status": "401", "kore_route": "-", "koreserver": "KoreServer/", "host": "wb-bots.korebots.com", "hostname": "app-57c5bc4bb7-4m2r2", "server_name": "_", "request_completion": "OK", "status": 401, "connection_requests": 2, "request_uri": "/api/1.1/rtm/start", "request_method": "POST", "request_content_type": "application/json", "request_content_length": "122", "request_total_length": 729, "args": "-","is_args": "", "x-traceid":"-", "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36" }

{ "@timestamp": "2021-02-23T08:27:47+00:00", "msec": 1614068867.926, "remote_addr": "172.1.0.32", "x_forwarded_for": "172.1.0.32", "remote_port": "2431", "pipelined": ".", "body_bytes_sent": "53", "bytes_sent": 698, "request_time": 0.008, "upstream_response_time": "0.008", "upstream_response_length": "53", "upstream_status": "401", "kore_route": "-", "koreserver": "KoreServer/", "host": "wb-bots.korebots.com", "hostname": "app-57c5bc4bb7-4m2r2", "server_name": "_", "request_completion": "OK", "status": 401, "connection_requests": 3, "request_uri": "/api/1.1/users/sts?rnd=wb729", "request_method": "POST", "request_content_type": "application/json;charset=UTF-8", "request_content_length": "2", "request_total_length": 861, "args": "rnd=wb729","is_args": "?", "x-traceid":"-", "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36" }

[allinurl]

Sorry for the late reply. This should work;

goaccess access.log --log-format='{"@timestamp": "%dT%t+%^", "remote_addr": "%h", "bytes_sent": "%b", "request_time": "%T", "host": "%v", "status": "%s", "request_uri": "%U", "request_method": "%m", "request_content_type": "%M", "http_user_agent": "%u"}' --date-format=%Y-%m-%d --time-format=%T

You just need to specify the fields that goaccess can parse and are available in your log.

{
    "@timestamp": "%dT%t+%^",
    "remote_addr": "%h",
    "bytes_sent": "%b",
    "request_time": "%T",
    "host": "%v",
    "status": "%s",
    "request_uri": "%U",
    "request_method": "%m",
    "request_content_type": "%M",
    "http_user_agent": "%u"
}

[mahesh-kore]

Thank you