search

allinurl / goaccess

GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.
https://goaccess.io
MIT License
18.47k stars 1.11k forks source link

Custom JSON format for Caddy log structure #2699

Open FanelliMarco opened 3 months ago

FanelliMarco commented 3 months ago

hi allinurl I'm trying to use GoAccess to analyze my Caddy logs, but I'm having trouble creating a custom JSON format (using docker)

{"level":"info","ts":1624526415.449846,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"192.168.0.1","remote_port":"22","client_ip":"192.168.0.3","proto":"HTTP/1.1","method":"GET","host":"example.com","uri":"/","headers":{"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"X-Forwarded-Host":["example.com"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"X-Forwarded-For":["10.0.0.0"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0  Chrome/127.0.0.0 Safari/537.36"],"Accept-Language":["en-US,en;q=0.9"]}},"bytes_read":0,"user_id":"","duration":0.001574238,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}

this is my goaccess.conf file for now

addr 0.0.0.0
port 7890
daemonize false
real-time-html true
log-format json={"level":"%e","ts":%T,"logger":"%v","msg":"%r","remote_ip":"$.request.remote_ip","remote_port":"$.request.remote_port","client_ip":"$.request.client_ip","proto":"$.request.proto","method":"$.request.method","host":"$.request.host","uri":"$.request.uri","bytes_read":%b,"user_id":"%e","duration":%D,"size":%b,"status":%s}, ignore-null
time-format %s
date-format %s
debug-file /var/log/goaccess/debug.log
log-file /var/log/caddy/access.log
output /var/www/goaccess/index.html

i get this errors

FILE: /var/log/caddy/access.log
2024-07-31 00:28:18 ==1== Parsed 2 lines producing the following errors:
2024-07-31 00:28:18 ==1==
2024-07-31 00:28:18 ==1== Token 'h-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"X-Forwarded-Host":["example.com"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"X-Forwarded-For":["10.0.0.0"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0  Chrome/127.0.0.0 Safari/537.36"],"Accept-Language":["en-US,en;q=0.9"]' doesn't match specifier '%s'
2024-07-31 00:28:18 ==1==
2024-07-31 00:28:18 ==1== Format Errors - Verify your log/date/time format
2024-07-31 00:50:40  [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}
2024-07-31 00:50:40 ==1== GoAccess - version 1.9.3 - Jul 22 2024 13:14:37
2024-07-31 00:50:40 ==1== Config file: /etc/goaccess/goaccess.conf
2024-07-31 00:50:40 ==1== https://goaccess.io - <hello@goaccess.io>
2024-07-31 00:50:40 ==1== Released under the MIT License.
2024-07-31 00:50:40 ==1==
2024-07-31 00:50:40 ==1== FILE: /var/log/caddy/access.log
2024-07-31 00:50:40 ==1== Parsed 2 lines producing the following errors:
2024-07-31 00:50:40 ==1==
2024-07-31 00:50:40 ==1== A valid date is required.
2024-07-31 00:50:40 ==1== A valid date is required.
2024-07-31 00:50:40 ==1==
2024-07-31 00:50:40 ==1== Format Errors - Verify your log/date/time format
2024-07-31 00:59:12  [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}
2024-07-31 00:59:12 ==1== GoAccess - version 1.9.3 - Jul 22 2024 13:14:37
2024-07-31 00:59:12 ==1== Config file: /etc/goaccess/goaccess.conf
2024-07-31 00:59:12 ==1== https://goaccess.io - <hello@goaccess.io>
2024-07-31 00:59:12 ==1== Released under the MIT License.
2024-07-31 00:59:12 ==1==
2024-07-31 00:59:12 ==1== FILE: /bin/sh
2024-07-31 00:59:12 ==1== Parsed 10 lines producing the following errors:
2024-07-31 00:59:12 ==1==
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1==
2024-07-31 00:59:12 ==1== Format Errors - Verify your log/date/time format
allinurl commented 3 months ago

This should do it:

# goaccess access.log --log-format=CADDY --date-spec=min

2024-07-30-211748_562x313_scrot

FanelliMarco commented 3 months ago

I updated goaccess.conf as follows

addr 0.0.0.0
port 7890
daemonize false
real-time-html true
log-format CADDY
date-spec min
debug-file /var/log/goaccess/debug.log
log-file /var/log/caddy/access.log
output /var/www/goaccess/index.html

it gives me this error

2024-07-31 06:28:49  [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}
2024-07-31 06:28:49 ==1== GoAccess - version 1.9.3 - Jul 22 2024 13:14:37
2024-07-31 06:28:49 ==1== Config file: /etc/goaccess/goaccess.conf
2024-07-31 06:28:49 ==1== https://goaccess.io - <hello@goaccess.io>
2024-07-31 06:28:49 ==1== Released under the MIT License.
2024-07-31 06:28:49 ==1==
2024-07-31 06:28:49 ==1== FILE: /bin/sh
2024-07-31 06:28:49 ==1== Parsed 10 lines producing the following errors:
2024-07-31 06:28:49 ==1==
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1==
2024-07-31 06:28:49 ==1== Format Errors - Verify your log/date/time format
allinurl commented 3 months ago

It looks like the first 10 lines of your JSON log may not be valid. Could you please share the first 20 lines directly from your access.log?

FanelliMarco commented 3 months ago

these are the same logs that I provided earlier basically. I don't know if I'm doing anything in particular wrong. In goaccess.conf file, i have specified the log format as CADDY, which is not compatible with the JSON log format produced by Caddy (i think).

{"level":"info","ts":1722377868.638059,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"1234","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXXXXXX","uri":"/","headers":{"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"X-Forwarded-Host":["XXXXXXX"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["XXXXXXX"],"Accept-Language":["en-US,en;q=0.9"]}},"bytes_read":0,"user_id":"","duration":0.001574238,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
{"level":"info","ts":1722377884.6235218,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"1234","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXXXXXX","uri":"/","headers":{"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Fetch-Mode":["navigate"],"Upgrade-Insecure-Requests":["1"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Gpc":["1"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"X-Forwarded-Proto":["https"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Priority":["u=0, i"],"Sec-Fetch-User":["?1"],"X-Forwarded-Host":["XXXXXXX"],"User-Agent":["XXXXXXX"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Accept-Encoding":["gzip, deflate, br, zstd"]}},"bytes_read":0,"user_id":"","duration":0.000091731,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
{"level":"info","ts":1722399905.8303173,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"1234","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXXXXXX","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"User-Agent":["XXXXXXX"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Host":["XXXXXXX"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Site":["none"]}},"bytes_read":0,"user_id":"","duration":0.00088854,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
{"level":"info","ts":1722452074.5979362,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"XXXX","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"anonymized-host","uri":"/favicon.ico","headers":{"X-Forwarded-Host":["XXXXXXX"],"X-Forwarded-Proto":["https"],"User-Agent":["XXXXXXX"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["image"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Site":["same-origin"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Authorization":["REDACTED"],"Priority":["u=1, i"],"Referer":["XXXXXXX"],"Sec-Gpc":["1"],"If-Modified-Since":["Mon, 14 Feb 2022 05:51:54 GMT"],"If-None-Match":["W/\"47e-17ef6c99890\""],"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Ch-Ua-Platform":["\"Windows\""]}},"bytes_read":0,"user_id":"root","duration":0.000941437,"size":0,"status":304,"resp_headers":{"X-Xss-Protection":["1; mode=block"],"Etag":["W/\"47e-17ef6c99890\""],"Date":["Wed, 31 Jul 2024 18:54:34 GMT"],"Content-Security-Policy":["default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; font-src 'self'; frame-ancestors 'self'; base-uri 'self'; form-action 'self';"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=0"],"Last-Modified":["Mon, 14 Feb 2022 05:51:54 GMT"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains; preload"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["SAMEORIGIN"],"Vary":["Accept-Encoding"]}}
{"level":"info","ts":1722377868.638059,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"XXXX","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXX","uri":"/","headers":{"Priority":["u=0, i"],"Sec-Ch-Ua":[""Not)A;Brand";v="99", "Brave";v="127", "Chromium";v="127""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8"],"Sec-Fetch-Site":["none"],"X-Forwarded-Host":["XXX"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":[""Windows""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["anonymized-user-agent"],"Accept-Language":["en-US,en;q=0.9"]}},"bytes_read":0,"user_id":"","duration":0.001574238,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm="restricted""]}}
{"level":"info","ts":1722377884.6235218,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"XXXX","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXX","uri":"/","headers":{"Sec-Ch-Ua":[""Not)A;Brand";v="99", "Brave";v="127", "Chromium";v="127""],"Sec-Fetch-Mode":["navigate"],"Upgrade-Insecure-Requests":["1"],"Sec-Ch-Ua-Platform":[""Windows""],"Sec-Gpc":["1"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"X-Forwarded-Proto":["https"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Priority":["u=0, i"],"Sec-Fetch-User":["?1"],"X-Forwarded-Host":["XXX"],"User-Agent":["anonymized-user-agent"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8"],"Accept-Encoding":["gzip, deflate, br, zstd"]}},"bytes_read":0,"user_id":"","duration":0.000091731,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm="restricted""]}}
{"level":"info","ts":1722399905.8303173,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"XXXX","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXX","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8"],"Priority":["u=0, i"],"Sec-Ch-Ua":[""Not)A;Brand";v="99", "Brave";v="127", "Chromium";v="127""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":[""Windows""],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"User-Agent":["anonymized-user-agent"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Host":["XXX"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Site":["none"]}},"bytes_read":0,"user_id":"","duration":0.00088854,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm="restricted""]}}

maybe i need to put in the goaccess.conf file something like this

addr 0.0.0.0
port 7890
daemonize false
real-time-html true
log-format "%h %^[%d:%t %^] \"%r\" %s %b \"%u\" \"%H\" \"%R\""
date-spec %d:%t
debug-file /var/log/goaccess/debug.log
log-file /var/log/caddy/access.log
output /var/www/goaccess/index.html
allinurl commented 3 months ago

That same command using the CADDY log format works for me.

Please try using --no-global-config and try:

# goaccess access.log --log-format=CADDY --date-spec=min
FanelliMarco commented 3 months ago

i change goaccess.conf with this

addr 0.0.0.0
port 7890
daemonize false
real-time-html true
no-global-config true
log-format CADDY
date-spec min
debug-file /var/log/goaccess/debug.log
log-file /var/log/caddy/access.log
output /var/www/goaccess/index.html

and still give me the same error, at this point i really don't know

2024-08-04 18:18:06  [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}
2024-08-04 18:18:06 ==1== GoAccess - version 1.9.3 - Jul 22 2024 13:14:37
2024-08-04 18:18:06 ==1== Config file: /etc/goaccess/goaccess.conf
2024-08-04 18:18:06 ==1== https://goaccess.io - <hello@goaccess.io>
2024-08-04 18:18:06 ==1== Released under the MIT License.
2024-08-04 18:18:06 ==1==
2024-08-04 18:18:06 ==1== FILE: /bin/sh
2024-08-04 18:18:06 ==1== Parsed 10 lines producing the following errors:
2024-08-04 18:18:06 ==1==
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 Cleaning up resources...
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1==
2024-08-04 18:18:06 ==1== Format Errors - Verify your log/date/time format
2024-08-04 18:18:07  [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}