search

allinurl / goaccess

GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.
https://goaccess.io
MIT License
18.38k stars 1.11k forks source link

Hundreds of Invalid Requests #2275

Closed f00d4tehg0dz closed 2 years ago

f00d4tehg0dz commented 2 years ago

Unsure as to why I'm getting so many failed requests over this data. I only want the request_url, timestamp, http_host, and remote_address. The dashboard and console show hundreds of invalid requests

This is my config file details

time-format %T
date-format %d/%b/%Y
log-format %^,%^,%^,"request_url:""%U","timestamp:""%d:%T",%^,"http_host:","%q", "remote_address""%h","http_referer:"

Slimmed-down example

{"remote_address":"55.555.55.555",
"http_accept_language":"",
"http_version":"HTTP/1.1",
"request_url":"/example/example/",
"request_type":"GET",
"upstream_country_code":"US",
"request_time":"0.001",
"scheme":"https",
"http_referer":"",
"http_user_agent":"Mozilla/5.0",
"http_host":"example.com",
"sent_cache_control":"max-age=300, must-revalidate",
"status":"200",
"content_type":"text/html; charset=UTF-8",
"timestamp":"10/Feb/2022:15:17:11 +0000",
"timestamp_iso8601":"2022-02-10T15:17:11+00:00",
"http_x_forwarded_for":""}

The script I'm running inside of Ubuntu 20.04 LTS

zcat 1_*.gz | /home/ubuntu/goaccess/./goaccess -p goaccess_nginx_log.conf -access.log

allinurl commented 2 years ago

goaccess supports JSON formats, so you should be able to use this:

goaccess access.log --log-format='{"remote_address":"%h","http_version":"%H","request_url":"%U","request_type":"%m","request_time":"%T","http_referer":"%R","http_user_agent":"%u","http_host":"%v","status":"%s","content_type":"%M","timestamp":"%d:%t %^"}' --date-format=%d/%b/%Y --time-format=%T