allinurl / goaccess

GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.
https://goaccess.io
MIT License
17.78k stars 1.09k forks source link

The "Unique Visitors Per Day" pane always showing a single dot #2686

Closed Zabrane closed 1 month ago

Zabrane commented 1 month ago

Hi @allinurl

I'm struggling with the first pane always showing a single dot (see below image).

I'm running GoAccess on Ubuntu 22.04 LTS:

# goaccess --version
GoAccess - 1.9.2.
For more details visit: https://goaccess.io/
Copyright (C) 2009-2024 by Gerardo Orellana

Build configure arguments:
  --enable-utf8
  --enable-geoip=mmdb

GoAccess log format is:

--log-format='%V %h %^[%d:%t %^] \"%r\" %s %b \"%R\" \"%u\"' --date-format="%d/%b/%Y" --time-format="%T"

Apache log format is:

LogFormat "%V %{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy

uniq_visitor

Here is a sample from Apache log:

www.foxxyme.com 83.99.151.68 - - [30/May/2024:22:32:00 +0200] "GET /bic-bic-cristal-re-new-recharges-stylo-bille-pointe-moyenne-1-0-mm-noir-etui-carton-3_SKU450864.html HTTP/1.1" 200 19813 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GeedoProductSearch; +http://www.geedo.com/product-search.html) Chrome/79.0.3945.88 Safari/537.36"
www.foxxyme.com 66.249.76.70 - - [30/May/2024:22:32:01 +0200] "GET /lipton-lipton-cannette-ice-tea-peche-33-cl_SKU496032.html HTTP/1.1" 200 21302 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.6422.141 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
www.foxxyme.com 69.171.249.6 - - [30/May/2024:22:32:01 +0200] "GET /papeterie/papier/papiers-speciaux/cartes-visite_C102_102010501.html?SearchParameter=%26%40QueryTerm%3D*%26Attr_NbFeuilles%3D15%26ContextCategoryUUID%3DQkIKBQEHg2gAAAFyljMJwFu1%26OnlineFlag%3D1%26%40GroupSize.ManufacturerName%3D0%26%40Rk.Attr_Produit_Vendable_O-N%3Dfalse%255E0%26%40Rk.InStock%3Dtrue%255E100 HTTP/1.1" 200 23708 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
www.foxxyme.com 193.31.125.248 - - [30/May/2024:22:32:01 +0200] "GET /uni-ball-uni-ball-feutre-technique-uni-pin-serie-1-pointe-biseautee-chisel-coloris-noir_SKU451157.html HTTP/1.1" 200 23490 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/22.0.1216.0 Safari/537.2"

What's wrong? Help appreciated. Thanks in advance.

Zabrane commented 1 month ago

@allinurl I also noticed two things in my Unique Visitors Per Day:

  1. I've less entries in the Panel Options menu than your demo: Screenshot 2024-05-30 at 22 47 52

Screenshot 2024-05-30 at 22 48 53

  1. Only the data is shown without the time (HH:MM:SS) uniq_visitor-2

Screenshot_2024-05-30_at_22_53_01

allinurl commented 1 month ago

The one dot in the graph is that you only have one day's worth of data. If you want to add more specificity, please pass --date-spec=min. e.g.,

# goaccess access.log --log-format=COMBINED -o report.html --date-spec=min
Zabrane commented 1 month ago

@allinurl wow ... it worked, thanks.

But how about the second remark. Why do I've less entries in the Panel Options menu?

Screenshot_2024-05-30_at_23_35_07

allinurl commented 1 month ago

There's no seconds specificity. I'm not sure if it's worth adding; it will change quite quickly. As for columns, are you logging the time served? What server are you running?

Zabrane commented 1 month ago

goaccess access.log --log-format=COMBINED -o report.html --date-spec=min

Why COMBINED and not VCOMBINED?

Zabrane commented 1 month ago

[...] As for columns, are you logging the time served? What server are you running?

Not sure if I fully understand the question.

Should I change Apache logging format to this?

LogFormat "%V %{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" **%T/%D**" proxy

Is this what you're referring to? If yes, what change should i make to the GoAccess log format?

allinurl commented 1 month ago

Please post a few lines from your access log so I can take a look

Zabrane commented 1 month ago

@allinurl Sure. I did post them on my first post above. Here you are:

www.foxxyme.com 83.99.151.68 - - [30/May/2024:22:32:00 +0200] "GET /bic-bic-cristal-re-new-recharges-stylo-bille-pointe-moyenne-1-0-mm-noir-etui-carton-3_SKU450864.html HTTP/1.1" 200 19813 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GeedoProductSearch; +http://www.geedo.com/product-search.html) Chrome/79.0.3945.88 Safari/537.36"
www.foxxyme.com 66.249.76.70 - - [30/May/2024:22:32:01 +0200] "GET /lipton-lipton-cannette-ice-tea-peche-33-cl_SKU496032.html HTTP/1.1" 200 21302 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.6422.141 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
www.foxxyme.com 69.171.249.6 - - [30/May/2024:22:32:01 +0200] "GET /papeterie/papier/papiers-speciaux/cartes-visite_C102_102010501.html?SearchParameter=%26%40QueryTerm%3D*%26Attr_NbFeuilles%3D15%26ContextCategoryUUID%3DQkIKBQEHg2gAAAFyljMJwFu1%26OnlineFlag%3D1%26%40GroupSize.ManufacturerName%3D0%26%40Rk.Attr_Produit_Vendable_O-N%3Dfalse%255E0%26%40Rk.InStock%3Dtrue%255E100 HTTP/1.1" 200 23708 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
www.foxxyme.com 193.31.125.248 - - [30/May/2024:22:32:01 +0200] "GET /uni-ball-uni-ball-feutre-technique-uni-pin-serie-1-pointe-biseautee-chisel-coloris-noir_SKU451157.html HTTP/1.1" 200 23490 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/22.0.1216.0 Safari/537.2"
allinurl commented 1 month ago

It looks like it doesn't align with the **%T/%D**. You might want to restart apache. Once you've done that, you can proceed with:

# goaccess access.log --log-format='%v %h %^[%d:%t %^] "%r" %s %b "%R" "%u" %^/%D' --date-format=%d/%b/%Y --time-format=%T
Zabrane commented 1 month ago

@allinurl one more question if you don't mind. Why I can't see these two panels with my current config even if i'm explicitely specifying them with --enable-panel :

        --enable-panel=MIME_TYPE \
        --enable-panel=VIRTUAL_HOSTS
allinurl commented 1 month ago

Assuming you've got the right format, it should work. Please post your config file and the full command you are using.

Zabrane commented 1 month ago

@allinurl I don't use a config file (--no-global-config). This is my full command line:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:    22.04
Codename:   jammy

$  goaccess --version
GoAccess - 1.9.2.
For more details visit: https://goaccess.io/
Copyright (C) 2009-2024 by Gerardo Orellana

Build configure arguments:
  --enable-utf8
  --enable-geoip=mmdb

$ goaccess -a -q \
        --no-global-config \
        --log-format='%v %h %^[%d:%t %^] "%r" %s %b "%R" "%u" %^/%D' \
        --date-format="%d/%b/%Y" \
        --time-format="%T" \
        --enable-panel=REFERRERS \
        --enable-panel=KEYPHRASES \
        --enable-panel=MIME_TYPE \
        --enable-panel=VIRTUAL_HOSTS \
        --date-spec=min \
        --output=/tmp/goaccess.html \
        /var/lib/apache2.4/log/access.html
  1. Am i missing something here?
  2. Is a config file mandatory?
allinurl commented 1 month ago

The log you posted is missing MIME_TYPEs, but everything else should work. Run goaccess as shown below and see the results:

# goaccess /var/lib/apache2.4/log/access.html -a --log-format='%v %h %^[%d:%t %^] "%r" %s %b "%R" "%u" %^/%D' --date-format=%d/%b/%Y --time-format=%T -o /tmp/goaccess.html

Also, make sure the panels are enabled in the HTML navigation bar on the left. Click the little circle next to each panel name to toggle them on/off.

Zabrane commented 1 month ago

@allinurl

I've tried GoAccess - 1.9.2 on macOS but i can't see the MIME_TYPE panels.

Screenshot 2024-06-01 at 12 20 55

The log you posted is missing MIME_TYPEs, but everything else should work. Run goaccess as shown below and see the results:

I've changed my Apache log format to include this one %{Content-Type}o :

LogFormat "%V %{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{Content-Type}o\" **%T/%D**" proxy

And it works. I can see the MIME_TYPE in the logs :-)

www.foxxyme.com 90.45.192.246 - - [01/Jun/2024:12:33:13 +0200] "GET /the-premium-solution-psn-cartouche-compatible-laser-pro-noir-canon-045h-1246c002-l1-ct045hb-pro_SKU440207.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 17_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1" "application/json" **0/51352**

But what I need to change to GoAccess --log-format= to be able to read the MIME_TYPE and display it in the associated pane (which I still can't see) ?

Help appreciated.

Zabrane commented 1 month ago

@allinurl found it:

 %M The MIME-type of the requested resource. (In Apache LogFormat: %{Content-Type}o) 

It's working. Thanks a lot for these GREAT software and your support.

allinurl commented 1 month ago

Great to hear it worked and you found it helpful! If you have any more questions, just let me know.