search

allinurl / goaccess

GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.
https://goaccess.io
MIT License
18.6k stars 1.11k forks source link

Question about output html takes a long time #1345

Open death00 opened 5 years ago

death00 commented 5 years ago

My log's size is about 636M. When I use command : goaccess -a -d --date-spec=hr -f /var/log/nginxLog/access.log -p /home/user/app/goaccess-1.3/config/goaccess.conf This takes about 30s, it's ok. But when I use command to generate html : goaccess -a -d --date-spec=hr -f /var/log/nginxLog/access.log -p /home/user/app/goaccess-1.3/config/goaccess.conf -o /var/log/nginxLog/html/go-access.html This takes me about 30min. Is this phenomenon normal? Or my command be optimized?

My os is cat /proc/version Linux version 3.10.0-862.2.3.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) #1 SMP Wed May 9 18:05:47 UTC 2018 cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) cat /proc/cpuinfo| grep "physical id"| sort| uniq| wc -l 1 cat /proc/cpuinfo| grep "cpu cores"| uniq cpu cores : 2 cat /proc/cpuinfo| grep "processor"| wc -l 4 cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c 4 Intel(R) Xeon(R) Gold 61xx CPU cat /proc/meminfo MemTotal: 16266968 kB MemFree: 4483364 kB MemAvailable: 12208160 kB Buffers: 253088 kB Cached: 6972844 kB df -lh Filesystem Size Used Avail Use% Mounted on /dev/vda1 99G 7.7G 87G 9% /

allinurl commented 5 years ago

Can you try removing -d? that's probably the bottleneck as it needs to resolve all those IPs. Though, please note that the OS will cache those look-ups as well.

death00 commented 5 years ago

@allinurl Your suggestion is useful. When I removing -d, it takes only about 40s,

Though, please note that the OS will cache those look-ups as well.

What does look-ups mean? And how I can remove this cache?

allinurl commented 5 years ago

caching is good, so when you run it for the second time should perform better. Take a look at this article for details on a reverse dns look-up.

death00 commented 5 years ago

Thank you so much for your help. By the way, are you considering adding a feature that combines multiple htmls(or allow json to be converted to html)? In my project, I generate a html related to yesterday's log everyday. Now, I want to view html for a few days.