memory leak or must need so much memory?

allinurl / goaccess

GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.

https://goaccess.io

MIT License

18.12k stars 1.1k forks source link

memory leak or must need so much memory? #577

Open toontong opened 7 years ago

toontong commented 7 years ago

./goaccess --version GoAccess - 1.1.

just upgrade from goaccess-0.9.8 then appear this.

CMD was zcat ./*/access.log-$DATE.gz | goaccess --no-query-string > ./report/report.$DATE.html file size： 5.8G access.log-20161124.gz has 7~10 files.

allinurl commented 7 years ago

Did you run the same command in v0.9.8 to see how much it consumes there?

My guess is that you have 7-10 5.8G compressed files (~40.6G), so if you uncompress them, the amount would be a lot. So probably you need the on-disk storage. BTW, how many records are you parsing? e.g., zcat ./*/access.log-$DATE.gz | wc -l

If you still would like to try the in-memory storage, please take a look here.

There are some obvious optimizations that I want to do to the default hash tables which I'm sure will help in this type of cases. Hoping to get to this soon!

toontong commented 7 years ago

no -a , -q only.

zcat ./*/access.log-$DATE.gz | wc -l 1,034,567,817
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                       
35435 root      20   0 31.1g  28g 1028 R 100.0 45.9  81:51.90 /usr/local/bin/goaccess-0.9.8 --no-query-string   
呆~。 the same as verison 0.9.8, it is running, not finish yet, also used so many memory.