search

GilbN / geoip2influx

A python script that will parse the nginx access.log and send geolocation metrics and log metrics to InfluxDB
MIT License
73 stars 15 forks source link

Recommended memory ? #15

Closed sim-pml closed 1 year ago

sim-pml commented 1 year ago

Hi ,

First of all, great dashboard and script works really nicely and gives a ton of useful information.

Are you able to advise on what specs should be used for this? I found that running it for 3/4 hours after accumlating data , it starts slowing down and influxdb uses a ton of memory (vm only has 4GB as it's only running nginx) so quickly ends up in swap. I was thinking of offloading processing for it elsewhere to take the load off nginx. I just figured being a docker instance, the footprint might be a bit smaller.

If you think this is more of an influxDB issue, I am happy to be told to go elsewhere :) . Nginx is quite busy I had 20,000+ total requests for the 3-4 hours I had it running.

Many thanks,

GilbN commented 1 year ago

Hi. You could try and optimize the queries. I haven't looked at it for a long time. I don't use it. But the more data the query fetches the more ram it will take up. iirc nginx and grafana did a collaboration, maybe that works better.

sim-pml commented 1 year ago

Thanks for the quick response.

We have managed to contain memory use by limited cpu/memory in the docker-compose.yml.

Will see how it performs as more data is gathered but I think this is a workaround for the moment.

Code to fix in docker-compose.yml :

influxdb:
  image: influxdb:1.8-alpine
  mem_limit: 1500M
  mem_reservation: 500M
  memswap_limit: 1500M
  cpus: 1