logstash-plugins / logstash-filter-geoip

Apache License 2.0
64 stars 80 forks source link

Add `interval` option to config auto-update schedule #188

Open kaisecheng opened 3 years ago

kaisecheng commented 3 years ago

Since 7.14, geoip check database updates every 24h since the server start time. The suggestion is to add interval option to allow users to change the checking interval in hours. The number should be >= 24 and < 168 (7 days)

xpack.geoip.download.interval=72
kaisecheng commented 3 years ago

open question

  1. Is the interval only for database update check? Is the expiry routine running every day to respect the 30 days expiry rule?
  2. What is the motivation of have interval config? Checking updates every three days can save some server load considering users deploy ten thousand instances.
kaisecheng commented 3 years ago
  1. elasticsearch check if db is valid on every use of geoip processor and tag doc if that’s not the case. cleanup of expired databases is done with ingest.geoip.downloader.poll.interval
  2. Check every 3 days is to reduce bandwidth, it’s configurable to allow for checking more aggressive (every day) if customer wants to always have freshest data possible or for tests where it’s easier to change this interval and wait seconds instead of days