search

cookandy / cloudflare-elk

Quickly analyze your Cloudflare logs with ELK
3 stars 3 forks source link

Maxmind GeoLite Database No Longer Available #2

Open jeffh-cloudflare opened 3 years ago

jeffh-cloudflare commented 3 years ago

Upon start of the Docker container, the following message is displayed:

geolite.maxmind.com not found

After performing an nslookup, I found that the fully qualified domain no longer exists. Maxmind has an article on their website that speaks to this and states they discontinued the original GeoLite database offering. It's been replaced with GeoLite2. Access to the GeoLite2 database does require registration and authentication.

GeoLite Legacy databases are now discontinued https://support.maxmind.com/geolite-legacy-discontinuation-notice/

GeoLite2 Free Geolocation Data https://dev.maxmind.com/geoip/geoip2/geolite2/

GeoIP2 Downloadable Databases https://dev.maxmind.com/geoip/geoip2/downloadable/

I am in the process of trying to figure out how to upgrade your container instance to utilize the updated Maxmind GeoLite2 database and will add more to this issue when I determine the best course of action.

jeffh-cloudflare commented 3 years ago

I found there's a GeoLite2-City.mmdb file in the root of the container that is 0 bytes. When the Docker container starts, it fails to download the database from the original URL. There's a process that involves LogStash looking for the database. Since the file is empty, it causes LogStash to fail to start. You can log into the browser interface, but the cloudflare index is not present and none of the elements in the dashboard function as expected.

[2021-05-21T23:27:47,438][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>"}

[2021-05-21T23:27:47,454][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:inadd'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:402:in `block in converge_state'"]}

[2021-05-21T23:27:47,478][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit

This can be resolved by creating an account on the MaxMind website, generating a license key, then replacing the current curl command in /scripts/pre-start.sh:

curl http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz | gunzip > /GeoLite2-City.mmdb

With the following:

curl https://download.maxmind.com/app/geoip_download\?edition_id\=GeoLite2-City\&license_key\=YOUR_LICENSE_KEY\&suffix\=tar.gz | gunzip > /GeoLite2-City.mmdb

The \ escape characters are required, otherwise the curl will fail with an error stating the MaxMind license key is invalid.

Unfortunately, the 'cloudflare=*' index is still not available, even after doing this, so the dashboard is still empty and Kibana refuses to display any data.