search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.88k stars 1.84k forks source link

Geolite Database dependency #404

Open saratvemulapalli opened 3 years ago

saratvemulapalli commented 3 years ago

Geolite database is used for testing GeoIP feature in OpenSearch, it has sample data to test against. It uses mmdb format built by Maxmind (https://dev.maxmind.com/geoip/geoip2/geolite2/).

Today we just depend on Elastic to get this information and we have to build our own sample data to remove the dependency.

org.elasticsearch:geolite2-databases:20191119 is used in the ingest-geoip processor for tests. https://github.com/opensearch-project/OpenSearch/blob/main/modules/ingest-geoip/build.gradle#L48

dblock commented 3 years ago

@saratvemulapalli maybe edit this issue to say what we actually want to do here?

anasalkouz commented 3 years ago

Hi @saratvemulapalli, I see PL already submitted to replace elasticsearch dependencies. Could you please close the issue if there is nothing else pending.

anasalkouz commented 3 years ago

Closing the issue, please feel free to re-open if needed

saratvemulapalli commented 3 years ago

@anasalkouz some how i missed this. I know we have taken care of Securemock, mocksocket and JNA. But did we also remove the geolite test data dependency?

dblock commented 3 years ago

@anasalkouz some how i missed this. I know we have taken care of Securemock, mocksocket and JNA. But did we also remove the geolite test data dependency?

I am pretty sure we haven't. Do you know where this is and how it's used @saratvemulapalli?

saratvemulapalli commented 3 years ago

Yup, looks like its not. Ref: https://github.com/opensearch-project/OpenSearch/blob/main/modules/ingest-geoip/build.gradle#L48

minalsha commented 2 years ago

This is a dependency we need to get rid of but we are still using it for Search. @anasalkouz is this something you can look into since it is related to Search?

unilynx commented 1 year ago

attempting to trim down a docker image I came across modules/ingest-geoip/GeoLite2-City.mmdb today which still ships with opensearch-2.6.0-linux-x64.tar.gz.

Is this supposed to be there? I was under the impression that maxmind forbids redistribution of the geolite2 database, especially if they're older than 30 days (https://www.maxmind.com/en/geolite2/eula 6c)

(If I'm misunderstanding this and the files shipped with opensearch are up to date and properly redistributable... even better)

dblock commented 1 year ago

@vamshin ^

heemin32 commented 1 year ago

Hi @unilynx, could you kindly share what was your reason of trimming down a docker image?

unilynx commented 1 year ago

Hi @unilynx, could you kindly share what was your reason of trimming down a docker image?

It's not about one of the official Opensearch containers, but a CMS package totalling about 2GB (mostly due to Chrome and LLVM) where opensearch is in the 5th or 6th layer or so. This was the next low-hanging fruit (about 60MB ?) after already trimming out the bundled OpenJDK and Opensearch plugins I wouldn't need.