search

GitSquared / node-geolite2-redist

Redistribution of MaxMind GeoLite2 GeoIP databases as an npm library
https://gitsquared.github.io/node-geolite2-redist/
Other
81 stars 18 forks source link

download specific dbs #19

Closed atlanteh closed 2 years ago

atlanteh commented 2 years ago

Hey, is there a way to only download a specific database and not all them?

GitSquared commented 2 years ago

No, this is not currently supported. Could you tell me a bit more about your usecase for this?

atlanteh commented 2 years ago

Thanks for the super fast reply! Well I only need the country iso_code to redirect the user to the right place based on that so I don't need the rest of the dbs.. I don't want to give the container any write access to any persistence for security reasons, so I want it to write to in memory "/tmp" but the city db would take a lot of memory without me needing it..

GitSquared commented 2 years ago

Thanks for the explanation. Downloading only the country DB would save you about 37MB tops, which is really negligible on most systems those days, so I'm not entirely convinced it's worth adding a new feature for. Are you working in an IoT context?

atlanteh commented 2 years ago

@GitSquared actually on my computer I'm seeing different values City is 70MB ASN is 7MB Country is 5.5MB image

I don't feel this is negligible. I'm not using IOT, but I'm working on an app that should consume a lot of memory, and should be very fast, so I'm trying to save up as much as possible, and saving 77MB of memory on each container just by removing these DBs sounds great to me.

Is there a reason not to implement this? I mean this feels like a basic feature request - download only what you need, no?

GitSquared commented 2 years ago

Sorry, you're right — I was looking at the compressed archives that the lib download, not the size of the unpacked databases.

I'm working on an app that should consume a lot of memory ... so I'm trying to save up as much as possible

Note that if you're writing data to /tmp, that's disk storage on most systems, not RAM - unless you mount your whole container in a ramdisk

Is there a reason not to implement this? I mean this feels like a basic feature request - download only what you need, no?

It is pretty basic indeed, but I haven't touched this lib in a long time and wanted to clean it up a bit, so its a bit more of an investment on my end. But I guess i'll take the opportunity to finally do that ;)

Should have something for you very soon, this week at least.

atlanteh commented 2 years ago

That's amazing. thank you so much! Regarding the /tmp please have a look here:

https://aws.amazon.com/about-aws/whats-new/2018/03/amazon-ecs-adds-support-for-shm-size-and-tmpfs-parameters/

The tmpfs parameter allows you to mount a temporary volume in memory. This removes the need to store data in your container instance and allows containers to run in a read-only mode, which can improve the security of your service.

GitSquared commented 2 years ago

interesting, thanks for the link!

GitSquared commented 2 years ago

Hey — just published https://github.com/GitSquared/node-geolite2-redist/releases/tag/v3.0.0 with this feat added! You can simply open() a DB or a list of DB like ['GeoLite2-Country', 'GeoLite2-ASN'] and it'll just download what you need. Detailed examples here.

Lmk if you have any questions or trouble!

atlanteh commented 2 years ago

Thanks so much for this awesome work! I tested this and it works great. The customStorageDir didn't work, so I created a quick PR: https://github.com/GitSquared/node-geolite2-redist/pull/21