oom problem when convert mmdb to json

maxmind / mmdbinspect

look up records for one or more IPs/networks in one or more .mmdb databases

Apache License 2.0

136 stars 14 forks source link

oom problem when convert mmdb to json #78

Open Paper-Dragon opened 2 weeks ago

Paper-Dragon commented 2 weeks ago

This command uses up all my memory, is it possible to optimize it while converting every ip?

memory: 32G command:

mmdbinspect -db GeoLite2-City.mmdb  0.0.0.0/0 >> GeoLite2-City.json

oschwald commented 2 weeks ago

I believe the current implementation reads all of the data into memory before encoding it to JSON. I do agree that it would be better to stream the records to stdout as we read them. This is a bit harder to do with the Go JSON library as it doesn't have streaming functionality built in, but it does seem doable.