search

ipinfo / mmdbctl

mmdbctl is an MMDB file management CLI supporting various operations on MMDB database files.
Apache License 2.0
93 stars 13 forks source link

MMDB with nested structures break read and export #2

Closed jhg03a closed 1 year ago

jhg03a commented 2 years ago

When attempting to use this project to export from an mmdb with nested objects it throws an error: err: failed to get record for next subnet: maxminddb: cannot unmarshal map into type string. The read operation also fails with an error: err: couldn't get data for <requested_ip>.

In both these cases it seems to stem from the assumption that records are a map[string]string instead of map[string]interface{}. Unfortunately that opens up a new host of problems with representation in output formats incapable of nested structures natively like csv/tsv.

UmanShahzad commented 2 years ago

Yeah this is definitely an assumption we've made deliberately, because our own use cases only entailed strings and we realized that letting it become arbitrary is both not useful to us and causes fundamental issues with some input/ouput formats as you already mentioned.

I think the most we could go for is trying to support data types that can be trivially converted back & forth to string, like numbers and booleans. But arrays and maps will always be super opinionated.

jhg03a commented 2 years ago

Be aware that the commercial maxmind datasets leverage both arrays and maps which might affect the usability of the project in a broader audience such as if you're performing augments/updates.

I've been able to mostly work around and get it working for me by deleting code around the use of CSV/TSV because the underlying mmdb libraries still support the nested objects and json can express them fine when going to marshal/unmarshal them.

UmanShahzad commented 2 years ago

Yeah of course. IPinfo (https://ipinfo.io) also ships MMDB files and we try to stay away from arrays/maps because of some of these limitations.

I think the best we'll be able to do then is similar to what you've done but by simply handling the error and printing something, saying that CSV/TSV support is not available for arrays/maps, when they're detected.

jhg03a commented 2 years ago

Switching from map[string]string to map[string]interface{} also opened up a new host of issues as well where things like integer values were getting dropped or being able to import nested objects too.

jhg03a commented 2 years ago

Data enrichment can look something like export of an mmdb to json. Then use standard json manipulation tools like jq or jmespath, and reimport into a new mmdb.

UmanShahzad commented 1 year ago

@jhg03a This is fixed - can you provide feedback?