maxmind / mmdbwriter

Go library for writing MaxMind DB (mmdb) files
Apache License 2.0
116 stars 28 forks source link

Dedupe data when inserting into tree #10

Closed oschwald closed 4 years ago

oschwald commented 4 years ago

This deduplicates data when inserting it into the tree. This is a classic case of trading CPU time for reduced memory usage. This is about 10% slower in my testing, but it makes it much easier to use the writer for a large tree without using up all of your memory. Previously, it was on the end user to deduplicate data to prevent excess memory usage.

This also changes the Tree.Get method to return the interface directly rather than a pointer to the interface. The nil interface should be treated as no value being returned.