m-lab / uuid-annotator

Produces metadata locally for every connection on each server.
Apache License 2.0
0 stars 0 forks source link

Support binary maxmind db to reduce RAM #10

Closed stephen-soltesz closed 4 years ago

stephen-soltesz commented 4 years ago

The first version of the uuid-annotator used upwards of ~2.5GB RSS as a result of expensive CSV load operations. At steady state, the Go process reported ~1GB of "in use" RAM. With four experiments currently using the uuid-annoator, this meant that ~6GB of RAM was claimed but unused, limiting the capacity of the rest of the system.

Using the binary Maxmind DB, we can reduce the RSS by 10x. The aggregate RAM usage of the uuid-annotator with the binary Maxmind DB should be safe to deploy to production. The image below illustrates the dramatic drop in RAM usage with this change.

uuid-rss2

This change also makes the rawfile package (formerly zipfile) agnostic about the underlying filetype, leaving the data readers to interpret the format and providing helper functions such as ReadFileFromTar().


This change is Reviewable

coveralls commented 4 years ago

Pull Request Test Coverage Report for Build 87


Changes Missing Coverage Covered Lines Changed/Added Lines %
ipannotator/ipannotator.go 41 43 95.35%
<!-- Total: 73 75 97.33% -->
Files with Coverage Reduction New Missed Lines %
ipannotator/ipannotator.go 2 95.74%
<!-- Total: 2 -->
Totals Coverage Status
Change from base Build 70: -1.4%
Covered Lines: 278
Relevant Lines: 282

💛 - Coveralls
stephen-soltesz commented 4 years ago

https://github.com/m-lab/dev-tracker/issues/534