Closed stephen-soltesz closed 4 years ago
Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | ||
---|---|---|---|---|---|
ipannotator/ipannotator.go | 41 | 43 | 95.35% | ||
<!-- | Total: | 73 | 75 | 97.33% | --> |
Files with Coverage Reduction | New Missed Lines | % | ||
---|---|---|---|---|
ipannotator/ipannotator.go | 2 | 95.74% | ||
<!-- | Total: | 2 | --> |
Totals | |
---|---|
Change from base Build 70: | -1.4% |
Covered Lines: | 278 |
Relevant Lines: | 282 |
The first version of the uuid-annotator used upwards of ~2.5GB RSS as a result of expensive CSV load operations. At steady state, the Go process reported ~1GB of "in use" RAM. With four experiments currently using the uuid-annoator, this meant that ~6GB of RAM was claimed but unused, limiting the capacity of the rest of the system.
Using the binary Maxmind DB, we can reduce the RSS by 10x. The aggregate RAM usage of the uuid-annotator with the binary Maxmind DB should be safe to deploy to production. The image below illustrates the dramatic drop in RAM usage with this change.
This change also makes the
rawfile
package (formerlyzipfile
) agnostic about the underlying filetype, leaving the data readers to interpret the format and providing helper functions such asReadFileFromTar()
.This change is