search

oschwald / geoip2-golang

Unofficial MaxMind GeoIP2 Reader for Go
ISC License
1.89k stars 195 forks source link

Large memory usage #48

Closed pjebs closed 5 years ago

pjebs commented 5 years ago

Firstly I want to say thank you for the library.

Secondly, I have noticed that it is using 800-1000 megs of memory. Does that sound correct?

@alexclifford

oschwald commented 5 years ago

No, that does not sound correct. I suppose if you are looking up quite a few IPs in a loop, you would likely be generating a significant amount of garbage, causing high memory usage.

pjebs commented 5 years ago

How much memory should or be? Is it possible that there is a leak in Close(). We are downloading a new version of database every 24 hours and closing and reloading new database

oschwald commented 5 years ago

If you are using the default mmap mode, the reader itself hardly takes any real memory. The amount would be measured in KB. It will use additional virtual memory as the database is memory mapped. I'd guess most of the memory usage you see is from garbage and memory not returned to the system by the garbage collector. I can't really speak to the internals of your application.

pjebs commented 5 years ago

If that is the case, how can we force garbages collector to release?

pjebs commented 5 years ago

I'll try this: FreeOSMemory from debug package

pjebs commented 5 years ago

Found the issue: we were downloading from s3 using recommended code which uses aws.s3manager and aws.WriteAtBuffer (which has a massive memory leak). Using GetObject() solved it.