ChimeraCoder / anaconda

A Go client library for the Twitter 1.1 API
MIT License
1.14k stars 247 forks source link

Memory leak (possible goroutine leak) in throttledQuery #229

Open carbocation opened 6 years ago

carbocation commented 6 years ago

I run a long-lived binary that queries multiple social media providers. I use anaconda to query Twitter. It makes hundreds of queries per minute, and over the course of days/weeks, it ends up using all available RAM on the system.

Profiling, it seems that throttledQuery itself is the source of the leaks. Here is the inuse_space graph from the profile (note: it's an SVG and can be clicked on / zoomed in): profiled image

As you can see, I terminated the program when 300MB had leaked (the rest of the program uses a few dozen MB total). I'm guessing that this is a goroutine leak, but I'm still working on profiling that.

carbocation commented 6 years ago

Memory leaks are anticipated in the docs ( https://github.com/ChimeraCoder/anaconda#query-queue-persistence ) but it seems that rather than coding around them, it might be possible to address the leaks by restructuring the function itself.