Useragent spawns too many threads under unknown circumstances

[varac]

This is weird. Today nagios reported that try+dev have too many threads (in general). These are the images from pnp4nagios:

From dev.pix pnp4nagios threads graph:

From try.pix pnp4nagios threads graph:

On wazokazi, this graph looks pretty good. From wazokazi pnp4nagios threads graph:

I suspected the useragent and indeed it is, looking at try.pix:

root@pixelated:~# cat /proc/$(pgrep pixelated-user)/status | grep Threads
Threads:    2913

root@pixelated:~# dpkg -l |grep pixelated
ii  pixelated-server                          0.2~104.gbpb9a7d0         all                       Runs pixelated as a server
ii  pixelated-user-agent                      0.6~247.gbp3772c8         all                       API to serve the pixelated front-end requests

So somehow, under some circumstances, there are threads steadyly increasing, while with the same useragent version this doesn't happen (like on wazokazi or unstable).

[varac]

I deployed to try to install the newest versions of everything. Lets see if this changes anything. But this behaviour is still present on dev.pix to investigate.

[bwagnerr]

The only suspicion I have for this is the re-creation of soledad mechanism, if you get an invalid token, it creates a new soledad instance and spawns 10 new threads. I was thinking the old soledad object might not get out of scope and be garbage collected, that would cause an increase on the number of threads over time

[varac]

@bwagnerr: Is there a leap issue for this ? Are the soledad devs aware of this ? How much work would it be to fix it ? When we're focussing on fixing the mem leakage we should consider fixing a thread leakage as well.