Memory leak on the tunnel helper

[whirm]

Current devel tunnel helper is leaking memory.

Apparently it only happens if exit mode is enabled, (either that or the lower amount of users on that kind of helper is not making the leak so obvious).

UPDATE: the non-exit tunnel helpers are also leaking, just at about half the rate than the others.

[lfdversluis]

2000 hype! (sorry :smile: )

[devos50]

2000 party :beers:

[whirm]

With the increase in users this is getting pretty bad. 3x memory consumption in less than 24h.

Anybody up for this one?

[whirm]

@devos50 both manhole and the cpu and memory profilers are integrated into the twisted dispersy tracker plugin, look at the help output.

Note that for twistd to find the tracker plugin you need to add the dispersy directory to the PYTHONPATH and call twistd from the dispersy's repo parent directory (IE, if you clone dispersy to your $HOME, add $HOME/dispersy to the PYTHONPATH and run twistd from your $HOME)

You can look at dispersy/scripts/dispersy-tracker@.service for an example on how it gets started.

[devos50]

@whirm do you have the memory dumps of the big tunnel helpers?

[whirm]

Yup, my internet connection broke overnight and I forgot to run it in screen, so we only have a 12 hours run, but should be enough.

Can you receive 26 MB attachments on your mail?

[whirm]

Sadly, there's still a leak, In a couple of days, memory consumption has grown by 1GB...

[devos50]

Can you give me the memory dumps again?

[devos50]

Also, does the consumption grow slower now?

[whirm]

I'll start one with memory dumping ASAP

re growth: Yeah, I'd say about ~4 times slower.

[devos50]

That's some good news already so a part of it got fixed!

[whirm]

yeah, it should last a week across reboots :)

[devos50]

@whirm I modified the main.py script to dump memory every half an hour and started it on the debian container so in a few days I have some useful data hopefully.

[synctext]

productive weekend!

[devos50]

The increase in memory is not due to a memory leak. Instead, the problem is in the hidden community, where entries in the intro_point_for dictionary are created if you start acting as introduction point for other peers. This dictionary keeps growing, keeping references to TunnelExitSocket which in turn can become pretty big. Items are never removed from intro_point_for which means that nodes act as introduction point for other peers as long as they are running Tribler.

I'm not really sure what's the best way to solve this since I'm not familiar with the implementation of the hidden community in Tribler. When a node stop being an introduction point for a torrent? @synctext do you have any ideas about this?

Update: also, relay_session_keys in TunnelCommunity becomes pretty big after a while. Maybe keys for a circuit not properly cleaned up?

[lfdversluis]

@devos50 Sounds like #2089

[whirm]

I've created an issue for this so it can be dealt by the students involved in the tunnel community. @synctext I'll leave the actuall assigning to you :)

[lfdversluis]

Sorry, I meant looks like #2068

[whirm]

@lfdversluis thanks! I've updated #2012

[devos50]

@lfdversluis yes that one is related indeed. The tunnel and hidden community need some attention since there are various bugs left.