search

Tribler / dispersy

The elastic database system. A database designed for P2P-like scenarios, where potentially millions of computers send database updates around.
http://www.tribler.org
87 stars 43 forks source link

Fix memory leakage Tribler.dispersy.candidate.Candidate #442

Open synctext opened 9 years ago

synctext commented 9 years ago

we have a small memory leak.

4000 Tribler.dispersy.candidate.Candidate objects are in memory after 30h. 145313 Tribler.dispersy.candidate.Candidate objects are in memory after 784h.

Suspects for memory leakage. Our automatic reports with difficult 60MByte traces: http://jenkins.tribler.org/job/Test_performance_Tribler_idle_24H_nightly_devel/605/artifact/output/memprof/2180/

Elric has some tools parsing this. These can identify the possible dictionary.

Using Dowser we see the following suspects: Tribler.dispersy.candidate.Candidate Tribler.dispersy.candidate.WalkCandidate Tribler.dispersy.member.Member (probably linked to their key: Tribler.dispersy.crypto.M2CryptoPK)

Tribler shows 1108 channels, Dowser stores Tribler.Main.Utility.GuiDBTuples.Channel

At 11:00am, after running for 24hours: dowser_1100am_dump

8 hours later: dowser_1700_dump

NielsZeilemaker commented 9 years ago

How many communities did you have in memory?

synctext commented 9 years ago

standard testing. When doing 'mp3' search certains objects become big. They nicely disappear with a result list of just a few items. Same for channel browsing. That is all good.

These dispersy items never decrease. Slowly grow with each new discovered peers.

Standard Tribler default communities.

synctext commented 9 years ago

After running Tribler for 4 weeks 4 days and 16hours: thousands of objects in memory.

ps aux | grep tri
pouwelse 13289  0.0  0.0   4472     0 pts/20   S+   Jul15   0:00 /bin/sh ./tribler.sh
pouwelse 13293 29.1 18.4 5631764 1416800 pts/20 Sl+ Jul15 13711:40 python2.7 Tribler/Main/tribler.py

tribler_6 4 0pre_with_145313_candidate_objects_in_memory__dowser_aug2015

tribler_6 4 0pre_with_145313_candidate_objects_in_memory__htop_aug2015

Only a few channels loaded: tribler_6 4 0pre_with_145313_candidate_objects_in_memory__debug_panel_aug2015

NielsZeilemaker commented 9 years ago

Could you start the wx inspectiontool (ctrl+i), and then in the console

for community in dispersy.get_communities():
    print community.get_classification(), len(community.candidates)
synctext commented 8 years ago

2.2 GByte memory usage after running from 5Nov - 1Dec. Plus 231GByte of IO and 100h of CPU. (devel 785f442ec6fef5fffaa33bfbf086bd22bf401c87)

ps aux | grep tribler && date
pouwelse 26676 16.2 29.2 6479460 2252572 pts/23 Sl+ Nov05 6016:18 python2.7 Tribler/Main/tribler.py
Tue Dec  1 14:51:32 CET 2015

tribler_dowser__210409_dispersycandidate_and_walkcandidate_instances_28days_uptime__1dec2015

tribler_htop_memory_leakage_and_231gbyte_io_in_28days__1dec2015

NielsZeilemaker commented 8 years ago

And the output of the command I posted above?

synctext commented 8 years ago

hey! @NielsZeilemaker Can't get that WX inspection tool to start. Even tried understanding the documentation.. http://wiki.wxpython.org/Widget%20Inspection%20Tool

NielsZeilemaker commented 8 years ago

The key is only bound in the debug panel, first go there

synctext commented 8 years ago

Looks healthy.. Still missing 210000 instances... tribler_wx_inspection_tool_output

NielsZeilemaker commented 8 years ago

And

for community in dispersy.get_communities():
    print community.get_classification(), len(community.request_cache._identifiers)
synctext commented 8 years ago

Hmmm.. tribler_wx_inspection_tools2

NielsZeilemaker commented 8 years ago

And after

import gc
gc.collect()
synctext commented 8 years ago

checking..

synctext commented 8 years ago 
gc.collect()
68

trying dowser again..

NielsZeilemaker commented 8 years ago

Let's remove the communities one by one. Verify the memory usage afterwards

c = dispersy.get_communities()[0]
c.unload_community()

print "unloaded", c.get_classification()
print "remaining"

for community in dispersy.get_communities():
    print community.get_classification()

import gc
gc.collect()

Not sure how Tribler will respond :smile: And remove them one by one.

synctext commented 8 years ago

after the unload attempt:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/community.py", line 1069, in unload_community
    self._request_cache.clear()
  File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/requestcache.py", line 230, in clear
    assert isInIOThread(), "RequestCache must be used on the reactor's thread"
AssertionError: RequestCache must be used on the reactor's thread
NielsZeilemaker commented 8 years ago 
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
    c = dispersy.get_communities()[0]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload()

print "remaining"
for community in dispersy.get_communities():
    print community.get_classification()

import gc
gc.collect()
synctext commented 8 years ago

Thnx Niels! Sadly after 4 weeks Tribler ran out of memory to use. Starting dowser and WX inspection tool surely didn't help. Will try again.

*** Error in `python2.7': munmap_chunk(): invalid pointer: 0x00007fa5f32b5a20 ***
Aborted (core dumped)
synctext commented 8 years ago 
Python 2.7.10 on linux2, wxPython 2.8.12.1 (gtk2-unicode)
NOTE: The 'obj' variable refers to the object selected in the tree.
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
    c = dispersy.get_communities()[0]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload()

print "remaining"
for community in dispersy.get_communities():
    print community.get_classification()

import gc
gc.collect()

 print "remaining"
  File "<input>", line 1
    print "remaining"
    ^
IndentationError: unexpected indent
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
    c = dispersy.get_communities()[0]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload()
unloaded ChannelCommunity
print "remaining"
remaining
for community in dispersy.get_communities():
    print community.get_classification()

HiddenTunnelCommunity
ChannelCommunity
ChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
SearchCommunity
DiscoveryCommunity
AllChannelCommunity
import gc
gc.collect()
76584

Memory was growing a bit, even after 75 of object where removed

XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650  0.0  0.0   4476   868 pts/23   S+   18:50   0:00 /bin/sh ./tribler.sh
pouwelse 17654 32.7  3.6 2056020 279700 pts/23 Sl+  18:50   0:15 python2.7 Tribler/Main/tribler.py
pouwelse 17687  0.0  0.0   9496  2140 pts/1    S+   18:51   0:00 grep tribler
Fri Dec  4 18:51:22 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650  0.0  0.0   4476   672 pts/23   S+   Dec04   0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.0  7.4 3729632 570700 pts/23 Sl+  Dec04 604:01 python2.7 Tribler/Main/tribler.py
pouwelse 26722  0.0  0.0   9500  2092 pts/1    R+   13:52   0:00 grep tribler
Tue Dec  8 13:52:56 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650  0.0  0.0   4476   296 pts/23   S+   Dec04   0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.4  9.5 3836896 737932 pts/23 Sl+  Dec04 980:07 python2.7 Tribler/Main/tribler.py
pouwelse 29688  0.0  0.0   9500  2148 pts/1    S+   17:03   0:00 grep tribler
Thu Dec 10 17:03:49 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650  0.0  0.0   4476   296 pts/23   S+   Dec04   0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.4 10.2 3849048 784964 pts/23 Sl+  Dec04 981:30 python2.7 Tribler/Main/tribler.py
pouwelse 30878  0.0  0.0   9500  1996 pts/1    S+   17:13   0:00 grep tribler
Thu Dec 10 17:13:14 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650  0.0  0.0   4476   296 pts/23   S+   Dec04   0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.4 10.2 3849048 788756 pts/23 Sl+  Dec04 981:52 python2.7 Tribler/Main/tribler.py
pouwelse 30914  0.0  0.0   9500  2184 pts/1    S+   17:15   0:00 grep tribler
Thu Dec 10 17:15:17 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650  0.0  0.0   4476   296 pts/23   S+   Dec04   0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.4 10.4 3849048 800956 pts/23 Sl+  Dec04 982:36 python2.7 Tribler/Main/tribler.py
pouwelse 31020  0.0  0.0   9500  2084 pts/1    S+   17:19   0:00 grep tribler
Thu Dec 10 17:19:41 CET 2015

After 1 unload:

Tribler.dispersy.candidate.Candidate
Min: 33606 Cur: 33607 Max: 33609 TRACE
Tribler.dispersy.candidate.WalkCandidate
Min: 25785 Cur: 25786 Max: 25788 TRACE
synctext commented 8 years ago

@NielsZeilemaker Here you can see my embarrassing fiddling. It does not free any candidate.Candidate, status after multiple unload() attempts:

Tribler.dispersy.candidate.Candidate
Min: 33612 Cur: 33692 Max: 33693 TRACE
Tribler.dispersy.candidate.WalkCandidate
Min: 25779 Cur: 26109 Max: 26122 TRACE

Console logfile:

Python 2.7.10 on linux2, wxPython 2.8.12.1 (gtk2-unicode)
NOTE: The 'obj' variable refers to the object selected in the tree.
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
    c = dispersy.get_communities()[0]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload()

print "remaining"
for community in dispersy.get_communities():
    print community.get_classification()

import gc
gc.collect()
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
    c = dispersy.get_communities()[0]
    print "unloaded", c.get_classification()

do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
@blocking_call_on_reactor_thread
def do_unload2():
    c = dispersy.get_communities()[1]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload2()
unloaded HiddenTunnelCommunity
gc.collect()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
NameError: name 'gc' is not defined
import gc
gc.collect()
236
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload3(int num):
  File "<input>", line 2
    def do_unload3(int num):
                         ^
SyntaxError: invalid syntax
def do_unload3(num):
    c = dispersy.get_communities()[num]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload3(2)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "<input>", line 3, in do_unload3
  File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/community.py", line 1069, in unload_community
    self._request_cache.clear()
  File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/requestcache.py", line 230, in clear
    assert isInIOThread(), "RequestCache must be used on the reactor's thread"
AssertionError: RequestCache must be used on the reactor's thread
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload3(num):
    c = dispersy.get_communities()[num]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload3(2)
unloaded ChannelCommunity
gc.collect()
12
do_unload(3)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/util.py", line 47, in helper
    return blockingCallFromThread(reactor, func, *args, **kargs)
  File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/util.py", line 238, in blockingCallFromThread
    result.raiseException()
  File "<string>", line 2, in raiseException
TypeError: do_unload() takes no arguments (1 given)
do_unload3(3)
unloaded PreviewChannelCommunity
gc.collect()
156
do_unload3(4)
unloaded ChannelCommunity
gc.collect()
63
do_unload3(5)
unloaded ChannelCommunity
do_unload3(5)
unloaded ChannelCommunity
do_unload3(6)
unloaded PreviewChannelCommunity
do_unload3(7)
unloaded PreviewChannelCommunity
for community in dispersy.get_communities():
    print community.get_classification()

PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
ChannelCommunity
ChannelCommunity
SearchCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
AllChannelCommunity
for community in dispersy.get_communities():
    print community.get_classification()

PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
ChannelCommunity
ChannelCommunity
PreviewChannelCommunity
SearchCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
AllChannelCommunity
do_unload3(7)
unloaded SearchCommunity
do_unload3(7)
unloaded SearchCommunity
for community in dispersy.get_communities():
    print community.get_classification()

PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
ChannelCommunity
ChannelCommunity
PreviewChannelCommunity
SearchCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
AllChannelCommunity
do_unload3(3)
unloaded HiddenTunnelCommunity

do_unload3(5)
unloaded ChannelCommunity
do_unload3(6)
unloaded PreviewChannelCommunity
gc.collect()
183
do_unload3(7)
unloaded PreviewChannelCommunity
do_unload3(8)
unloaded ChannelCommunity
do_unload3(9)
unloaded PreviewChannelCommunity
do_unload3(10)
unloaded PreviewChannelCommunity
do_unload3(11)
unloaded SearchCommunity
do_unload3(0)
unloaded PreviewChannelCommunity
do_unload3(1)
unloaded PreviewChannelCommunity
do_unload3(2)
unloaded PreviewChannelCommunity
do_unload3(4)
unloaded PreviewChannelCommunity
do_unload3(3)
unloaded PreviewChannelCommunity
do_unload3(5)
unloaded PreviewChannelCommunity
do_unload3(6)
unloaded PreviewChannelCommunity
do_unload3(7)
unloaded SearchCommunity
do_unload3(8)
unloaded DiscoveryCommunity
do_unload3(9)
unloaded PreviewChannelCommunity
do_unload3(10)
unloaded PreviewChannelCommunity
do_unload3(11)
unloaded PreviewChannelCommunity
for community in dispersy.get_communities():
    print community.get_classification()

PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
ChannelCommunity
PreviewChannelCommunity
ChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
SearchCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
AllChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
for community in dispersy.get_communities():
    print community.get_classification()

PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
ChannelCommunity
PreviewChannelCommunity
ChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
SearchCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
AllChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
do_unload3(49)
unloaded PreviewChannelCommunity
do_unload3(50)
unloaded AllChannelCommunity
gc.collect()
80
NielsZeilemaker commented 8 years ago

So then probably something else is keeping a reference to the candidates. So that would be Tribler. Is there a debug panel which has any lists of candidates?