Open synctext opened 9 years ago
How many communities did you have in memory?
standard testing. When doing 'mp3' search certains objects become big. They nicely disappear with a result list of just a few items. Same for channel browsing. That is all good.
These dispersy items never decrease. Slowly grow with each new discovered peers.
Standard Tribler default communities.
After running Tribler for 4 weeks 4 days and 16hours: thousands of objects in memory.
ps aux | grep tri
pouwelse 13289 0.0 0.0 4472 0 pts/20 S+ Jul15 0:00 /bin/sh ./tribler.sh
pouwelse 13293 29.1 18.4 5631764 1416800 pts/20 Sl+ Jul15 13711:40 python2.7 Tribler/Main/tribler.py
Only a few channels loaded:
Could you start the wx inspectiontool (ctrl+i), and then in the console
for community in dispersy.get_communities():
print community.get_classification(), len(community.candidates)
2.2 GByte memory usage after running from 5Nov - 1Dec. Plus 231GByte of IO and 100h of CPU. (devel 785f442ec6fef5fffaa33bfbf086bd22bf401c87)
ps aux | grep tribler && date
pouwelse 26676 16.2 29.2 6479460 2252572 pts/23 Sl+ Nov05 6016:18 python2.7 Tribler/Main/tribler.py
Tue Dec 1 14:51:32 CET 2015
And the output of the command I posted above?
hey! @NielsZeilemaker Can't get that WX inspection tool to start. Even tried understanding the documentation.. http://wiki.wxpython.org/Widget%20Inspection%20Tool
The key is only bound in the debug panel, first go there
Looks healthy.. Still missing 210000 instances...
And
for community in dispersy.get_communities():
print community.get_classification(), len(community.request_cache._identifiers)
Hmmm..
And after
import gc
gc.collect()
checking..
gc.collect()
68
trying dowser again..
Let's remove the communities one by one. Verify the memory usage afterwards
c = dispersy.get_communities()[0]
c.unload_community()
print "unloaded", c.get_classification()
print "remaining"
for community in dispersy.get_communities():
print community.get_classification()
import gc
gc.collect()
Not sure how Tribler will respond :smile: And remove them one by one.
after the unload attempt:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/community.py", line 1069, in unload_community
self._request_cache.clear()
File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/requestcache.py", line 230, in clear
assert isInIOThread(), "RequestCache must be used on the reactor's thread"
AssertionError: RequestCache must be used on the reactor's thread
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
c = dispersy.get_communities()[0]
c.unload_community()
print "unloaded", c.get_classification()
do_unload()
print "remaining"
for community in dispersy.get_communities():
print community.get_classification()
import gc
gc.collect()
Thnx Niels! Sadly after 4 weeks Tribler ran out of memory to use. Starting dowser and WX inspection tool surely didn't help. Will try again.
*** Error in `python2.7': munmap_chunk(): invalid pointer: 0x00007fa5f32b5a20 ***
Aborted (core dumped)
Python 2.7.10 on linux2, wxPython 2.8.12.1 (gtk2-unicode)
NOTE: The 'obj' variable refers to the object selected in the tree.
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
c = dispersy.get_communities()[0]
c.unload_community()
print "unloaded", c.get_classification()
do_unload()
print "remaining"
for community in dispersy.get_communities():
print community.get_classification()
import gc
gc.collect()
print "remaining"
File "<input>", line 1
print "remaining"
^
IndentationError: unexpected indent
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
c = dispersy.get_communities()[0]
c.unload_community()
print "unloaded", c.get_classification()
do_unload()
unloaded ChannelCommunity
print "remaining"
remaining
for community in dispersy.get_communities():
print community.get_classification()
HiddenTunnelCommunity
ChannelCommunity
ChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
SearchCommunity
DiscoveryCommunity
AllChannelCommunity
import gc
gc.collect()
76584
Memory was growing a bit, even after 75 of object where removed
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650 0.0 0.0 4476 868 pts/23 S+ 18:50 0:00 /bin/sh ./tribler.sh
pouwelse 17654 32.7 3.6 2056020 279700 pts/23 Sl+ 18:50 0:15 python2.7 Tribler/Main/tribler.py
pouwelse 17687 0.0 0.0 9496 2140 pts/1 S+ 18:51 0:00 grep tribler
Fri Dec 4 18:51:22 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650 0.0 0.0 4476 672 pts/23 S+ Dec04 0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.0 7.4 3729632 570700 pts/23 Sl+ Dec04 604:01 python2.7 Tribler/Main/tribler.py
pouwelse 26722 0.0 0.0 9500 2092 pts/1 R+ 13:52 0:00 grep tribler
Tue Dec 8 13:52:56 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650 0.0 0.0 4476 296 pts/23 S+ Dec04 0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.4 9.5 3836896 737932 pts/23 Sl+ Dec04 980:07 python2.7 Tribler/Main/tribler.py
pouwelse 29688 0.0 0.0 9500 2148 pts/1 S+ 17:03 0:00 grep tribler
Thu Dec 10 17:03:49 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650 0.0 0.0 4476 296 pts/23 S+ Dec04 0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.4 10.2 3849048 784964 pts/23 Sl+ Dec04 981:30 python2.7 Tribler/Main/tribler.py
pouwelse 30878 0.0 0.0 9500 1996 pts/1 S+ 17:13 0:00 grep tribler
Thu Dec 10 17:13:14 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650 0.0 0.0 4476 296 pts/23 S+ Dec04 0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.4 10.2 3849048 788756 pts/23 Sl+ Dec04 981:52 python2.7 Tribler/Main/tribler.py
pouwelse 30914 0.0 0.0 9500 2184 pts/1 S+ 17:15 0:00 grep tribler
Thu Dec 10 17:15:17 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650 0.0 0.0 4476 296 pts/23 S+ Dec04 0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.4 10.4 3849048 800956 pts/23 Sl+ Dec04 982:36 python2.7 Tribler/Main/tribler.py
pouwelse 31020 0.0 0.0 9500 2084 pts/1 S+ 17:19 0:00 grep tribler
Thu Dec 10 17:19:41 CET 2015
After 1 unload:
Tribler.dispersy.candidate.Candidate
Min: 33606 Cur: 33607 Max: 33609 TRACE
Tribler.dispersy.candidate.WalkCandidate
Min: 25785 Cur: 25786 Max: 25788 TRACE
@NielsZeilemaker Here you can see my embarrassing fiddling. It does not free any candidate.Candidate, status after multiple unload() attempts:
Tribler.dispersy.candidate.Candidate
Min: 33612 Cur: 33692 Max: 33693 TRACE
Tribler.dispersy.candidate.WalkCandidate
Min: 25779 Cur: 26109 Max: 26122 TRACE
Console logfile:
Python 2.7.10 on linux2, wxPython 2.8.12.1 (gtk2-unicode)
NOTE: The 'obj' variable refers to the object selected in the tree.
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
c = dispersy.get_communities()[0]
c.unload_community()
print "unloaded", c.get_classification()
do_unload()
print "remaining"
for community in dispersy.get_communities():
print community.get_classification()
import gc
gc.collect()
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
c = dispersy.get_communities()[0]
print "unloaded", c.get_classification()
do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
@blocking_call_on_reactor_thread
def do_unload2():
c = dispersy.get_communities()[1]
c.unload_community()
print "unloaded", c.get_classification()
do_unload2()
unloaded HiddenTunnelCommunity
gc.collect()
Traceback (most recent call last):
File "<input>", line 1, in <module>
NameError: name 'gc' is not defined
import gc
gc.collect()
236
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload3(int num):
File "<input>", line 2
def do_unload3(int num):
^
SyntaxError: invalid syntax
def do_unload3(num):
c = dispersy.get_communities()[num]
c.unload_community()
print "unloaded", c.get_classification()
do_unload3(2)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 3, in do_unload3
File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/community.py", line 1069, in unload_community
self._request_cache.clear()
File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/requestcache.py", line 230, in clear
assert isInIOThread(), "RequestCache must be used on the reactor's thread"
AssertionError: RequestCache must be used on the reactor's thread
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload3(num):
c = dispersy.get_communities()[num]
c.unload_community()
print "unloaded", c.get_classification()
do_unload3(2)
unloaded ChannelCommunity
gc.collect()
12
do_unload(3)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/util.py", line 47, in helper
return blockingCallFromThread(reactor, func, *args, **kargs)
File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/util.py", line 238, in blockingCallFromThread
result.raiseException()
File "<string>", line 2, in raiseException
TypeError: do_unload() takes no arguments (1 given)
do_unload3(3)
unloaded PreviewChannelCommunity
gc.collect()
156
do_unload3(4)
unloaded ChannelCommunity
gc.collect()
63
do_unload3(5)
unloaded ChannelCommunity
do_unload3(5)
unloaded ChannelCommunity
do_unload3(6)
unloaded PreviewChannelCommunity
do_unload3(7)
unloaded PreviewChannelCommunity
for community in dispersy.get_communities():
print community.get_classification()
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
ChannelCommunity
ChannelCommunity
SearchCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
AllChannelCommunity
for community in dispersy.get_communities():
print community.get_classification()
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
ChannelCommunity
ChannelCommunity
PreviewChannelCommunity
SearchCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
AllChannelCommunity
do_unload3(7)
unloaded SearchCommunity
do_unload3(7)
unloaded SearchCommunity
for community in dispersy.get_communities():
print community.get_classification()
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
ChannelCommunity
ChannelCommunity
PreviewChannelCommunity
SearchCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
AllChannelCommunity
do_unload3(3)
unloaded HiddenTunnelCommunity
do_unload3(5)
unloaded ChannelCommunity
do_unload3(6)
unloaded PreviewChannelCommunity
gc.collect()
183
do_unload3(7)
unloaded PreviewChannelCommunity
do_unload3(8)
unloaded ChannelCommunity
do_unload3(9)
unloaded PreviewChannelCommunity
do_unload3(10)
unloaded PreviewChannelCommunity
do_unload3(11)
unloaded SearchCommunity
do_unload3(0)
unloaded PreviewChannelCommunity
do_unload3(1)
unloaded PreviewChannelCommunity
do_unload3(2)
unloaded PreviewChannelCommunity
do_unload3(4)
unloaded PreviewChannelCommunity
do_unload3(3)
unloaded PreviewChannelCommunity
do_unload3(5)
unloaded PreviewChannelCommunity
do_unload3(6)
unloaded PreviewChannelCommunity
do_unload3(7)
unloaded SearchCommunity
do_unload3(8)
unloaded DiscoveryCommunity
do_unload3(9)
unloaded PreviewChannelCommunity
do_unload3(10)
unloaded PreviewChannelCommunity
do_unload3(11)
unloaded PreviewChannelCommunity
for community in dispersy.get_communities():
print community.get_classification()
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
ChannelCommunity
PreviewChannelCommunity
ChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
SearchCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
AllChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
for community in dispersy.get_communities():
print community.get_classification()
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
ChannelCommunity
PreviewChannelCommunity
ChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
SearchCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
AllChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
do_unload3(49)
unloaded PreviewChannelCommunity
do_unload3(50)
unloaded AllChannelCommunity
gc.collect()
80
So then probably something else is keeping a reference to the candidates. So that would be Tribler. Is there a debug panel which has any lists of candidates?
we have a small memory leak.
4000 Tribler.dispersy.candidate.Candidate objects are in memory after 30h. 145313 Tribler.dispersy.candidate.Candidate objects are in memory after 784h.
Suspects for memory leakage. Our automatic reports with difficult 60MByte traces: http://jenkins.tribler.org/job/Test_performance_Tribler_idle_24H_nightly_devel/605/artifact/output/memprof/2180/
Elric has some tools parsing this. These can identify the possible dictionary.
Using Dowser we see the following suspects: Tribler.dispersy.candidate.Candidate Tribler.dispersy.candidate.WalkCandidate Tribler.dispersy.member.Member (probably linked to their key: Tribler.dispersy.crypto.M2CryptoPK)
Tribler shows 1108 channels, Dowser stores Tribler.Main.Utility.GuiDBTuples.Channel
At 11:00am, after running for 24hours:
8 hours later: