search

mpetazzoni / ttorrent

BitTorrent Java library with tracker and download client
http://mpetazzoni.github.com/ttorrent/
Apache License 2.0
1.39k stars 502 forks source link

bt-peer threads building up #95

Open rburch314 opened 10 years ago

rburch314 commented 10 years ago

I am using the ttorrent library to distribute a file from a "controller" machine to a bunch of "worker" machines and I have recently discovered that I a building up a bunch of bt-peer threads.

To give a high-level flow of my process the "controller" takes in a job and distributes the work out to a bunch of "worker" machines. Over the course of the job the "controller" will be updating a file that at the end of the job needs to be distributed out to the "worker" machines, which is where ttorrent comes in. To perform this distribution the "controller" generates a new .torrent file and starts up 4 Clients to act as seeders for the file (it will also stop any old seeder clients from the previous job). The .torrent file is distributed to the workers and they start up a Client to download and seed the file.

With my recent testing I encountered an issue with the "controller" running out of memory and after taking a jstack I found that I had accumulated ~700 bt-peer threads after running 20 or so jobs. Here is a sample of the jstack to show the state of the bt-peer threads.

"bt-peer(..643636)-recv" #3928 prio=5 os_prio=0 tid=0x00007fe8980fa800 nid=28447 runnable [0x00007fe839e5f000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x0000000648e10a98> (a sun.nio.ch.Util$2)
    - locked <0x0000000648e10a88> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000648e10890> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.read(PeerExchange.java:276)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.run(PeerExchange.java:322)

"bt-peer(..633837)-recv" #399 prio=5 os_prio=0 tid=0x00007fe884012800 nid=24212 runnable [0x00007fe83e5a4000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x0000000648d98510> (a sun.nio.ch.Util$2)
    - locked <0x0000000648d98500> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000648d98308> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.read(PeerExchange.java:276)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.run(PeerExchange.java:322)

"bt-peer(..633837)-recv" #400 prio=5 os_prio=0 tid=0x00007fe87c7f7800 nid=24210 runnable [0x00007fe8405c4000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x0000000648d96408> (a sun.nio.ch.Util$2)
    - locked <0x0000000648d963f8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000648d96200> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.read(PeerExchange.java:276)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.run(PeerExchange.java:322)

"bt-peer(..633837)-recv" #396 prio=5 os_prio=0 tid=0x00007fe8ac015800 nid=24208 runnable [0x00007fe83e7a6000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x0000000648d93ba8> (a sun.nio.ch.Util$2)
    - locked <0x0000000648d93b98> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000648d939a0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.read(PeerExchange.java:276)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.run(PeerExchange.java:322)

"bt-peer(..633837)-recv" #394 prio=5 os_prio=0 tid=0x00007fe864045000 nid=24206 runnable [0x00007fe83ecab000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x0000000648d9b838> (a sun.nio.ch.Util$2)
    - locked <0x0000000648d9b828> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000648d9b630> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.read(PeerExchange.java:276)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.run(PeerExchange.java:322)

"bt-peer(..346266)-recv" #255 prio=5 os_prio=0 tid=0x00007fe87c6e7000 nid=23698 runnable [0x00007fe841ddc000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x0000000648cb5618> (a sun.nio.ch.Util$2)
    - locked <0x0000000648cb5608> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000648cb5410> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.read(PeerExchange.java:276)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.run(PeerExchange.java:322)

"bt-peer(..346266)-recv" #253 prio=5 os_prio=0 tid=0x00007fe88400d800 nid=23696 runnable [0x00007fe841fde000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x0000000648cff058> (a sun.nio.ch.Util$2)
    - locked <0x0000000648cff048> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000648cfee50> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.read(PeerExchange.java:276)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.run(PeerExchange.java:322)

"bt-peer(..346266)-recv" #251 prio=5 os_prio=0 tid=0x00007fe8ac07e800 nid=23694 runnable [0x00007fe8421e0000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x0000000648ccd130> (a sun.nio.ch.Util$2)
    - locked <0x0000000648ccd120> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000648cccf28> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.read(PeerExchange.java:276)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.run(PeerExchange.java:322)

"bt-peer(..346266)-recv" #247 prio=5 os_prio=0 tid=0x00007fe86401d800 nid=23690 runnable [0x00007fe8425e4000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x0000000648d04008> (a sun.nio.ch.Util$2)
    - locked <0x0000000648d03ff8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000648d03e00> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.read(PeerExchange.java:276)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.run(PeerExchange.java:322)

"bt-peer(..633932)-recv" #245 prio=5 os_prio=0 tid=0x00007fe87c081800 nid=23688 runnable [0x00007fe8427e6000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x0000000648cb14a0> (a sun.nio.ch.Util$2)
    - locked <0x0000000648cb1490> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000648cb1298> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.read(PeerExchange.java:276)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.run(PeerExchange.java:322)

"bt-peer(..633932)-recv" #243 prio=5 os_prio=0 tid=0x00007fe884009800 nid=23686 runnable [0x00007fe8429e8000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x0000000648cbd4d8> (a sun.nio.ch.Util$2)
    - locked <0x0000000648cbd4c8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000648cbd2d0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.read(PeerExchange.java:276)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.run(PeerExchange.java:322)

"bt-peer(..633932)-recv" #241 prio=5 os_prio=0 tid=0x00007fe8ac07a000 nid=23684 runnable [0x00007fe842bea000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x0000000648cc8fb8> (a sun.nio.ch.Util$2)
    - locked <0x0000000648cc8fa8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000648cc8db0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.read(PeerExchange.java:276)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.run(PeerExchange.java:322)

"bt-peer(..633932)-recv" #237 prio=5 os_prio=0 tid=0x00007fe864019800 nid=23680 runnable [0x00007fe842fee000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x0000000648d0ac98> (a sun.nio.ch.Util$2)
    - locked <0x0000000648d0ac88> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000648d0aa90> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.read(PeerExchange.java:276)
    at com.turn.ttorrent.client.peer.PeerExchange$IncomingThread.run(PeerExchange.java:322)

Now to be honest I am not sure if I am missing a step when closing down a Client, but I am suspicious that the issue is caused by the PeerExchange not interrupting the IncomingThread and OutgoingThread when it is closed. I also noticed that the Client code only closes the "connected" collection of peers and not the "peers" collection of peers so as you will see below I even tried closing those to see if it fixed my issue, but to no avail. To that end here is the code that stops the old clients and starts up new ones.

// The seeders variable is simply a Client[]

// Handle seeders
    for ( int i = 0; i < seeders.length; i++ ) {

        // Stop the old seeder and remove the torrent from the tracker
        seeders[i].stop();
        for ( SharingPeer peer : seeders[i].getPeers() )
            peer.unbind( true );

        tracker.remove( seeders[i].getTorrent() );

        // Initialize new seeder
        seeders[i] = new Client( InetAddress.getLocalHost(), SharedTorrent.fromFile( torrentFile, tempSeedDir ) );
        seeders[i].share();

    }

Would really appreciate any insight you might have to what I am doing wrong or if this is an actual issue, thanks.

alepar commented 8 years ago

Same problem here. Discovered it different way, though: my client jvm wouldn't exit after done with downloading torrent because of this lingering non-daemon bt-peer thread.

This thread hangs here: https://github.com/mpetazzoni/ttorrent/blob/fce1fc7990ee06b0b3968b7682166d0187cc78c7/core/src/main/java/com/turn/ttorrent/client/peer/PeerExchange.java#L294

Seems like that even though 'stop' flag of IncomingThread is being correctly set to true, thread never wakes up from select(), because no timeout was set. Well, I guess, unless some other peer tries to connect to mine.

c-riddell commented 8 years ago

Nice catch @alepar

zanella commented 8 years ago

189 and related to #142