grow the swarm gradually when using dht

[asapach]

this will periodically find more peers in dht, but will not allow the queue to grow beyond the size limit

[ivantodorovich]

This will take more than 15 minutes to reach the previous setting of 10000 peers.

Have you tested its performance?

[asapach]

I've tested this on a single-core linux VM (2GHz, 2GB) in the cloud (so bandwidth is not an issue).

• It takes around 1 min at 100% CPU to find 10,000 peers. The tragedy is that with default swarm size of 100 most of them go directly into the queue, but slow the machine down to a crawl.
• With this algorithm the queue grows pretty fast with minimal CPU impact, because DHT is not the only peer source. Unlike previously, the swarm keeps growing and doesn't get stuck with the initial 10,000 nodes.

[ivantodorovich]

Sometimes DHT is the primary source because of bad trackers.

At a rate of 10 peers / second, that may or may not end up establishing a successful connection, it would take a few minutes to populate the swarm.

Maybe reducing the timer interval to ~200 ms would help in those cases.

I haven't tested it, though. I may be mistaken. But I've encountered torrents where DHT is the only real source of peers.

[asapach]

@ivantodorovich, could you run your own benchmarks and share the results?

[ivantodorovich]

I will as soon as I get home :)

Also this might be related to https://github.com/mafintosh/torrent-stream/issues/14 since this PR will also solve DHT flooding the connection.

[ivantodorovich]

I'm trying to benchmark, but I'm seeing something strange:

I added a line to debug code.

var interval = setInterval(function () {
    console.log(swarm.queued, swarm.size, table.missingPeers)
    if (swarm.queued < swarm.size && table.missingPeers === 0) {
        table.findPeers(DHT_GROW_SIZE);
    }
}, DHT_GROW_INTERVAL);

Results are as expected until it keeps printing 0 100 6 over and over again. missingPeers === 6 every time, hence not looking for new peers. There might be a problem with bittorrent-dht..

I'm using this torrent to debug, since it's known to me to have low peers on tracker:

magnet:?xt=urn:btih:6a3c1b2aa35d0d1c2698326b2cadf1f04e5d3bdd&dn=Mixology+S01E07+HDTV+x264-LOL%5Bettv%5D&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80&tr=udp%3A%2F%2Ftracker.istole.it%3A6969&tr=udp%3A%2F%2Ftracker.ccc.de%3A80&tr=udp%3A%2F%2Fopen.demonii.com%3A1337

[asapach]

I think it just means that DHT is not able to find any more nodes, so it just waits and keeps retrying.

I'm getting around 11 peers for that particular torrent currently. The queue keeps growing and then eventually saturates and drops to zero.

[ivantodorovich]

I'm getting ~10 peers too. I was able to get better results with https://github.com/feross/bittorrent-dht/pull/7 (~25)

On uTorrent, though, I get lot more peers:

Seeds: 31 of 51 connected (31 in swarm)
Peers: 4 of 61 connected (6 in swarm)

Results after 4 mins, and keeps increasing.

[mafintosh]

@asapach @ivantodorovich What is the status on this? Does this LGTM? Also @Ivshti can I get your opinion on this?

[asapach]

I think it's possible that the performance problems I was seeing might be fixed by 4a38613abdb2cb56a328d25d2a3f963b183fc288. I'd like to play with the variables and maybe as suggested by @ivantodorovich make it more aggressive. @mafintosh, could you try it as well to see what values work best for you?

[Ivshti]

@mafintosh I tested this and the performance is sometimes acceptable, sometimes unacceptable and much better with 1) the old dht.js and 2) the new update to bittorrent-dht. Outside of testing, I think this is a bad idea. DHT picks up speed gradually and gets faster and faster. If you let it run uninterrupted and unlimited for several seconds it will find thousands of peers, and on low-seeded torrents it would find a sufficient amount of peers. This modification loses that.

[asapach]

@Ivshti, how about now? I've increased the size: it will initially find 1000 peers and when the queue size drops it will find 1000 more.

[Ivshti]

This is way better. However, the DHT-related issue is now fixed and working fine.

DHT throttling is a great idea in general but I propose we keep things simple for now - in the future we can have something more sophisticated and well tested since balancing between DHT and peer connections is tricky and varies router to router, OS to OS. Sometimes it's better to stop the DHT early, sometimes it's better to keep it running and make new connections.

[asapach]

My primary scenario is a long-running torrent: I want to create a torrent-stream, leave it running for a couple of hours, download one file, leave it for a day, download another file, etc. Currently I'm seeing that the swarm size drops over time and the only way to get more peers is to destroy the torrent-stream and create another one. This should address that.