search

webtorrent / webtorrent-hybrid

WebTorrent (with WebRTC support in Node.js)
https://webtorrent.io
MIT License
519 stars 98 forks source link

downloading only 1 torrent #116

Closed tudor-pop closed 2 years ago

tudor-pop commented 4 years ago

Related to: https://github.com/webtorrent/bittorrent-tracker/issues/199 https://github.com/webtorrent/webtorrent/issues/1349

Latest version ^4.0.1

OS: macOS Catalina 10.15.4, nodejs 12.16.3, npm 6.14.5

I'm trying to do browser -> server upload but after the first webtorrent.add(magnetUri, t=>{..}) it just halts. This weekend I was trying to find out what could be the issue and here are my findings Setup:

webtorrent-hybrid/examples/node-download.js

const WebTorrent = require('../index')
const client = new WebTorrent({
    dht: false,
    tracker: {
        announce: ["ws://localhost:8765","http://localhost:8765/announce","udp://localhost:8765"]
    }
})
const uris = [
    'magnet:?xt=urn:btih:106e92bba52ebe993717971c02fd251681f641e4&dn=website_ad.json&tr=ws%3A%2F%2Flocalhost%3A8765&tr=http%3A%2F%2Flocalhost%3A8765%2Fannounce&tr=udp%3A%2F%2Flocalhost%3A8765'
      ,
    'magnet:?xt=urn:btih:89ce5fa31606e833ab3606c5a79de914588335cd&dn=maymounkov2002kademlia.pdf&tr=ws%3A%2F%2Flocalhost%3A8765&tr=http%3A%2F%2Flocalhost%3A8765%2Fannounce&tr=udp%3A%2F%2Flocalhost%3A8765'
]

console.log('uris:\t', uris)
function add(uri) {
    client.add(uri, function (torrent) {
        const files = torrent.files
        files.forEach(file => {
                console.log('file:\t\t', file.name)
        })
    })
}
for (let it of uris)
    add(it)

JS code run in browser:

        const trackers = process.env.REACT_APP_WS.split(',');
    window.client = new WebTorrent({
        dht: false,
        tracker: {
            announce: trackers
        }
    });
        let opts = {
        name: this.generateTorrentName(files),
        // store: ChunkStore,
        announce: trackers,
        // private: true
    };
    console.log("hashing files", files);
    this.client.seed(files, opts, (torrent) => {
        console.log('Client is seeding ', torrent);
        let fileHashes = [];
        for (let file of torrent.files) {
            fileHashes.push({
                name: file.name,
                length: file.length,
                hash: torrent.magnetURI
            });
        }
        this.torrent = torrent;
        resolve({fileHashes});

    });

Update 1 it seems that when using an array of torrents the function onConnect is not getting called inside peer.js(which should mean that the signaling data is not getting through somehow or is not being correctly set...I also removed the parallel package to run it sync just to be sure and once it worked fine! <3)

Update 2 It seems that on my machine the problem is with the number of generated offers. By default there are 10 but it hangs when this line is reached. If I change that to 1..8 it seems to work fine & it downloads all my browsers seeded torrents. If I set that to 15 it just crashes with exit code 139 (interrupted by signal 11: SIGSEGV). There is a comment that says the offer generation can be slow but in my case it just hangs. I will try to find out why... such a strange issue...

Update 3 Issue is creating too many peers const peer = new Peer(opts) from simple-peer module. Anything over 15 peers will cause a nodejs sigfault. Tried it on Node 10/12/14. 10 peers will hang, 9 will sometimes work and anything below always works

Update 4 So I the limitation is with Bitorrent-tracker....I would need your advice @feross since you put this in place in 2015. Don't know if you remember but if you do, can you clarify why are we creating 10 peers to connect to 1 single peer when going through WSTracker? from these lines Math.min(50,10) and 10 or 50 peers. Having 10-50 offerId -> Peer() seems wasteful because 1 nodejs host tries to start 10 peer connections to a browser to download the same content... only 1 peer with 1 offerId seems more logical. The same is happening reverse, 1 browser tries to start 10 connections to the other peer. What do you think? Another issue is that if I call .add(torrentURI) only once and set that number(DEFAULT_ANNOUNCE_PEERS) to 50 (default value) then again node will crash with sigfault. This also means that it can't scale beyond 50 .add calls on a backend within the same process :(

KilianKilmister commented 4 years ago

Oof. this is above my expertise. It's to much work to set up an environment for checking it out myself, but i dug through all the relevant code and i can't i can't really add anything to this conversation. All i can do is suggest trying nodes child_module.fork() to easily create a subprocess. This should help with the limit-issue.

You could also try to iterate over the array asyncronously. this might do something:

async function add(...args) { // dont forget the spread operator on calling this `add(...uris)`
  for await (const uri of args) {
    client.add(uri, function (torrent) {
      const files = torrent.files
      files.forEach(file => {
    console.log('file:\t\t', file.name)
      })
    })
  }
}

Just a shot n the dark, but worth a try.

But again, this is to high for me. I can just about use a unix-socket. I've done very little with web-technology but i'll keep an eye on this as i planned to use it in the future

tudor-pop commented 4 years ago

nope that doesn't work. A child process might solve the issue but it won't scale unfortunately

sergethompson commented 3 years ago

I am definitely running into the same problem. I think this technology is so cool. It would be very cool if this could be fixed!

github-actions[bot] commented 2 years ago

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?