Bugout in the browser takes minutes, or several page refreshes before connecting to my own tracker servers

[draeder]

Inspired by your project webrtc-signaling-mesh, I created a library called tracker-swarm. It uses my fork of fake-bittorrent-tracker (forked for issues with a dependency => waiting on a PR) to test the tracker-swarm nodes then share them with the swarm if they work. It then tests all of the trackers every 60 seconds and removes non-working trackers from list of trackers shared among the tracker-swarm.

I spun up a couple of tracker-swarm nodes inside of Heroku to test things out. Tracker-swarm is working pretty well. But when I try connecting to Bugout using those tracker-swarm nodes, sometimes it takes minutes or several browser page refreshes to get a connection. I remember when I first started playing with Bugout, this was a common experience in my testing, with or without my own tracker servers.

One thing to note is that I'm testing in the same browser on duplicate tabs. I have not tried testing between different networks, and maybe that is a differentiator for this issue. I've been testing with Chrome @ latest. On that note, Safari @ latest pretty much never works with Bugout. I can create a separate issue for that if you want.

It's also notable that If I finally get a connection between peers, but move away from those browser tabs for some period of time, then go back and refresh one or the other, the problem reoccurs.

The issue might be that there are some delays with WebTorrent getting the torrent? If so, that might be outside of Bugout. But if not, Bugout may not be doing something in a timely fashion?

Any thoughts, ideas, fixes about how to quicken the Bugout peer connection experience would be awesome!

[draeder]

I think this is an issue with WebTorrent itself. Tonight, everything is snappy. And I think that's because it's a low volume time on the WebTorrent network.

[draeder]

I'm closing this .. as I realized I had some things set up wrong with my test instances of bugout. Sorry for the clutter in your inbox.

[chr15m]

@draeder all good, sorry it took me so long to reply!

[chr15m]

@draeder one thing I was curious about is what happens if two different Bugout instances are given a completely different set of trackers from eachother from the swarm? Isn't it the case that they would not be able to find each other?

[draeder]

@chr15m Right now apps get all of the trackers in the swarm, but can only use those that share the same application name. Although I need to improve how this works. I've been thinking about how to make it more sane. The only answer I keep coming back to is using DHT (which isn't trivial, at least for me that is until I just looked at bitorrent-dht again; I have some ideas). Then, that gets me thinking about DHT in the browser, which I'm kind of obsessed with.

[draeder]

@chr15m Using DHT with tracker-swarm and Bugout (or any browser based p2p app, for that matter) is still a bit of a mystery to me from the perspective of how to scale it.

I rewrote tracker-swarm and have it working better now. Check out the readme and please let me know if you have any feedback.

The problem with what I've built is that the array of trackers for a swarm ID or app ID could potentially scale to infinity. (What is the theoretical or practical limit to a webtorrent URI length, anyway?)

Gathering a (random) limited set of trackers from a swarm ID and app ID makes a lot of sense. But as you stated, how would the browsers be able to talk if they get a different set of trackers from the initiating peer? They can just hope that the set of trackers they get shares the signaling.

I thought I had a flash of brilliance for a moment there, by making the swarm node announce the list of trackers on DHT. But then was quickly deflated when I remembered I can only announce 1000 bytes of arbitrary data. And even with that, how would I tie together the WebRTC SDP from the originating peer to the other trackers, and pass the WebSocket through? Basically, I would be rewriting WebTorrent at that point.

Of course, the whole goal here is to eliminate central servers altogether. My effort with tracker-swarm is to let a regular Joe spin up a tracker swarm node wherever they want so they can use libraries like Bugout more effortlessly. By democratizing tracker servers, I think it helps. It's not perfect, though.

[draeder]

@chr15m Thinking this through some more, I think this approach will work for tracker discovery. It's a kind of half-baked DHT, but I think it will work without too much overhead since there are far less trackers than there would be peers:

1.) When the first peer requests a set of trackers to announce to, it includes its appId and peer address with the request. A random set of n trackers is returned. Each swarm node in that set stores the appId when its tracker instance receives a start message from the peer address. This increases its peer count for that appId by 1.

2.) When a new peer with the same appId requests a set of trackers to announce to, a lookup message is sent to the tracker-swarm to return the nodes that have that appId. If the responding nodes have peers for that appId, it gets added to the returned set of trackers up to n trackers. If there are less than n trackers, random trackers are added to the returned set to fill the gap, which causes 1.) to repeat for those new random trackers.

3.) When a node's tracker instance receives a disconnect message, the node's peer count for that appId is decreased by 1. If the appId has no peers, the node drops the appId altogether.

This should allow for an efficient tracker lookup scheme in the tracker swarm. Since tracker-swarm tests trackers every time a new node joins or leaves the swarm as well as every 60 seconds, overloaded/non-working trackers should never be in the set of trackers a peer gets. And, since the set of n will add new random trackers to fill the gap, peers should, in theory, always have trackers that share connections with other peers, and always have working trackers to announce to.