Publish a BEP explaining how to add support for WebTorrent

kumavis commented 10 years ago

from the readme:

We hope established torrent clients (uTorrent, Transmission, Vuze, etc.) will add support for WebRTC and the WebTorrent extensions so they can swarm with peers from both the normal and web networks.

Is there an existing BEP for this or should we create one?

feross commented 10 years ago

No, but I'm working on one. I'll have something to share soon. Let's use this issue to track the progress.

kumavis commented 10 years ago

Great! :smile_cat:

iadj commented 10 years ago

For reference, what's a BEP?

yipperr commented 10 years ago

http://www.bittorrent.org/beps/bep_0010.html no offence how hard is it to google.. as for any argument stemming for references the important references are there in the github readme

bbarrows commented 10 years ago

A BEP is a BitTorrent Enhancement Proposals. Checkout:

http://www.bittorrent.org/beps/bep_0000.html

For a list of "official" ones.

On Sun, Nov 2, 2014 at 5:14 PM, iadj notifications@github.com wrote:

For reference, what's a BEP?

— Reply to this email directly or view it on GitHub https://github.com/feross/webtorrent/issues/168#issuecomment-61432620.

Brad Barrows bbarrows@calpoly.edu

iadj commented 10 years ago

I see. Thank you bbarrows, appreciate it!

bbarrows commented 10 years ago

Didn't see a bunch of people had already replied. Sorry for duplicating.

My buddy at the company actually wrote an entire BitTorrent client in Javascript, check out his app https://chrome.google.com/webstore/detail/jstorrent/anhdpjpojoipgpmfanmedjghaligalgb . He ended up leaving the company and completely re wrote the client from scratch. BitTorrent, the company, was not too excited about this alternative client though. I do not believe they are not going to want to add the ability for people to Torrent without having to download traditional native clients (which the revenue stream is based on) but hopefully no one at the company has too much say over which BEPs are accepted or not.

yipperr commented 10 years ago

@bbarrows this was to be expected but bitorrent and utorrent are not the only clients that's there. there are a tons of opensource one's like deluge, qbittorrent, transmission and many more its perfectly possible to implement this as an optional plugin (depending on how these clients are written)

and the prospect of web peers and seeds that can form huge swarm of their own and can augment exisiting swarms over the client all just from a tab on a browser without additional installation is efficient and very promising

bbarrows commented 10 years ago

Ya I was just referring to those two clients. assuming they are the majority is dumb. I am sure your right in that there are a ton of users of open-source clients On Nov 3, 2014 12:01 AM, "yipperr" notifications@github.com wrote:

@bbarrows https://github.com/bbarrows this was to be expected but bitorrent and utorrent are not the only clients that's there. there are a tons of opensource one's like deluge, qbittorrent, transmission and many more its perfectly possible to implement this as an optional plugin (depending on how these clients are written)

and the prospect of web peers and seeds that can form huge swarm from a tab on browser without additional installation is efficient and very promising

— Reply to this email directly or view it on GitHub https://github.com/feross/webtorrent/issues/168#issuecomment-61448278.

eliooses commented 9 years ago

+1

wI2L commented 8 years ago

@feross Any news about this?

frandavid100 commented 8 years ago

I'm a Transmission user and I'd also like to know if theres any news regarding a BEP that Transmission can adopt.

bbarrows commented 8 years ago

Sorry, this is a pessimistic comment. I saw an email about this thread and I was thinking that one big reason that BitTorrent works is because people are giving as they are receiving right?

But to connect the WebTorrent and "regular" BitTorrent networks you would basically be asking people to proxy other peoples BitTorrenting connections through their networks. They are not getting anything in return for allowing this. Also, what if someone downloads something illegal through on of these "proxies" and the proxy owner gets in trouble for it (as their IP would be the one exposed).

Obviously it would be very attractive to connect WebTorrent with the TCP/UDP BitTorrent network but because the only option available for peer to peer connections in a browser (minus Chrome Extensions I believe) is WebRTC I just do not see any way to connect the two networks without a proxy. But maybe I am missing something?

I guess you might find some people that would be willing to run these proxy servers to connect WebRTC to TCP/UDP but they would probably end up dealing with the same issue TOR Exit node owners have to deal with.

yciabaud commented 8 years ago

Hi @bbarrows , We want some clients to add webtorrent support for communicating with webrtc peers directly, not using them to proxy anything. They will be able to share their content with both networks acting as a normal peer. The client has control about what it shares, he makes the bridge between the two networks because he is able to send blocks of he torrents he is owning on both sides.

With this solution, the whole network is not reachable from the web but we are sticking with the original architecture of bittorrent, sharing the content we need with peers on both sides.

Does it answers your concerns or have I misunderstood something?

bbarrows commented 8 years ago

I guess I assumed to much from looking at the diagram at https://github.com/feross/webtorrent it looks like those arrows are saying that BitTorrent traffic travels through the hybrid to the other WebTorrent clients (which would make the "hybrid" a proxy and it wouldn't make any sense to proxy your own traffic as far as I can see).

For a regular BitTorrent client to find WebTorrent peers I see a site saying a BEP for trackers is "forthcoming" and I imagine there is also some peer discovery DHT work going on. I was going to say that regardless if those become accepted and implemented wouldn't you always prefer to be running on the BitTorrent network? I was thinking A TCP/UDP client could communicate with peers from both the regular BitTorrent and WebTorrent network but from my very very limited knowledge of WebRTC it sounded like a WebTorrent client would be stuck in its own network as far finding peers itself?

But I guess if the BitTorrent clients have something like a WebRTC server running then WebTorrent clients could communicate? I'll have to read through some implementations/docs some more.

Thanks for taking your time and responding. Really nice to think that a peer to peer network without a separate client (other than a browser) would be possible.

yciabaud commented 8 years ago

Yeah I can understand this mistake, the diagram often leads to questions regarding hybrid clients...

Here is a new diagram I made to explain the tracker side of the process: untitled 1

Talking a bit about WebRTC, we cannot just connect to a peer with IP and port. To establish a connection we need a way to exchange SDP offers and answers to make some kind of handshake before we can actually connect. Therefore we chose to add a RTC signalling feature to our bittorrent tracker in order to allow WebRTC peers to connect to each other.

This signalling system relies on a websocket transport in order to be able to forward the messages between peers without polling the tracker.

Due to the way WebRTC works, we did not made the effort to bring DHT in the browser since we will still need a central signalling server and the browser cannot handle many WebRTC connections efficiently ATM (But the discussion is still open #288).

To summarize, the tracker BEP should cover:

Websocket transport
Signalling message handling
WebRTC peer management

What do you think about this solution? Can you give us some advice on the BEP redaction process? I don't know exactly what @feross has made so far but I am interested in learning how we can handle this.

yciabaud commented 8 years ago

The main question I have is: Do we need to split this in different BEPs or not? Then we will need to create a rst file and push it to bittorrent/bittorrent.org

Do you think we are ready to publish a BEP right now @bbarrows ?

bbarrows commented 8 years ago

It looks to me like this would require a very, very large BEP possibly. But since a lot of this is using existing protocols maybe those parts would be abstracted from the BEP? Sorry, I really have no idea.

For example, a relatively simple addition to the protocol required this much documentation: http://www.bittorrent.org/beps/bep_0015.html

To get started I would imagine having a working example environment/network would be very helpful.

This page details the BEP process http://www.bittorrent.org/beps/bep_0001.html

I would imagine that if the BEP editors believe the WebTorrent network/swarms provided a large enough benefit (increase in peers with low enough overhead) they would be interested in discussing it in the BEP forum which sounds like the first stage to getting a BEP accepted. I personally have never tried to submit anything so I wouldn't be a very good resource.

I'll message one of the "BEP editors" on FB (I dont think he is on GitHub) and see what he thinks/link him to this.

yciabaud commented 8 years ago

Thank you, I believe it should be easier to separate websocket transport from WebRTC signalling, I will try to write a draft of those BEPs and reach the BEP editors to have their feedback.

@feross Is it OK for you if I try to make the process going further? What is the status on your side? Have you already discussed about this with the BEP editors?

feross commented 8 years ago

Yeah, they're open to a BEP. I'm in contact with folks at BitTorrent, Inc. Happy to have you get started on a BEP, @yciabaud. I can help out once you get it going. Sorry for the huge delay. There's been so much going on across all the webtorrent repos, it's hard to keep up.

yciabaud commented 8 years ago

Yeah sure I can see all the issues you are managing on the repo I am involved in. I am trying to help you when I can but there are so many of them!

I am starting to write 2 BEPs:

Websocket tracker
WebRTC peer connection

I will push it here first to have your opinion. Keep going on your hard work!

feross commented 8 years ago

@yciabaud I don't think there's a need for a WebRTC peer connection BEP. Once a connection gets established, the wire protocol is 100% identical. Unless I'm missing something, I think we can cover this in a single BEP. What were you envisioning?

WebRTC is already specced out by the IETF (wire protocol) and W3C (API).

yciabaud commented 8 years ago

@feross the webrtc bep covers signalling in the tracker, I guess we can merge this but beps are small units and websocket tracker can live on its own. I will ask the bep editors to know what they think.

feross commented 8 years ago

WebSocket tracker and WebRTC signaling are the same thing. I created WebSocket trackers in order to enable WebRTC signaling. I think there should be a single BEP.

yciabaud commented 8 years ago

I think websocket can be implemented without signalling, that was my point but one bep should be easier to handle so I will make only one and see how it goes.

yciabaud commented 8 years ago

OK here is a first draft, I made a new branch in my fork do we create a branch in this repo to discuss on a PR? BEP v0: https://github.com/yciabaud/webtorrent/blob/beps/bep_webrtc.rst

yciabaud commented 8 years ago

I made a PR to discuss the details of the protocol #881.

wI2L commented 8 years ago

The BEP looks good so far, but I regret the lack of extension for some sort of authentication with an announce request. HTTP trackers usually have a request path containing a passkey and there is the BEP 41 for UDP tracker.

Adding an extra authfield within the announce request object, or a flags array that could contain fields for future extensions could be considered.

@feross @yciabaud Any thoughts ?

yciabaud commented 8 years ago

Thanks @wI2L for the review, it must not be clear enough but in the end of the document we are saying that the payloads can be extended with extra properties to deal with this kind of situations.

Do you think it is enough to cover the authentication cases? How would you have written it for better understanding?

wI2L commented 8 years ago

@yciabaud It was my bad, I didn't noticed the last section, but I think this should be enough for future extensions (eventually as I said an extra array field yet to be named could be envisaged to gather all the extra fields). Private trackers actually using HTTP(s) could simply get the user passkey from the websocket tracker url ws://myserver.com/<passkey>.

yciabaud commented 8 years ago

@wI2L no they cannot use a path for the websocket endpoint, they would have to add an extra param passkey to the data structure.

wI2L commented 8 years ago

@feross @yciabaud Any chance we could discuss of using a different format for the payloads encoding ? Announce interval for webclients is very low in comparison with standard tracker, not to mention the signaling payloads the tracker has to forward, it result in a tracker that must deal with a lot of payloads. JSON has its benefits, but a fast and efficient binary serialization format seems to fit the webtorrent use case to me (MessagePack, Protobuf).

wI2L commented 8 years ago

For reference : https://github.com/yciabaud/webtorrent/pull/1 Changes we made with @yciabaud I tried to be verbose in payload fields description. Suggestions are welcome. I am fully dedicated to see a full BEP coming out asap so I/we can start implementing more trackers and clients for webtorrent.

wI2L commented 7 years ago

UP @feross @yciabaud Can we make this active again ? Implementing this protocol in my Go tracker was very messy. One unexpected behaviour was about the binary fields, info_hash and the offer_id fields. I had to use some tricky things to avoid it being converted to utf8 back and forth. Using instead hexadecimal for those fields would be alot easier.

gitcoinbot commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 0.2 ETH (112.13 USD @ $560.65/ETH) attached to it.

If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
Questions? Checkout Gitcoin Help or the Gitcoin Slack
$28,747.86 more funded OSS Work available on the Gitcoin Issue Explorer

writeprovidence commented 6 years ago

hello @backkem i would need some more insight on how to publish this article. will it be publish on medium? pls am waiting for your reply. thanks.

writeprovidence commented 6 years ago

Or should i publish it here on github?

wI2L commented 6 years ago

Send a PR, this sound the easiest way, it will allow contributors and others persons with interest about this to discuss about it.

Le lun. 6 août 2018 à 23:09, writeprovidence notifications@github.com a écrit :

Or should i publish it here on github?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/webtorrent/webtorrent/issues/168#issuecomment-410853792, or mute the thread https://github.com/notifications/unsubscribe-auth/AGN7EXz8sYAkmKj1yic4EITO3C1RDp5Dks5uOLCTgaJpZM4Cz9l1 .

writeprovidence commented 6 years ago

ok@wl2l

wI2L commented 6 years ago

You can look at the document I started on my branch last year.

Le jeu. 9 août 2018 à 23:19, Gitcoin.co Bot notifications@github.com a écrit :

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 3 months, 4 weeks from now. Please review their action plans below:

1.

writeprovidence https://gitcoin.co/profile/writeprovidence has started work.

i still want to try and figure things out on this .

Learn more on the Gitcoin Issue Details page https://gitcoin.co/issue/webtorrent/webtorrent/168/598.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/webtorrent/webtorrent/issues/168#issuecomment-411900989, or mute the thread https://github.com/notifications/unsubscribe-auth/AGN7EdLdEdQxYsFQp4QZehWkD9ps3LkGks5uPKd3gaJpZM4Cz9l1 .

writeprovidence commented 6 years ago

pls can i have a link to the document?

On Thu, Aug 9, 2018 at 9:31 PM, William Poussier notifications@github.com wrote:

You can look at the document I started on my branch last year.

Le jeu. 9 août 2018 à 23:19, Gitcoin.co Bot notifications@github.com a écrit :

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 3 months, 4 weeks from now. Please review their action plans below:

1.

writeprovidence https://gitcoin.co/profile/writeprovidence has started work.

i still want to try and figure things out on this .

Learn more on the Gitcoin Issue Details page https://gitcoin.co/issue/webtorrent/webtorrent/168/598.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/webtorrent/webtorrent/issues/ 168#issuecomment-411900989, or mute the thread https://github.com/notifications/unsubscribe-auth/ AGN7EdLdEdQxYsFQp4QZehWkD9ps3LkGks5uPKd3gaJpZM4Cz9l1

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/webtorrent/webtorrent/issues/168#issuecomment-411903941, or mute the thread https://github.com/notifications/unsubscribe-auth/Ad6C1bbWaGlxg0i6YN4Vr1aBtRvi5SIzks5uPKoXgaJpZM4Cz9l1 .

gitcoinbot commented 6 years ago

@deguvunor Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

[x] warning (3 days)
[ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 6 years ago

@deguvunor Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!