search

anacrolix / torrent

Full-featured BitTorrent client package and utilities
Mozilla Public License 2.0
5.49k stars 622 forks source link

Webtorrent support #138

Closed faddat closed 4 years ago

faddat commented 7 years ago

Issuehunt badges

Does this client by any chance support webtorrent?

If no, anyone know of a go torrent solution that does?

thanks!

IssueHunt Summary #### [anacrolix anacrolix](https://issuehunt.io/u/anacrolix) has been rewarded. ### Backers (Total: $250.00) - [steelbrain steelbrain](https://issuehunt.io/u/steelbrain) ($50.00) - $200.00 have been anonymously funded. ### Submitted pull Requests - [#393 Webtorrent support](https://issuehunt.io/r/anacrolix/torrent/pull/393) --- ### Tips - Checkout the [Issuehunt explorer](https://issuehunt.io/r/anacrolix/torrent/) to discover more funded issues. - Need some help from other developers? [Add your repositories](https://issuehunt.io/r/new) on IssueHunt to raise funds. --- IssueHunt has been backed by the following sponsors. [Become a sponsor](https://issuehunt.io/membership/members)
anacrolix commented 7 years ago

Do you refer to https://github.com/feross/webtorrent ? Can you link to a spec on bittorrent clients interacting with webrtc clients?

faddat commented 7 years ago

Not offhand, but I'm happy to google for it....

To leverage existing WebRTC implementations a dedicated instance of a Chromium browser is used. This 'bridge' mediates communication between Vuze and WebTorrent peers. For the original implementation see https://webtorrent.io/

....ouch.

Maybe I can find better in the node.js?

anacrolix commented 7 years ago

That's as much as I found. If you found specifics in the node.js code, that may expedite adding support for this.

faddat commented 7 years ago

@feross

feross commented 7 years ago

Hey! We'd love to help make WebTorrent support happen here. We're working on a spec. I'll share here when it's complete. At the moment you can check out the following files, to see what's going on:

To sum up what it would take to support this:

When a ws:// or wss:// tracker is detected, create a websocket connection to that server. There are a few extra announce message types to facilitate establishing webrtc connections "offer" and "answer", but everything else is essentially the same.

Once connected to a peer over WebRTC, you can treat the remote peer exactly the same -- the wire protocol is exactly the same.

faddat commented 7 years ago

Referenced in:

https://github.com/dawn-network/glogchain/issues/#10 https://github.com/dawn-network/glogchain/issues/#11

mattlepacek commented 7 years ago

Webtorrent support here would be amazing. The current node.js implementations are limited by chromes implementation of webrtc causing cpu time to multiply with each torrent being seeded. I'm interested in sponsoring Webtorrent support

anacrolix commented 7 years ago

I'll revisit this when there's a spec.

KeizerDev commented 7 years ago

Any updates on this?

anacrolix commented 7 years ago

No progress. There's no spec, and WebRTC is abysmal to work with.

Sean-Der commented 6 years ago

Hey @anacrolix !

I have been working on Go native WebRTC https://github.com/pions/webrtc and would love to get it to the point that you could use it!

It currently has full support for Audio/Video (send and recv) and I am maybe two weeks out from finishing DataChannels. We have ~5 developers involved, so things are moving pretty quick! If you are interested would love to chat/figure out what is most important to you. All the devs are on the Gophers slack in the #pion channel

anacrolix commented 6 years ago

That sounds great @Sean-Der . I'm guessing WebRTC is not so simple as a stream of bytes, when data-channels are ready we can look at integrating those. Specifically dial/accept interfaces are required, and a way to send/receive bytes. A spec on how the peer protocol operates over WebRTC is still required.

Unrelated, there was a comment earlier about websocket trackers, I've yet to see any in the wild. Some examples of those will be helpful. Implementing those should be very easy.

jimmywarting commented 6 years ago

Grate work @Sean-Der

anacrolix commented 6 years ago

Any update @Sean-Der?

Sean-Der commented 6 years ago

@jimmywarting thank you!

@anacrolix Yea! DataChannels support landed in pion-WebRTC, you can check it out here there is still plenty of work to do but 'it works'

I will track a crack at implementing it myself, there are probably bugs and would hate for you to have to deal with them. Will start a branch and drop it here

backkem commented 5 years ago

Quick update on pions/webrtc: we've finished implementing DTLS and plan to land it in master as soon as December 1st. This enables WebRTC in pure Go!

I reached out in chihaya/chihaya#388 for the tracker side of things.

Regarding the client side. It seems we need to:

  1. Introduce a new Webtorrent network type and revise any logic that assumes a hard coded amount of networks.
  2. Generalize the Peer struct so it can store Webtorrent peers alongside IP peers. The main difference is that they have a list of SDP offers instead of an IP+Port.
  3. Allow the Client to connect using these new type of peers. We have to choose between:
    • Passing these offers to the client as a address string like the current implementation.
    • Modifying the client to accept something more complex than just an address string. The Peer object itself for example.
  4. Create a WebRTC socket. The Listener is likely the most tricky part. It seems you have to open a bunch of PeerConnections and advertise their offer via the websocket tracker. TBD if we can use pions/dcnet to handle the connection itself.

After that we should be able to pipe the Bittorrent protocol over the WebRTC socket as usual.

@anacrolix what are your thoughts on this? Am I missing any big steps? How would you favor to handle the Webtorrent peers alongside regular IP-based peers?

anacrolix commented 5 years ago

@backkem It looks like you've covered all the important factors. Can you link to the WebRTC Dial/Addr/Listen equivalents so I can get a feel for how it will sit alongside the other socket types?

Vladislavik commented 5 years ago

Any news about support webrtc? It is https://github.com/gorilla/websocket That work pretty with websockets on Go

anacrolix commented 5 years ago

@Vladislavik No news on WebRTC.

jpillora commented 5 years ago

https://github.com/pion/webrtc

A pure Go implementation of the WebRTC API

Sean-Der commented 5 years ago

@jpillora I would love to see that happen :)

I wish I had the bandwidth to help, but I am always behind with pion/webrtc. If you start a PR I can answer any questions/make improvements that are need to make it happen.

jpillora commented 5 years ago

I also wish I had the bandwidth :P posting just in case we have any generous volunteers

On Mon, 5 Aug 2019 at 12:29 pm Sean DuBois notifications@github.com wrote:

@jpillora https://github.com/jpillora I would love to see that happen :)

I wish I had the bandwidth to help, but I am always behind with pion/webrtc. If you start a PR I can answer any questions/make improvements that are need to make it happen.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/anacrolix/torrent/issues/138?email_source=notifications&email_token=AAE2X473IAARJYPWTFJX5HLQC6F6ZA5CNFSM4CZIEDI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QQLNQ#issuecomment-518063542, or mute the thread https://github.com/notifications/unsubscribe-auth/AAE2X425XYVHXXG6QNERZODQC6F6ZANCNFSM4CZIEDIQ .

anacrolix commented 5 years ago

Does anyone have any numbers on the level of Webtorrent participation in the BitTorrent swarm? How about with alternative transports such as Websocket and QUIC? I could consider putting the time in to implement this myself if I know there's enough value in it.

lgrahl commented 5 years ago

WebSockets will not work in browsers as discussed in length in https://github.com/webtorrent/webtorrent/issues/1492.

QUIC is interesting but it's just a trial at the moment and I can pretty much promise that the API is going to be drastically changed. We should probably wait for WebTransport to become widely available and support peer-to-peer. But then it is definitely something we should jump on, although WebTransport is and will not be limited to QUIC. I want to emphasize this again: QUIC is only so interesting because it will be shipped with a lower-level API and not be tied into this giant RTCPeerConnection bundle everyone hates. If DTLS/SCTP would be granted the same treatment, it would be on a similar level.

When we're talking about trials, another alternative is to support Mozilla's dweb WebExtension trial which would require an extension but it's very interesting.

lgrahl commented 5 years ago

That being said, there's zero reason to not go with pion/webrtc right now since this is a Go implementation. You have no idea how privileged you are in the world of WebRTC to have this stack. Please, jump on it now. You can still try fancy new technology and APIs later.

Sean-Der commented 5 years ago

Thanks @lgrahl :)

@feross do you have any numbers around WebTorrent participation? @anacrolix I think you are in a really unique position to create the best WebTorrent client.

Everyone else is stuck doing Electron/Chromium if you got it working it would be dramatically more performance/less costly. Could make a really big impact, also the portability of Go would be a huge improvement also.

feross commented 5 years ago

WebTorrent Desktop has around 2,214,602 installs.

The browser version of webtorrent is a little bit harder to measure since folks can install it from npm and then ship it to millions of users potentially, and we'll just see that as a single installation in the npm stats. However, here's what we know:

Community members who run a WebTorrent tracker server like @DiegoRBaquero, @alxhotel, or @jakefb may have additional stats that they can share about how many peers are online at any given time, but the WebTorrent project has no information about this since we do not run tracker servers.

Sean-Der commented 5 years ago

Those are some big numbers, nice work @feross!

Is it true what I said about no light weight clients being available? Everything requires Electron/Chromium?

backkem commented 5 years ago

Yes, all existing clients listed in webtorrent/webtorrent#369 use the JavaScript Webtorrent implementation under the hood. Libtorrent would be the other major implementation but they are struggling with picking a WebRTC lib since they all have tradeoffs. I think we don't have that problem. Pion WebRTC is a clean and stabilizing implementation and it will be easy to get any issues we run into addressed. The biggest work, aside from the missing spec (which isn't that complex), will be around instrumenting the peers that have different data and behavior associated with them as I described above.

backkem commented 5 years ago

In order to kick this off I started on super bare bones port over at backkem/gowttest. It's not meant to become a client, just an example. It can currently connect to the websocket tracker, advert a WebRTC offer and receive answers (not parsed yet). With a little imagination this solves most of the missing spec problem. I'll keep working on it sporadically but I sadly don't have much spare time ATM. Feel free to ask questions, create PRs (let me know via issue/WIP PR) or take the code and run with it.

anacrolix commented 5 years ago

Thank you @backkem! This should be enough for me to go on.

backkem commented 5 years ago

Ok, On the tracker side of things. I think we need to:

Thoughts @anacrolix? Specifically:

anacrolix commented 5 years ago

Yes, all those things sound fair. I think at one point there was some weak support for connection re-use that was abandoned for simplicity. As part of redesigning the announce interface, I'd probably bring that back, so WebSocket connections can be retained, and have a callback on inbound messages. At a minimum, I'd have something like an Announce.Run(someCallback) error, that persists any connection machinery, and returns when it fails or is cancelled.

nektro commented 4 years ago

An example of a torrent thats being seeded on UDP as well as WS trackers

magnet:?xt=urn:btih:08ada5a7a6183aae1e09d831df6748d566095a10
&dn=Sintel
&tr=udp://explodie.org:6969
&tr=udp://tracker.coppersurfer.tk:6969
&tr=udp://tracker.empire-js.us:1337
&tr=udp://tracker.leechers-paradise.org:6969
&tr=udp://tracker.opentrackr.org:1337
&tr=wss://tracker.btorrent.xyz
&tr=wss://tracker.fastcast.nz
&tr=wss://tracker.openwebtorrent.com
&ws=https://webtorrent.io/torrents/
&xs=https://webtorrent.io/torrents/sintel.torrent
Sean-Der commented 4 years ago

@anacrolix What can I do to help with this coding wise or answering questions? Would funding be helpful at all?

My motivation is just getting more people using Pion. The more people using it the more bug reports and all that great stuff :)

anacrolix commented 4 years ago

@Sean-Der it's a matter of time and motivation at the moment. Funding will always help.

backkem commented 4 years ago

Libtorrent is also working on this in arvidn/libtorrent#4123. I may provide some extra inspiration.

Can we fix the labels on this? It seems neither blocked or waiting for feedback at this point.

anacrolix commented 4 years ago

A spec would still be nice, but otherwise I think everything needed is now available. Changes related to https://github.com/anacrolix/torrent/issues/356 will overlap with this somewhat.

anacrolix commented 4 years ago

I'm still happy to implement this with appropriate funding. It's several weeks of work.

vijayanandnandam commented 4 years ago

@anacrolix can you open a patreon / some other appropriate funding campaign to fund this? I am happy to contribute some, but may not be able to fund the entire effort. It will also show how many are interested in getting this implemented

nektro commented 4 years ago

@vijayanandnandam there is one: https://patreon.com/anacrolix. Albeit, I will admit there should be more tiers..

Sean-Der commented 4 years ago

@anacrolix sorry for the delay

Would you mind do something that allows single donations/has a set goal? I have a few people who have offered to pay for Pion work, and I would love to redirect them to your GoFundMe/Patreon/X!

When you have that I am going to push the Pion community towards it! I will put it on Twitter/Pin on Slack and donate myself, see what we can make happen :)

anacrolix commented 4 years ago

@vijayanandnandam I have Patreon, Liberapay, and GitHub Sponsors, you can view that stuff at the top of the project page when you click the Sponsor button.

@Sean-Der I've created https://issuehunt.io/r/anacrolix/torrent/issues/138, I think that should work? I'm open to suggestions.

Thanks to @backkem for his existing sponsorship.

Sorry everyone else for the noise.

issuehunt-oss[bot] commented 4 years ago

An anonymous user has funded $100.00 to this issue.

issuehunt-oss[bot] commented 4 years ago

An anonymous user has funded $100.00 to this issue.

Sean-Der commented 4 years ago

@anacrolix and there it is :)

If you hit any bugs/have questions we are in Slack and would love to help. I am sure you will hit some roadblocks along the way, but will do my best to unblock you!

anacrolix commented 4 years ago

@Sean-Der thanks.

issuehunt-oss[bot] commented 4 years ago

@steelbrain has funded $50.00 to this issue.

anacrolix commented 4 years ago

The Dialer and Listener interfaces have been abstracted out in #380, which is the first step toward implementing this. I've created https://github.com/anacrolix/torrent/milestone/2 to track any issues and PRs that progress this issue.

Sean-Der commented 4 years ago

Anything I can do to help @anacrolix ? Happy to get involved with any of the WebRTC stuff.