Support WebTorrent Protocol

[kevincox]

WebTorrent is a new implementation of the bittorrent protocol designed to work over WebRTC. This allows browsers to download torrents without plugins. However, due to the limitations of WebRTC it isn't possible to make the protocol compatible with bittorrent (or uTP).

If libtorrent supported WebTorrent it would be able to communicate with both the existing swarm as well as the browser swarm that is restricted to WebTorrent only.

[DiegoRBaquero]

+1

[B00tLdr]

+1! This would make it possible to have something like popcorntime natively in the browser :D Maybe use something like: https://github.com/xhs/librtcdc for this. @feross could maybe add some info if this is possible/easy.

[RobinHoutevelts]

Having support for WebTorrent would be huge

[kevincox]

I would also like to reiterate that while having a good swarm of bittorrent, and a good swarm of webtorrent are both really important, libtorrent would be in the critical situation of bridging the two which is critical to the success of webtorrent.

[ghost]

Huge +1, I think Webtorrent integration is part of the future of Bittorrent; casual swarms and torrent streaming in-browser are killer apps.

edit: I always mix up the names for Bitcoin and Bittorrent, sorry!

[ngosang]

+1

[gubatron]

WebTorrent should be the one to submit a patch with the WebRTC transport implementation.

AFAIK they had to change a few things on their side bittorrent-protocol wise to make it work on browsers, who better than them to do this.

As of now, there's simply two separate networks, without libtorrent having this transport the WebRTC will never be able to access the seeds on the TCP/UDP side.

However, from a practical standpoint, will people keep a browser tab open to seed? possibly creating memleaks on their browser, slowing down their day to day work? or will this just add a massive number of seeds that will ultimately be a huge performance hit for the existing bittorrent network?

[kevincox]

However, from a practical standpoint, will people keep a browser tab open to seed?

This is hard to say, however with web technologies quickly growing there is no reason why some people wont run their torrents in the background, just like the do with native apps (no tab required). I feel the line between desktop and web is growing thin and not providing features to web apps because they are web apps is becoming an antiquated view point. You may be right, but I suspect that the playing field will change very quickly.

[transitive-bullshit]

@arvidn I'm a webtorrent contributor interested in adding support to libtorrent. Do you have any advice on where to start or potential gotchas to be aware of at the outset?

[aldenml]

@fisch0920 My suggestion is start digging peer_connection and peer_connection::start for clues of how to create a possible "webrtc_peer_connection". Do you have a WebRTC library candidate, or you want to recreate the protocol?

[yipperr]

@kevincox i agree the line between apps and browsers are going thin there is also the issue where webtorrent might negatively impact the swarm by allowing sequential file transfer of media files to be selected the relative ease for media files to be streamed into the video tag of the browser might make it appealing for people to opt for sequential downloads more often that the rarest first and also since lot of people noticed the synergies and obvious usage as this popcorn time in a browser this kind of compounds the facts and might really cause concern on swarm stability

[arvidn]

@fisch0920 aldenml's advice is right btw. the peer_connection is the base class connecting peers to the rest of the bittorrent engine. bt_peer_connection derives from it and implements the bittorrent protocol, there's also web_peer_connection and http_seed_connection for the two kinds of web seeding protocols. deriving from peer_connection to implement a new peer protocol is how it's intended to work. now, I wouldn't be surprised if there will be some rough edges, as its interface has been designed around bt_peer_connection

[obv-mikhail]

Might be possible to use this https://github.com/mozilla/webrtc-standalone for webtorrent support.

[aldenml]

Do you see this in the Makefile?

LIBS = \
$(GECKO_OBJ)/mfbt/libmfbt.a.desc \
$(GECKO_OBJ)/layout/media/webrtc/libwebrtc.a.desc \
$(GECKO_OBJ)/media/libopus/libmedia_libopus.a.desc \
$(GECKO_OBJ)/media/libvpx/libmedia_libvpx.a.desc \
$(GECKO_OBJ)/media/libjpeg/libmedia_libjpeg.a.desc \
$(GECKO_OBJ)/media/libspeex_resampler/src/libspeex.a.desc \
$(GECKO_OBJ)/netwerk/srtp/src/libnksrtp_s.a.desc \
$(GECKO_OBJ)/media/mtransport/standalone/libmtransport_standalone.a.desc \
$(GECKO_OBJ)/media/webrtc/signalingstandalone/signaling_ecc/libecc.a.desc \
$(GECKO_OBJ)/xpcom/libxpcomrt/libxpcomrt.a.desc \
$(GECKO_OBJ)/dom/media/standalone/libmedia_standalone.a.desc \
$(GECKO_OBJ)/netwerk/standalone/libnecko_standalone.a.desc \
$(GECKO_OBJ)/intl/unicharutil/util/standalone/libunicharutil_standalone.a.desc \
$(GECKO_OBJ)/security/nss/lib/pk11wrap/static/libpk11wrap_s.a.desc \
$(GECKO_OBJ)/security/nss/lib/freebl/static/libfreebl_s.a.desc \
$(GECKO_OBJ)/db/sqlite3/src/libdb_sqlite3_src.a.desc \
$(GECKO_OBJ)/memory/fallible/libfallible.a.desc \
$(GECKO_OBJ)/media/libyuv/libyuv_libyuv/libyuv.a.desc

how many MB is this standalone library?

I think a good starting point is find a minimal WebRTC communication stack, or figure out a clear interface to plug inside libtorrent as another protocol.

[jscinoz]

@aldenml https://github.com/xhs/librtcdc suggested by @B00tLdr should be a good fit - it's certainly much more lightweight than webrtc-standalone, since librtcdc it implements data channels only, whereas webrtc-standalone implements all the media related functionality of WebRTC too, which isn't necessary for WebTorrent support.

[aldenml]

yes, I saw the link right after I left my comment, thanks

[arvidn]

Just for the record. The receive buffer in peer_connection right now is a complete mess. Mostly because it provides "convenience" functions to advance read pointers and store the current message size being received. Things that are mostly useful for the bittorrent peer, but not really for anyone else. So, if anyone will be looking at integration with webRTC, it would probably make sense to simplify the receive buffer too.

[dessalines]

I'd like to help out with webRTC support, where would be a good place to start?

EDIT: Yo @aldenml , I'd like to start working on this. I have libtorrent building correctly on my machine, but I suppose I need a good way to test out changes that I've made, IE, how to run the client_test.

Also, I'm familiarizing myself with peer_connection.cpp, but I'd definitely need some help figuring out the architecture, mainly where the best place to put the new connection type should be, and how best to include the suggested webRTC library, https://github.com/xhs/librtcdc .

[dessalines]

@aldenml Sorry if this pinged you twice, I edited my comment above to ping you, but I'm not sure if github edits will re-ping someone. If there's some other client like IRC or gitter that would be better for possibly bringing me up to speed on this, let me know.

[aldenml]

not a problem (and thanks). I don't have experience with any example in libtorrent, only the tests.

[adamvoss]

@dessalines Thanks working on this! I have dug into your questions.

Running client_test

client_test is included in the repo under the /examples folder. Executing make client_test from within that directory compiles it and puts the result in the same directory; alternatively, there is an entry in 'examples/Jamfile' if your are using bjam rather than make. Then a simple ./client_test <filename.torrent> in that directory to run it.

Where to get started

arvin has advice in this comment. Paraphrasing:

peer_connection is the base. There are 3 implementations: bt_peer_connection, http_seed_connection, and web_peer_connection

You would start with a new implementation, webtorrent_peer_connection (or similar), deriving from peer_connection

by convention, it would appear your ".hpp" file would go in /include/libtorrent and your ".cpp" file in /src

Visual representation of hierarchy:

peer_connection
└─ bt_peer_connection
└─ web_connection_base
│  └─ http_seed_connection
│  └─ web_peer_connection
└─ webtorrent_peer_connection    ( <-- you add this)

How to include librtcdc

I am not familiar enough with the repository to say definitively. However, you can always get started and move the files later. For the sake of development, what I would do is just use git submodule to pull down the code at the root of the project.

git submodule add https://github.com/xhs/librtcdc.git

(I am guessing the root because I see ed25519 has its own folder at the root). Later it can be decided whether a submodule is okay or if a copy of the source will be included and where the final location should be.

[arvidn]

I think pulling in librtcdc as a submodule would probably be best. one challenge I would expect is that the peer_connection class has a pretty rigid "framework". It owns the socket, it implements rate limits, SOCKs proxies, some send- and receive- buffer management and some other high level features. The receive buffer part is not great unfortunately, it's fairly tied to BitTorrent protocol. You specify how many bytes you expect to receive and then you get called every time more bytes are received.

If librtcdc itself owns sockets, that might make it difficult (at least if you want to integrate with the main message pump/asio.io_service in libtorrent). It the library let you pass byte in and it passes bytes back out again, it should make it easier to integrate on top of libtorrent peer connections.

About the headers, I'm hoping to move towards having only public headers live in include/libtorrent and have private headers live in include/libtorrent/aux_ (and in the libtorrent::aux namespace). Some internal classes do that, but not all yet.

[ducdigital]

+1 any progress so far?

[nkosi23]

@gubatron @yipperr

or will this just add a massive number of seeds that will ultimately be a huge performance hit for the existing bittorrent network?

i agree the line between apps and browsers are going thin there is also the issue where webtorrent might negatively impact the swarm by allowing sequential file transfer of media files to be selected the relative ease for media files to be streamed into the video tag of the browser might make it appealing for people to opt for sequential download

I read the bittorent white paper a long time ago so my knowledge of the protocol is not very deep, so you really triggered my curiosity here. Why would a large number of seeds affect the performance of the network ? And what's the deal with sequential file transfers ? Could you please introduce me to the situation, or give me links me to relevant documentation (whichever is the easiest for you).

Many thanks !

[arvidn]

@reddyBell The concerns I believe were attempted to be conveyed are:

1. in-order downloading defeating rarest-first (and even piece distribution). You can google "bittorrent rareset first" for in-depth studies and descriptions. I have some old slides about this, starting on page 10: http://libtorrent.org/bittorrent.pdf

2. not accepting incoming connections. An increasing portion of peers that cannot accept incoming connections, the worse the swarm performance and robustness will be. Specifically, it may be increasingly difficult to find a peer to connect to (when flooded by peers that you can't connect to).

[kevincox]

@arvidn As far as I understand these problems aren't part of the protocol, but assumptions on how the protocol will be used. Am I understanding this correctly or are there intrsinic problems with the webtorrent protocol?

[arvidn]

I don't know enough about webtorrent or WebRTC to know if any of those are intrinsic or just general concerns.

[transitive-bullshit]

@arvidn great summary of the concerns

1. agreed that most webtorrent usage will practically be sequential downloads via media playback, but this may be offset by the fact that all webtorrent playback seeds as well as downloads for the entirety of playback as opposed to normal torrent engines where many users typically stop seeding after download completes. this is really just a hypothesis on both sides of the argument, and i honestly feel that we won't know how overall swarm health is affected until experimenting with it in practice.

2. definitely a valid concern -- the DHT scales very well, however, in terms of num peers and as long as we have some way of explicitly differentiating between webtorrent-only peers and webtorrent+bittorrent peers, then i don't seeing this being an issue in practice.

[adamvoss]

It sound like there are not insignificant changes to libtorrent to make this happen, and perhaps librtcdc would be a good placeholder implementation while making those changes. However, looking into it further, librtcdc does not seem like would be a good fit for bridging with browser-based clients at this time. Based on its issue tracker, it appears to not be very compatible with web browsers (https://github.com/xhs/librtcdc/issues/9, https://github.com/xhs/librtcdc/pull/11#issuecomment-159529822). The README indicates it is being rewritten (as of ~Feb 2017), so if enough time passes it may be the right solution after all.

[rllola]

+1

Anyone actively working on this ? I would be interested in contributing.

[arvidn]

not that I know of

[graingert]

@rllola ^

[rllola]

@graingert Yes ?

[lgrahl]

Hey guys. I'm throwing in RAWRTC here as an alternative to librtcdc. Feel free to ping me if you have any questions about it.

[zodman]

+1

[leethax666]

With Peertube getting more popular, this would be awesome

[Conobi]

+1

[abomm]

Yes, this would be very useful!

[alexandre-janniaux]

Hi, is there someone working on the issue currently ? I'd love solve this one otherwise.

[backkem]

I highly doubt it. In addition, the WebTorrent BEP (webtorrent/webtorrent#168) still isn't finished. You would have to build against the current JS WebTorrent implementation or try to get the BEP finished first.

[dessalines]

There are already several torrent clients which support webtorrent : https://github.com/webtorrent/webtorrent/issues/369

[alexandre-janniaux]

@dessalines: actually, I don't want "a torrent client which support webtorrent", but I want "libtorrent to support webtorrent" as I already have some code written with it aiming at being used this way too.

I've taken a look at librtcdc, librtcdcpp and rawrtc, and have the following feelings for now:

• rawrtc and openwebrtc should not be used : they are far too big for just supporting data channels
• librtcdcpp isn't very maintained nor has a lot of support, and has some conception flaws according to the main fork's commits
• librtcdc isn't very maintained too, except from the same guy as the librtcdcpp fork, but still is very small and should work. Maybe adding tests and build system for cross compilation would be a good idea.
• datachannel (c++) is dead and never worked.

Thus, I might go on the librtcdc option and check the BEP later.

[backkem]

@alexandre-janniaux Maybe check out this issue as well. You may be able to judge if this makes using the chromium WebRTC implementation a viable option.

[lgrahl]

@alexandre-janniaux I'm wondering why you believe RAWRTC is too big. In what way is it bigger than librtcdc? Both require usrsctp, some ICE implementation and OpenSSL for DTLS. OpenWebRTC and Google's libwebrtc are not even in the same league regarding their size and dependencies.

Edit: I explained differences in detail in https://github.com/rawrtc/rawrtc/issues/8#issuecomment-285772026.

Edit 2: In terms of LOC and complexity, you have to consider the dependencies as well. libnice is very high level whereas the ICE implementation of re/rew is pretty low-level, so you need to do a lot of stuff yourselves such as interface gathering, parsing and generating SDP, ... which is fine because more control is good. However, don't forget this kind of code is part of libnice, so it's not less complexity just because librtcdc needs to do less on its own. But you're correct that RAWRTC's data channel implementation has more lines of code. All of that code is needed and may simply not be handled within librtcdc (e.g. EOR, sending and receiving large amount of data). I don't consider having those features (actually, not handling EOR is a bug) a bad thing.

I'm really puzzled by your estimation and I'm of course happy to answer any further questions.

[alexandre-janniaux]

@lgrahl Hi, thank you for your clues, I hope you don't take my message personnally. I believe I wasn't aware enough of what was behind. I mainly focused on which library would be the easiest to use for a first implementation, libtorrent integration being the first difficult part.

But I believe that whatever the choice, the library will have to be maintained and improved so as to match webtorrent current and future needs.

I will move to your gitter so as to ask you some questions in a few days, if you agree to answer them.

[ExE-Boss]

@lgrahl Actually, Firefox has fixed EOR handling over a year ago.

Also, for latest information, see documentation on MDN.

[lgrahl]

@alexandre-janniaux No worries. And yes, please do contact me. Regarding maintaining the lib, etc., I can explain the situation in detail on Gitter as well.

@ExE-Boss Yes, I'm aware (I did that and I mostly copied RAWRTC's code for that purpose :wink:). But Firefox's data channel implementation is not available as a library. librtcdc & co however (AFAIK) have not fixed EOR.

[leethax666]

I'd also like to add that archive.org just started using webtorrent with their dweb.me decentralized web mirror, so it's catching on

[bepvte]

any result from the gitter conversation? im really hyped up about this feature and cant wait for it to happen

[lgrahl]

@redg3ar I can only speak for myself but I have not been contacted by anyone regarding this issue so far.