search

deoxxa / bittorrent.js

BitTorrent implementation in JavaScript
Other
88 stars 11 forks source link

Hello! #3

Open deoxxa opened 10 years ago

deoxxa commented 10 years ago

This is a mail that I sent to a bunch of cool people, now in github form for easy following!

Hello!

You’re probably wondering why I gathered you here today. As some of you might
have seen already, I have the bittorrent package on npm. If you have seen it,
you’ve probably also seen that it’s unlikely it’ll ever quite be in a state
where it’s worthy of its title.

Anyway, on with things! The short of this email is that I’d very much like it if
one day a user could `npm install bittorrent’ and get a high-quality,
some-batteries-included, modular bittorrent library that works out of the box,
but I’m not personally capable of putting together such a system. I’d like to
know if any of you (or indeed any other people some of you might know) would be
interested in forming a loose team of sorts to achieve this goal.

If you’re interested, please feel free to reply or drop a comment at
https://github.com/deoxxa/bittorrent.js/issues/3. If not, just ignore this mail
and there’ll be no hard feelings. If you know anyone else who might be
interested, forward this mail along or point them to the github issue so they
can see what’s up :)

Regards,
Conrad
themasch commented 10 years ago

Always a pleasure working with you, count me in!

deoxxa commented 10 years ago

Great to hear!

On Saturday, April 26, 2014, Mark Schmale notifications@github.com wrote:

Always a pleasure working with you, count me in!

— Reply to this email directly or view it on GitHubhttps://github.com/deoxxa/bittorrent.js/issues/3#issuecomment-41464965 .

themasch commented 10 years ago

So what's the plan? Do you think about creating a whole client, something like deluge? Or just a generic, reusable library?

deoxxa commented 10 years ago

Well, I'm seeing a lot of duplicated work in the current crop of BitTorrent libraries on npm. I understand that different projects have different requirements, and sometimes it might make sense to have a little bit of duplication (especially if it creates choice), but I'd like it if there was less wasted effort around BitTorrent stuff in node.

BitTorrent is a deceptively simple protocol, in that you can have an operational parser and serialiser in just a little bit of code, but implementing all the other bits (like interaction with trackers, dht, peers/scheduling, traffic limiting, etc) is a huge amount of work. It'd make me very happy if people were working together on this kind of thing and could take care of separate, smaller pieces, rather than the current situation where there are several "works for what I wanted" implementations kicking about.

I'm not sure if the solution will end up being a complete client, but I think that the solution should definitely lend itself to the easy (or at least easier than now) construction of such a thing.

themasch commented 10 years ago

I'm absolutely with you on this. There are at least 6 bencode libraries I'm aware of, some of them are maintained, most don't seem to be. All of then differ in api, results, following the spec and performance.

I've no problem with diversity and like to have the choice, but it's some what hard to pick the right package if there are 6 quite different packages for one so simple problem.

As some might know I'm one of the maintainers of the 'bencode' package and I'm pretty happy with performance and api but that's not what counts for a public reusable package. What this leads to is that I really think that every single open source project needs a lot of contributors which don't code but just give insight in what they need and what they expect from the library. Software that's only used and developed by a hand full of people tends to solve only the problems of these people while it could work for many more people.

TL;DR: we need more opinions ;)

jhermsmeier commented 10 years ago

I'm keen, count me in. I'll write more later.

deoxxa commented 10 years ago

Awesome stuff!

mafintosh commented 10 years ago

There is a lot of excellent bittorrent stuff on npm - almost all of which is heavily modularized.

It would be nice if the goal of bittorrent was to be extremely lightweight and just combine a lot of existing node_modules into a easy to use bittorent client for those who just want a regular torrent client in node.

If something is missing or could be done better lets make a seperate module for that problem and everyone benefits (including all of mine and @feross' existing bittorrent stuff).

deoxxa commented 10 years ago

I agree about the volume of BitTorrent-related stuff on npm, it's great to see that so many people are interested (and motivated) to work on it. The thing that worries me is the massive amount of overlap in some of the packages. I'm guilty of this with my bencode-stream package - I probably should have contacted the maintainers of an existing package and worked with them to figure out the best way to add the features I needed.

EDIT: I also agree with you about what this module should be - I'd love it if this could almost just be a metapackage, pulling in the best options for all the different components of a full BitTorrent stack.

Just for illustrative purposes, here are some quick, probably inaccurate summaries of bittorrent-related things on npm.

bencode libraries

Name Sync Async Streaming Binary Keys Fast
bencode ⭕ ❌ ❌ ❌ ⭕
dht-bencode ⭕ ❌ ❌ ❌ ?
bencoding ⭕ ❌ ❌ ⭕ ⭕
bencode-js ⭕ ❌ ❌ ❌ ?
bencode-stream ❌ ⭕ ⭕ ⭕ ❌
libbencode ⭕ ❌ ❌ ❌ ?
node-bencode ⭕ ❌ ❌ ❌ ?

wire protocol libraries

full clients

I can see the sense in having options, but I think that there's a lot of work being done twice (or more) with all this duplication. I can't see why everyone should be reimplementing bencoding and the wire protocol for every BitTorrent-related project.

The question that this raises for me is "why?". Why are people implementing these parts over and over? Are the existing implementations not flexible enough? Are their codebases too hard to approach? Is it simply a case of NIH on the part of everyone who's written BitTorrent modules for node?

jhermsmeier commented 10 years ago

The question that this raises for me is "why?". Why are people implementing these parts over and over? Are the existing implementations not flexible enough? Are their codebases too hard to approach? Is it simply a case of NIH on the part of everyone who's written BitTorrent modules for node?

As @themasch already noted, I think most of the time it's a combination of lack of documentation, functionality, performance, maintenance and / or the issue with finding a module for a specific purpose in the first place. Maybe even that interop between some modules would require more work, e.g. their APIs aren't really pluggable like lego blocks. Sometimes it's just for the sake learning about a certain problem, spec or implementation (for example, I just love to implement protocols).

I’d very much like it if one day a user could `npm install bittorrent’ and get a high-quality, some-batteries-included, modular bittorrent library that works out of the box

Sounds like it might be a good idea to put all small parts and the bittorrent module under an organisation on GitHub, and have bittorrent be a some form of meta package that ties all the small parts / modules (living under the org) together. I guess we could just start by asking the authors of the respective packages to transfer them to the org to ensure maintenance and further development or something like that.

Considering that there are 318 repositories in JavaScript somehow related to "torrent" at the moment, it might be a lot to sift through though.

deoxxa commented 10 years ago

Time to get started, I guess...

I'm going to go through the list and try to see just how much duplicated work there is, and if there are any commonalities that could be used to specify some sort of an interface for different components.

I'm ticking things off that aren't library-style projects, or that aren't related to BitTorrent.

deoxxa commented 10 years ago

A cursory examination of what's on github reveals that the most popular projects are mostly polarised between @feross's WebTorrent and @mafintosh's peerflix libraries. Most third-party library consumers seem to be using peerflix (or torrent-stream).

Less-popular projects seem to go down the route of implementing everything themselves.

themasch commented 10 years ago

@deoxxa for some reason your chart doesn't render properly on chrome, but thats another topic. And, the "bencode" package should indeed support "binary keys". If it doesn't, our testsuite sucks and we need to fix that ;).

On to important things: I really like the org-for-package-maintaining thing. The problem might be on how to pick which package you include in the org. I think its nonsens taking 5 bencode-libs in the org, same goes for wire-proto, DHT. Thats gonna be a tough process but as I see @deoxxa already started work. Nice.

deoxxa commented 10 years ago

I could go either way on the idea of an organisation. It would be great to have a set of "blessed" modules that are endorsed in a way, and that maybe had a common set of contributors, but I think there would have to be a huge emphasis on making it very easy for anyone to join in on the fun. I'd like to let everyone (who is interested) weigh in on what they think over the next week or so before I do anything in particular.

P.S. I've opened an issue on themasch/node-bencode for the binary key thing, it's a common problem and was part of the reason I wrote my own parser (along with having to parse some abnormally large files).

feross commented 10 years ago

@deoxxa Thanks for creating this issue and starting this discussion.

I actually think it's great that there are multiple implementations of the same bittorrent modules on npm! Having multiple robust solutions to the same problem tackled from different angles is great for exploring the problem space and discovering novel solutions. For example, it's great that @mafintosh felt free to write peerflix and explore sequential fetching, even if the client lacks some features of other clients. Now we know this can work, and future clients are more likely to support this feature.

Eventually, if there's a winner, it will happen naturally because a module solves the problem well with an API that people like. Naturally, pull requests will go to this module and it will increase in robustness and become a more obvious choice to newbies. No need to pick winners in advance, though it's fun to guess which modules are possible contenders.

Both my webtorrent and @mafintosh's peerflix have been using the following:

So they're pretty robust and have received many PRs and bugfixes. Beyond these, the rest of @mafintosh's modules are also quite robust, given the popularity of peerflix.

But that shouldn't stop people from writing competitors, imo. For example, I wrote bittorrent-protocol because I wanted to learn, and because I wanted to support an extension API so that extensions like ut_metadata could be supported without putting all the code in the main module.

Looking forward to working with you all!

the-eater commented 10 years ago

To explain why I started my "node-torrent" is basicly a year ago I couldn't find any usable Torrent client and started my own, I horribly failed (see EaterOfCode/EaTor). so a year later I came back to the subject and see how much I've learned by doing it again, with success I must say. and when I started I've not found any library that suited my needs.

also shameless promotion for mafintosh/utp if you guys want to build a good bittorrent client, you should definitely look into that repo