search

mpetazzoni / ttorrent

BitTorrent Java library with tracker and download client
http://mpetazzoni.github.com/ttorrent/
Apache License 2.0
1.38k stars 502 forks source link

Question: Upper limit to number of tracked files? #235

Closed wezell closed 5 years ago

wezell commented 5 years ago

I've been playing with the ttorrent code and as wondering the applicability to serve/share large numbers of files. Specifically, the tracker/seeder code - is there an upper limit to the number of files that it could both seed and track? I haven't dug into the code but was it designed with the intent to track large amounts of files, by which I am talking 1m files or more? Or was it not designed with targets like that in mind?

Dead-off commented 5 years ago

memory/perfomance usage on tracker side must be low. So i think tracker can track 1m files.

about seeder-side: In 1.5 version (v1,6 branch in repo) seeding many files is not possible. There was many problems, for example:

  1. seeding one torrent holds .torrent file in memory. So it can spend about 20kb memory for each torrent always (even if nobody downloads the torrent)
  2. library binds separate server socket for each torrent (https://github.com/mpetazzoni/ttorrent/issues/132)
  3. Also library creates instance of java.lang.Thread for each torrent

So in 2.0 version many of these problems were fixed. Torrents will be lazy-inited. So seeding of one file takes about 400 bytes (if nobody currently downloads the torrent). Also all incoming connections are listened on one port. In case when many peers download from the seeder different files i think bottleneck is networking. Finally we didn't test 1m files but our internal server seeds 10k files without performance problems. Also same server tracks these 10k torrents

Dead-off commented 5 years ago

if you don't have more questions you can close issue:)

wezell commented 5 years ago

Thanks for the quick and complete reply! Looks like there are a lot of nice improvements coming in 2.0.