search

cccc / media-backend

DEPRECATED backend of media.ccc.de
2 stars 0 forks source link

Torrents: smaller piece lengths #8

Closed astro closed 9 years ago

astro commented 9 years ago

I noticed that 31C3 video torrents have a piece length of 20 MB. The SHA1 hash size is 20 bytes per piece. For a sample 300 MB video that produces just 15 pieces or 300 bytes of hashes, which is well below standard MTUs.

I'd say it is a good idea to choose a piece length proportional to the file size so that you'll stay within 3000 pieces or 60 KB of hashes.

My use-case: with peerflix/popcorntime/bitford/webtorrent there are now Torrent clients that support streaming playback while the download is still in progress. To avoid passing potentially invalid data to the playback frontend these clients will wait for download of a full piece. The time this takes is proportional to the piece size. Hence, larger piece size results in a longer time to wait when starting playback or seeking in such clients/players. At 20 MB this is a substantial delay.

If this is not of high priority you can as well point me to the appropriate code location and I may produce a pull request myself.

Thanks for all the nice recordings and for media.ccc.de, Astro

astro commented 9 years ago

(For the record: I don't think it makes sense to regenerate old Torrents as it will fragment existing swarms. It would be great for future releases.)

manno commented 9 years ago

Hi astro. The torrents are generated by mirrorbrain and contain geo-located webseeds. Take a look at this http://svn.mirrorbrain.org/viewvc/mirrorbrain/trunk/mb/mb/hashes.py?view=markup

I deliberately regenerated all the torrents with a very large chunk size. Mirrorbrain used 256kb chunks by default. Sadly most torrent clients disregard the warning in BEP-19 ("A Not Recommended Client Implementation") and request 256kb chunks from webservers. Up to 50 times a second. This was a huge problem, even for some of our bigger mirrors since the random disk i/o was significant.

Maybe 2mb chunks would be better. However mirrorbrain uses the same configured chunk size for all files.

saerdnaer commented 9 years ago

Whats the current state of this issue?

manno commented 9 years ago

Soon I'm going to replace mirrorbrain with another solution that won't support torrents. No more webseeds = no more chunk size problems.

astro commented 9 years ago

-v