TEST: Evaluating BitTorrent

[evaristoc]

Some points:

• BitTorrent was previously used for a dataset, specifically academictorrents: http://academictorrents.com/details/030b10dad0846b5aecc3905692890fb02404adbf
  • Currently cited at https://github.com/caesar0301/awesome-public-datasets#education

My personal views:

• If I remember correctly BitTorrent failed to be an easy way to download the data - we had to use your private DropBox account to get access to the data at the end. This was the only experienced I had with BitTorrent so it can't be generalized.
• BitTorrent seemed hard to debug. This might not favour the use of the data. The external storage should be much easier to access and debug.
• BitTorrent is a general place to store big files and won't impose limits to the sizes (which are possible with specialized data storages), however differently to the data-specialized platforms, BitTorrent won't provide additional functionality for data descriptions, internal search engines, capabilities to organize the information, or graphical power, to mention some.
• Specialized storages would expose the datasets to a larger audience, something that BitTorrent won't do.
• Those data storages seems to appeal more the data users than BitTorrent. I have for example mentioned datadotworld in the DSR and people found the site very interesting. There is also different communities of users growing around them.
• Requires the installation of a Torrent downloader.

However my views do not necessarily exclude the use of BitTorrent as option.

[evaristoc]

Several people have been asked to try out the link to the file in BitTorrent. Contributors so far who also tested the dataset link:

• @timjavins (tested weekend, uncompleted test arguing there were not enough seeders and would take too long; EDIT: eventually the download was completed)
• @mcbarlowe (tested weekend)
• @Mehbub_Basha_twitter (Gitter fCC Contributors, tested weekend, India)
• @jp-sauve (Gitter fCC Contributors)
• @evaristoc (Ubuntu - ctorrent -, Europe)

[Contributors: Please add comments if you think necessary...]

From the tests by contributors on Gitter Data Science Room and Gitter Contributors Room I could point out:

• BitTorrent is ok as long as seeders can be found available. So far no many seeders were found for tests but when found they were good enough.
• Some of the contributors might offer themselves as future seeders.
• It might require some level of knowledge of BitTorrent. BitTorrent is a popular tool but not necessarily between everyone. Still very popular between those accessing videos, etc or similar files for recreation purposes. Also relevant for those who are sharing large Open Source distros.
• compared to other hosting providers for BitTorrent, academictorrents seems to have been a good host selection in the first place.
• @jp-sauve mentions some technical aspects to take in consideration (see his message below)
• There is also the issue of increasing number of leechers vs number of seeders; depending on how popular a dataset becomes, it could run out short of good seeders if the platform is short of users, at least temporarily.
• 2 seeders (one slow, one fast), 1 leecher => approx. 15min to 60min download (NOTE: this is not a generalisable figure)

Some references:

• http://academictorrents.com/details/030b10dad0846b5aecc3905692890fb02404adbf

[mcbarlowe]

I tested this by downloading the file in the AM EST on 11/4. At first seeders were an issue after waiting i'd say around 15 minutes one seeder popped up and then about a third of the way through another seeder with much higher bandwidth allowing me to complete the download in about 30-40 minutes with a total time of 1h 43 m for the entire file to download

[jp-sauve]

I'm in London, Ontario, which is EST, UTC -5 I began downloading at 9:31am today. Speeds fluctuated at first, with a single seeder, although that seeder sent almost immediately. Once the second seeder joined, the speeds went between 500KiB/s and my max, 1.5MiB/s. The file completed at 09:39. The trouble with torrenting the data is that it requires a torrent client be running in order to seed, and also that the user's ports be forwarded accurately to the pc running the client in order to accept incoming requests. This also requires a static LAN IP, or else the port forwarding will only be correct on occasion.

[timjavins]

My download couldn't start when first initialized, due to the seeder and leechers all being offline. This was when there was only one seeder. After @mcbarlowe and @jp-sauve completed theirs, I came back to this on Monday (yesterday) and completed mine. From start to finish, it took 3 hours to download because I used a VPN, which slowed it down. I am in Seattle, but my connection was routed through Canada while my TCP and UDP IP addresses were both German. My download came from @jp-sauve and @evaristoc, according to the IP addresses.

Torrenting has the advantage of being a bit more accessible to those whose access is restricted where they are located. It's really great once the file is well-established and widely distributed, which also gives it a fairly high fault tolerance.

On the other hand, accessibility is hindered when the files are not widely distributed (ie. when the peer & seeder count is low). There's a ramping phase, of which we have already had a taste. There is also a bit of learning involved with BitTorrent. The downloader must know how to download. Everybody knows how to use a browser. Not every person who is interested in data analysis knows how to use BitTorrent software, but I'm sure they are smart enough to figure it out. Still, it's enough to put some people off. There's also some stigma in the USA, especially in the eyes of many ISPs.

My experience downloading it via BitTorrent went as expected. In my opinion, it should be a backup or alternative download source, just like the Linux distros.