Upgrading to libtorrent 2.0, it's time.

[gubatron]

Will start a development branch libtorrent-2.0 to attempt to upgrade to libtorrent 2.0.

With libtorrent 2 we'll be supporting BitTorrent v2 and this gives us:

• A transition away from SHA1 as the algorithm to hash pieces and into SHA256. Google announced a technique for generating collisions. This would make the network and the contents promised by a torrent safer from spoofing, as well as spoofing torrent pieces.
• Merkle hash trees for pieces (instead of SH1 hashes of every piece), a lot smaller metadata in the info-dict of the .torrent. Meaning less latency opening magnet links. The Merkle tree is made from hashes of 16 kiB blocks, if there's a corruption that small it can be detected right away and no longer do you need to download a whole block (like in the case of big downloads that may have big pieces)
• Being able to detect corrupt data down to just 16kib also makes it possible to discover them and perhaps identify malicious peers faster.
• Regardless of piece size, hash tree leaves are always of 16 kiB (block size)
• The concept of piece size still exists. Hash trees of pieces (subtree of the full Merkle tree) are used.
• v2 .torrents must still contain piece hashes. .torrent files are not smaller for v2 torrents, however the info-dictionary is, which is what magnet links use to start downloading
• per-file hash trees. Every file in the torrent has a hash tree.
• Identical files will always have the same hash and can more easily be moved from one torrent to another and identified among different torrent swarms.
• All files will be aligned to pieces, which means there are implicit pad files after each file. This fixes very long-standing issues on the network that made it hard to have entire copies of a file because users may have just been seeding other files that were nearby in the torrent.
• torrent file lists are encoded more efficiently, instead of a flat list, the directory structure is stored as a tree (using bencoded dictionaries), this way directory names are mentioned only once inside the .torrent
• Piece sizes must be a power of 2 > 16,384 (16kIB)
• improved magnet links now prefixed with urn:btmh, look like magnet:?xt=urn:btmh:<tagged-info-hash>&dn=<name>&tr=<tracker-url>. "mh" stands for multi-hash format encoded in hexadecimal. It will have a two byte prefix of 0x12 0x20 and it's possible to include both a v1 (btih) and v2 (btmh) infohashes in a magnet for backwards compatibility.
• hybrid torrents, which can participate in both a v1 and a v2 swam at the same time, serving the same files. These have 2 info-hashes, a SHA-1 and a SHA-256 hash.

• Default disk I/O in libtorrent 2.0 uses memory mapped files, a lot of what used to be handled by libtorrent's disk caching subsystem is now handled by the underlying OS kernel, this simplifies the disk code and makes it more efficient when using modern disks and physical memory. The disk subsystem is also not threaded and more primitive than a memory mapped one. Hopefully, this won't be an issue with Android's file system, and in that case we'll have to implement a custom disk_interface and configure it when creating the session, instead of the storage_interface.

  According to the libtorrent site, we have to do the following things:

• [x] C++11 no longer supported, build with cxxtd=14 (We are building with c++17)
• [x] oldest boost is 1.67 (We are already building with boost 1.80)
• [ ] add_torrent_params::info_hash deprecated in favor of add_torrent_params::info_hashes which now uses an info_hash_t parameter which bundles both a SHA-1 and SHA-256 hash (used for hybrid torrents and magnet links with multi-hash support)
• [ ] torrent_removed_alert, torrent_deleted_alert, torrent_delete_failed_alert now have info_hash_t members, the old versions of these alerts are also deprecated.
• [ ] An info_hash_t object for a hybrid torrent will have both the v1 and v2 hashes set, it will compare false to asha1_hash of just the v1 hash.
• [ ] Calls to [torrent_handle::info_hash()](https://libtorrent.org/reference-Torrent_Handle.html#info_hash()) may need to be replaced by [torrent_handle::info_hashes()](https://libtorrent.org/reference-Torrent_Handle.html#info_hashes()), in order to get both v1 and v2 hashes.
• [ ] Each tracker (a vector of announce_entry_ has a list of endpoints (listen sockets) announced independently. Each announce_endpoin has an array of info_hashes indexed by their protocol version (thus ready for N future upgrades of the protocol)
• [x] Old merkle tree torrent support removed, since v2 has better support for merkle trees, where each file has its own merkle tree of 16kiB block leafs. add_torrent_params will no longer have the merkle_tree member, and instead has verified_leaf_hashes and merkle_trees members. (I believe we don't make much use of this in jlibtorrent's higher level Java API, I might be wrong)
• [ ] create_torrent class creates hybryd torrents by default. passing the v1_only or v2_only flag allows for the creation of non-hybrid torrents.
  • [ ] pad_file_limit and alignment parameters for create_torrent constructor have always been removed.
  • [ ] mutable_torrent_support flag is also always on
  • [ ] set_file_hash and file_hash() functions are obsolete, v12 torrents have a file_root() for each file (the merkle root of the sha256 pieces tree.
  • [ ] socket_type_t enum used to identify different kinds of sockets. udp has been deprecated in favor of utp.
  • [ ] socket_type_t is now used by peer_connect_alert, peer_disconnected_alert, incoming_connection_alert, listen_failed_alert, listen_succeeded_alert
  • [ ] DHT configuration options were usually separated from the main client settings, they've now been unified into the main settings_pack, lt::dht::dht_settings is now deprecated in favor of dht_* in settings_pack
  • [ ] the new dht_settings changes the dht custom storage constructor (see session_params). Instead of taking a dht_settings object, it now takes the full settings_pack object.
  • [x] stats_alert is deprecated, instead call session::post_torrent_updates(). This will post a state_update_alert containing the torrent_status of all torrents that have any updates since the last time the function was called. This allows for better scalability.
  • [ ] save_state() and load_state() on the session have been deprecated in favor of loading the session state up front using read_session_params() and construct the session from it.
  • [ ] The session state can be acquired, in the form of a session_params object, by calling session::session_state()
  • [ ] Use read_session_params() and write_session_params() to serialize and de-serialize the session_params object.
  • [ ] A lot of session constructors have been deprecated in favor of the ones that take a session_params object. The sessions_params object can be implicitly constructed from a settings_pack.
  • [x] Adding torrents by URL no longer supported. (If we need it we get the torrent bytes ourselves in java-land). Previously libtorrent would attempt to download the .torrent and in the meantime the partial torrent hash was used with a temporary hash of the URL until the torrent was loaded. In libtorrent 2, info-hashes cannot change. (we already do this on FrostWire ./com/frostwire/gui/bittorrent/TorrentFetcherDownload.java)
  • [x] add_torrent_params::url could also be used to add torrents by magnet link. This was also deprecated in the previous version and has been removed in libtorrent 2.0. Instead, use [parse_magnet_uri()](https://libtorrent.org/reference-Core.html#parse_magnet_uri()) to construct an add_torrent_params object to add to the session. This also allows the client to alter settings, such as save_path, before adding the magnet link.
  • [x] Clients should instead load their torrents from disk themselves, before adding them to the session. Possibly spawning their own threads.
  • [ ] Instead of configuring a custom storage by implementing storage_interface when adding a torrent you can now configure a disk subsystem by implementing the disk_interface when creating a session. Check use of libtorrent.i defined add_torrent_params.set_disabled_storage() used in SessionManager::fetchMagnet(...)
  • [x] add_torrent_params no longer has a storage_constructor member.
  • [x] cache_size setting is no longer used. Remove SettingsPack::cacheSize() and SettingsPack::cacheSize(int) methods

[gubatron]

Development branch libtorrent-2.0 started. ./build-macos.sh works, ./gradlew test is passing all existing tests.

Have yet to test the library on FrostWire and see if FrostWire compiles and then will start making a battery of tests, fixing and adding new tests on this project.