thalieht
commented
5 years ago Looks like a limitation in libtorrent: https://github.com/arvidn/libtorrent/search?q=too_many_pieces_in_torrent&unscoped_q=too_many_pieces_in_torrent
Closed wojciehm closed 4 years ago
thalieht
commented
5 years ago Looks like a limitation in libtorrent: https://github.com/arvidn/libtorrent/search?q=too_many_pieces_in_torrent&unscoped_q=too_many_pieces_in_torrent
Chocobo1
commented
5 years ago @wojciehm Are you able to load the torrent with other bittorrent client? utorrent couldn't load it too, could it be corrupted?
FranciscoPombal
commented
5 years ago btcheck can't open it either:
$ btcheck -lvi VMworld2018EU.torrent
String length overflow.It looks like the contents of the torrent are making these programs bump into some hardcoded limits of their bencode decoder implementations. For example, in btcheck, this is the limit for how big a string can be:
In src/btree.h:
// (...)
#define MAX_STRING_LENGTH (1<<25) // this is 33554432
// (...)And this is the end of the code path that leads to the failed execution:
In src/bdecode.c:
// (...)
if (length >= MAX_STRING_LENGTH) {
fprintf(stderr, "String length overflow.\n");
exit(EXIT_FAILURE);
// (...)The problem with libtorrent is probably similar.
FranciscoPombal
commented
5 years ago There appear to be 2 relevant limits in the libtorrent code (https://github.com/arvidn/libtorrent/search?q=max_pieces&unscoped_q=max_pieces):
(std::numeric_limits<int>::max)() - 1 //2147483646
0xC0000 //786432Then, 3 possible places where the "Too many pieces" error may be triggered, based on a few different conditions:
@wojciehm
Do you have the time to debug this and figure out which condition exactly is failing for libtorrent to produce this error? That way, it is possible to figure out which limit is being exceeded.
@arvidn ping, please take a look at this.
FranciscoPombal
commented
5 years ago @wojciehm @Chocobo1
I just compiled btcheck with a minor change (increased MAX_STRING_LENGTH to 1<<26).
The torrent file itself seems to be alright, though the size/piece size combination is not the best. The total file size is quite big, at 417.37 GiB, and the torrent file uses a piece size of only 256 KiB, making the total number of pieces 1 709 564 (!).
Here is the full result from btcheck:
@arvidn What do you think about relaxing the hardcoded limits of libtorrent a bit to accomodate decoding bigger .torrent files like this one?
With increasing file sizes, .torrent files like this will appear more and more. Granted, some of the responsibility falls upon the author of the .torrent file to make decent decisions about the piece size, but still.
Maybe libtorrent could allow big (read: "arbitrary") memory allocations for decoding .torrent files behind a flag/configure setting?
qBittorrent could then make use of such functionality to show a dialog along the lines of "Warning: this .torrent file is very big/contains many pieces. Attempting to open it may take a long time and/or consume a lot of system resources. Proceed?" when opening big files. I think it is not only important to protect against intentionally malicious torrent files, but also to give the user the option to make their own decisions for edge cases like these.
arvidn
commented
5 years ago how does this look? https://github.com/arvidn/libtorrent/pull/3877 anyone want to give it a try in qbt?
FranciscoPombal
commented
5 years ago @arvidn Just tested your PR, and added a few questions there. @Chocobo1 I'm not really sure how to test this in qBittorrent. Can you help?
Anyway, here is the output of dump_torrent (I redacted the value of the pieces key because it is too big):
FranciscoPombal
commented
5 years ago @Chocobo1
libtorrent 1.2.2 was recently released with a feature/fix that should enable a relatively simple fix for these kinds of issues:
add torrent_info constructor overloads to control torrent file limits
xavier2k6
commented
5 years ago just tried to add torrent file in OP with 4.2.0 beta1 & issue remains.
also this torrent here has same issues 62.1MB
Torrent Info Output
| Total Size | File Count | Pieces Count | Piece Length |
|---|---|---|---|
| 747GiB | 23878 | 3062828 | 256KiB |
FranciscoPombal
commented
5 years ago @xavier2k6 For the issue to be fixed: 1 - qBittorrent needs to use libtorrent >= 1.2.2 2 - the defaults for torrent file limits used in qBittorrent should be increased using the functionality introduced in libtorrent 1.2.2 3 - (optional) qBittorrent should expose a way to configure the torrent file limits parameters, so that users can still work around exceptionally large torrents that exceed the default limits. If this is implemented then step 2 can be ignored
IIRC so far we only have step 1.
jasonsyoung
commented
4 years ago I have written an implementation for this. I am not an expert in libtorrent or c++, so can someone take a look at my code? I have tested it locally. Here is the branch: https://github.com/jasonsyoung/qBittorrent/tree/load_torrent_limits
arvidn
commented
4 years ago @jasonsyoung it would be easier to review if you would create a pull request from that branch. You could still prefix the name with WIP: to make it clear you don't consider it done yet.
FranciscoPombal
commented
4 years ago The defaults in libtorrent are being raised: https://github.com/arvidn/libtorrent/pull/4815. This is enough for OP's torrent to work OOTB, if qBittorrent is compiled with a version of libtorrent recent enough to include that patch. Obviously this does not solve the more general problem that these limits should be configurable in qBIttorrent.
arvidn
commented
4 years ago even if they aren't configurable by end users, it might make sense for qbt to raise the limits it's using. knowing that you run on a desktop and that only humans add torrents might warrant greater limits. (and maybe the limits should stay in place for RSS or other automatically added torrents)
FranciscoPombal
commented
4 years ago @arvidn
even if they aren't configurable by end users, it might make sense for qbt to raise the limits it's using. knowing that you run on a desktop and that only humans add torrents might warrant greater limits. (and maybe the limits should stay in place for RSS or other automatically added torrents)
That is possible, but there is definitely the need for the distinction between manually and automatically added torrents. Many people run qbittorrent-nox and just add torrents automatically.
xavier2k6
commented
4 years ago With qBittorrent 4.3.0.1 which uses libtorrent 1.2.10 I am able to add VMworld2018EU.torrent from https://github.com/qbittorrent/qBittorrent/issues/10913#issue-466822591
I am still unable to add Tutorials.torrent from https://github.com/qbittorrent/qBittorrent/issues/10913#issuecomment-548953007
jasonsyoung
commented
4 years ago With
qBittorrent 4.3.0.1which useslibtorrent 1.2.10I am able to addVMworld2018EU.torrentfrom #10913 (comment)I am still unable to add
Tutorials.torrentfrom #10913 (comment)
The UI needs to be changed to support larger torrents. I have a branch that fixes this, but I was not able to get it completely working so it never got merged. Maybe a Dev will one day take it over :/.
FranciscoPombal
commented
4 years ago https://github.com/qbittorrent/qBittorrent/issues/12029 is focused on solving the general problem.
Please provide the following information
qBittorrent version and Operating System
Windows Server 2016, QBT 4.1.6
If on linux, libtorrent and Qt version
(type here)
What is the problem
When adding torrent file https://www.wojcieh.net/wp-content/files/VMworld2018EU.torrent I receive an error message: error: too many pieces in torrent.
What is the expected behavior
(type here)
Steps to reproduce
Download and try to use it.
Extra info(if any)
Torrent file was created on Windows 10 and QBT version 4.1.3