creating a hybrid torrent fails to add piece layers

[master255]

libtorrent version (or branch): 2.0

platform/architecture: Android

@arvidn When I create hybrid torrents (without v1 and v2 flags) - I cannot add them to the torrent session. I get an error:

java.lang.IllegalArgumentException: Can't decode data: the v1 and v2 file metadata does not match
at org.libtorrent4j.TorrentInfo.bdecode0(TorrentInfo.java:674)
at org.libtorrent4j.TorrentInfo.(TorrentInfo.java:46)

What to do? Hybrid torrents don't work? Or do they need to be added in some special way?

Code that creates hybrid torrents: https://github.com/aldenml/libtorrent4j/blob/b28e0ce4d29c14271fa9382b7b5f95e3cfe1e37f/src/main/java/org/libtorrent4j/TorrentBuilder.java#L436

Add torrent code: https://github.com/aldenml/libtorrent4j/blob/b28e0ce4d29c14271fa9382b7b5f95e3cfe1e37f/src/main/java/org/libtorrent4j/TorrentInfo.java#L663

[master255]

@arvidn Will this be fixed?

[master255]

Example file shared.zip

[master255]

This code:

final TorrentBuilder builder = new TorrentBuilder(); builder.path(pathFile); byte[] result = builder.generate().entry().bencode(); TorrentInfo torrentInfo = new TorrentInfo(result);

Gives this error:

2021-03-06 17:30:09.750 5042-6313 W/System.err: java.lang.IllegalArgumentException: Can't decode data: the v1 and v2 file metadata does not match 2021-03-06 17:30:09.753 5042-6313 W/System.err: at org.libtorrent4j.TorrentInfo.bdecode0(TorrentInfo.java:674) 2021-03-06 17:30:09.753 5042-6313 W/System.err: at org.libtorrent4j.TorrentInfo.(TorrentInfo.java:46)

Android 26api arm 64 Files shared.zip

[arvidn]

final TorrentBuilder builder = new TorrentBuilder();
builder.path(pathFile);
byte[] result = builder.generate().entry().bencode();

This does not resemble the libtorrent API. Can you translate it into which libtorrent calls it makes?

[aldenml]

Hi @arvidn, do you think the whole issue of not having the files in order for the v1 part could be a problem in this logic: https://github.com/arvidn/libtorrent/blob/b40801a3a3ca741665e34f49d403a491dd43c4bb/src/torrent_info.cpp#L1286

[master255]

@arvidn These are standard calls that all libtorrent applications use.

[arvidn]

@aldenml that logic only applies to "hybrid" torrents. i.e. that are both valid v2 and v1. In that case the v1 "part" is subject to the same file-ordering and padding rules as the v2 part is.

[master255]

@arvidn Code that creates hybrid torrents: https://github.com/aldenml/libtorrent4j/blob/b28e0ce4d29c14271fa9382b7b5f95e3cfe1e37f/src/main/java/org/libtorrent4j/TorrentBuilder.java#L436

Add torrent code: https://github.com/aldenml/libtorrent4j/blob/b28e0ce4d29c14271fa9382b7b5f95e3cfe1e37f/src/main/java/org/libtorrent4j/TorrentInfo.java#L663

[master255]

@arvidn What else do you need?

[arvidn]

Code that creates hybrid torrents: ... What else do you need?

I would like you to lead with that.

[arvidn]

except, that's still java code.

[master255]

@arvidn I have my own piece of code that I write. My program is even bigger than yours. And it contains a lot of C++ libraries. I can only optionally help fix bugs in the libraries. The first priority for me is my application and it has a lot of tasks.

Moreover, I don't understand why you don't see the native C++ calls in the links above. Maybe you need to create an android app to test your code? I can help with that. Just tell me.

[arvidn]

$ cat test_make_torrent.cpp 
#include <memory>
#include "libtorrent/libtorrent.hpp"

int main(int, char const** argv)
{
        lt::file_storage fs;
        lt::create_flags_t flags = {};
        lt::add_files(fs, argv[1], [](std::string f) { std::cout << f << '\n'; return true; }, flags);

        lt::create_torrent t(fs, 0, flags);
        lt::set_piece_hashes(t, argv[2], [](lt::piece_index_t) {});

        std::vector<char> buf;
        lt::bencode(std::back_inserter(buf), t.generate());

        auto ti = std::make_shared<lt::torrent_info>(buf.data(), buf.size());

        return 0;
}

as far as I can tell, this is what ends up being called. The problem is, this program works, so surely there must be something else happening. @master255 you seem to demand that I debug your reproduction to figure out how it's using the create_torrent class. I think it's your responsibility.

Please give me a small C++ program that reproduces the issue.

[arvidn]

looking at the shared.zip torrent file:

$ ./dump_torrent ~/Downloads/shared.torrent 
ERROR: one or more files are missing piece layer hashes

And indeed, there is no piece layers field in the torrent file.

Perhaps it's a property of the files you're creating the torrent from that triggers an issue. Does it happen consistently for you or just with certain files?

[master255]

@arvidn All the bugs I have happen all the time with this set of files. You can use it. shared.zip

you seem to demand that I debug your reproduction to figure out how it's using the create_torrent class.

You do not need to do a debug. I use the standard create_torrent method. Your program above is not complete. It doesn't use bdecode_node which I use. This method is probably the problem.

[arvidn]

You do not need to do a debug. I use the standard create_torrent method. Your program above is not complete. It doesn't use bdecode_node which I use. This method is probably the problem.

Actually, the torrent_info() constructor does that internally. But the error happens later anyway, and the reason the failure happens is because there is no piece layers field in the torrent.

[master255]

@arvidn So there is something wrong with set_piece_hashes_ex her code is here: https://github.com/aldenml/libtorrent4j/blob/b28e0ce4d29c14271fa9382b7b5f95e3cfe1e37f/swig/libtorrent.hpp#L261

[arvidn]

one theory is that there's a disk error that fails to be propagated. I discovered this: https://github.com/arvidn/libtorrent/pull/6037

[master255]

@arvidn Why is there a disk error at all? I can check if you want. I haven't rolled back to 1.2.12 yet :-)

[arvidn]

You do not need to do a debug. I use the standard create_torrent method.

There is no "default create_torrent method". unless you're referring to the make_torrent.cpp example. Does that also create a broken torrent for you?

Your program above is not complete. It doesn't use bdecode_node which I use. This method is probably the problem.

bdecode_node is a function. I've already diagnosed that the torrent is invalid. Assuming the torrent you posted above (in shared.zip) is the output of your torrent creator.

Trying various flags and various files, I cannot reproduce the issue where the output is an invalid torrent. It appears you cannot (or are unwilling to) reproduce the sequence of calls and arguments that cause this issue. I also get the impression that you're not particularly interested in finding out. Because you ask questions like:

Why is there a disk error at all?

Which is a bit hard to interpret. Surely you know how trouble-shooting and debugging works. The question claims that there is an error, without it having been established, and even if it had been established, the answer is so obvious. You propagate the error and print it, or catch it in de debugger, to inspect it. Surely you know this as well. The only reasonable interpretation is that you use this as some kind of rhetorical device. None of those interpretations are very flattering though (quite the opposite), that's the puzzling part.

[master255]

@arvidn

There is no "default create_torrent method".

Dear Arvin. I only want to help. These are the methods used: https://github.com/arvidn/libtorrent/blob/a44057b6e91a5b70a63f779ddd7eecb6ce7af450/src/create_torrent.cpp#L381

https://github.com/arvidn/libtorrent/blob/a44057b6e91a5b70a63f779ddd7eecb6ce7af450/src/create_torrent.cpp#L284

If they work for you, there must be some problem with the Android integration. It would be nice if you could reproduce the bugs yourself to fix them. Replaying the bug is the first thing you need to do to fix it. I'm willing to help you do that, with the help of Android.

[arvidn]

Replaying the bug is the first thing you need to do to fix it. I'm willing to help you do that

Please do. Can you modify my program to demonstrate the bug?

[arvidn]

I simplified it to this, since it's creating the torrent that's failing (till, assuming your shared.zip was the output torrent for you, that failed):

#include <memory>
#include "libtorrent/libtorrent.hpp"

int main(int, char const** argv)
{
        lt::file_storage fs;
        lt::create_flags_t flags = ... various flags to test ... ;
        lt::add_files(fs, argv[1], [](std::string) { return true; }, flags);

        lt::create_torrent t(fs, 0, flags);
        lt::set_piece_hashes(t, argv[2], [](lt::piece_index_t) {});

        std::vector<char> buf;
        lt::bencode(std::back_inserter(buf), t.generate());

        std::cout.write(buf.data(), buf.size());
        return 0;
}

That file can then be tested with dump_torrent and dump_bdecode.

[master255]

@arvidn https://github.com/aldenml/libtorrent4j/issues/163#issuecomment-792246132

The problem is not with the code, but with the device. To reproduce the bug you need a real Android arm phone.

This bug was not present in the library 1.2.12

[master255]

@arvidn I tested on Samsung Galaxy s8+ Android 8.0 arm64 and Xiaomi Redmi Android 10 arm32. The error is reproduced regardless of the Android version and system bits.

https://mega.nz/file/stlD0Q7B#10IlEI1nJM8_jxeAeoN4XgFB0aVijj_r2BvLYvLZvp4 https://mega.nz/file/dwEFRaJJ#jVzzRyM2Nc8gEQC8EqeWpR2C94jHpaaj6xy9S2Hd2qk

[arvidn]

@master255 did you reproduce this with a new version of libtorrent that include this patch, to propagate errors in set_piece_hashes()?

[master255]

@arvidn As you can see, the library already has canonical_files. So I had a library for the state of that commit. Your commit is much older. So my version of the library already had your commit.

[kovalensky]

@arvidn The question itself is off-topic, I just didn't want to start a new issue. I just saw the share mode extension to help the swarm, very interesting, I wanted to know if there are GUI torrent clients that support this? Thank you.

[arvidn]

@smell-of-rain please open a new ticket

[arvidn]

@master255 it's not clear what your second video is supposed to demonstrate. It would be a lot simpler if you would explain that in english and include the checks you make in the code in the message, as text (rather than video or screenshot).

perhaps you could set a break point at the calls into libtorrent and record what's being called. that way I may get an idea of what the problem is. As I've demonstrated in the small program above as well as the unit tests, I cannot get this to fail.

[master255]

@arvidn I used the libtorrent4j library. The code to create a torrent is standard and you've already given it. The problem why the error is not reproducing in your case is the data structure from which you are generating the torrent. I used a complex structure. Lots of files. They are all different sizes and have subdirectories which also have files in them. Try using a complex file structure.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[master255]

Error is present

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[master255]

Error is present

[master255]

not actual

[sijanec]

Hello!

I came across this bug when crawling the DHT for research purposes. I am not a (direct) libtorrent user not a (direct) user of the mentioned java wrapper library, but I noticed that a lot of torrent exhibit strange behavior -- while they contain meta version == 2 key in the info dict, they do not contain piece layers. I pasted the smallest torrent among those (e5dfcc39b252356c4a873937abcb56be9f6988ae) in decoded form here:

b:~/projects/travnik/www[0]# ../tmp/bencoding decode < $(ls --sort=size `grep metainfo ../tmp/insert.txt | cut -d\  -f3 | cut -d: -f1` | tail -n1) | jq
len: 1085
{
  "created by": "http://ni...ijanec.eu/sijanec/travnik mailto:tk@sijanec.eu",
  "creation date": 1675233369,
  "encoding": "UTF-8",
  "info": {
    "file tree": {
      "H35_Roac Selene_vamx.top.zip": {
        "": {
          "length": 667308,
          "pieces root": ".#.\".*.PO..n\u0004\\e.......\tA.....)E."
        }
      }
    },
    "length": 667308,
    "meta version": 2,
    "name": "H35_Roac Selene_vamx.top.zip",
    "piece length": 32768,
    "pieces": "..!....?j.....f..U\u001fQ.@1.\u001c.....\u0000\u001fM.\u000e&\fv\u0007UG.\u0003\u001f*.].uR.$\f.s....a^..}.?\u000b...O..\u001be*w]p\u000e....\u001a{..\u0018\u0007\u0001y\u0011B.~Z......6.5.4h...m\u007fX..Y..z\u0011...\u0001'\u0019.$.g..{..@#F\r\u0010\u0019.,+.D.\u0014'\ta.acW...\u000fT.l&.3%.C...UM~...\u000fX.X\u0018.M.....\u0004.%\u0019..\u0005.\u007f3\u0013\u00119..\u001e...;\u0011..(..BWQ`.V.0FkI.....dX7HZ\u0018..\u0012..Y..].>..@..:......i..4N.....Df\"...N.7}m.@*['.i.%\u0011)B.\u0014|.v...\u0002-.\"..\u0013.7[{/.\u0012F..6.km)\u0014OS....\r....GK.T\u0019\tC.\u0015.n.....s......:.`...\u0000.\u007f\u0018Q\u0011.....:...].\f..\u001b..\rN..?ZIX\b...X3.m:O.\b.........T."
  },
  "source": {
    "ip": "::ffff:121.231.50.223/30713",
    "v": "Tixati 3.14"
  }
}

A list of hashes of torrents you can find in the DHT that exhibit this behavior: http://upload.šijanec.eu./d/hashes.txt

And the corresponding torrent files: http://upload.šijanec.eu./d/missing.tar.gz

Among 17706 quite randomly sampled torrent files, 138 exhibit this bug. Among those, 34 were made by libtorrent.

[master255]

@sijanec Is this the latest version 2 of the library?

[arvidn]

@sijanec when you download a torrent via a magnet link (which you essentially are when you discover an info-hash via the DHT) you will only receive the info-dictionary. The piece layers field is outside of the info-dict. While downloading the torrent, parts of the hash tree will be requested on the fly and be complete by the time you finish the download

[sijanec]

Thanks a lot for the quick response! Sorry for bringing up an old issue and alerting both of you. I did not know that, I've only skimmed over the v2 standard.