search

transmission / transmission

Official Transmission BitTorrent client repository
https://transmissionbt.com
Other
11.76k stars 1.19k forks source link

transmission-create reads input files with incorrect size #6289

Open ghost opened 7 months ago

ghost commented 7 months ago

What is the issue?

For instance:

root@OpenWRT:~# transmission-create -p -t "https://mypersonalsite.com/tracker.php/47231234234dfsd/announce" -s 32MiB -o "/mnt/sda1/Alexa (29FPS).torrent" "/mnt/sda1/Private Torrents/29fps_ALEXA.mp4"
Creating torrent "/mnt/sda1/Alexa (29FPS).torrent"
1 file, 2.88 GB
86 pieces, 33.55 MB each
 done!
root@OpenWRT:~# ls -l "/mnt/sda1/Private Torrents/"
-rwxrwxrwx    1 root     root     32946875321 Aug 21  2021 29fps_ALEXA.mp4

So transmission-create thinks a 32 GB file is instead just 2.88 GB.

This behaviour happens on a number of different files (but not all), and I'm unable to find a pattern to predict when it will happen. As you can see in the example above, it's not related to any permissions issue (the file in question has its mode set to 777).

The example file above is my own personal video project, I do not use transmission for any illegal or unethical purposes so please do not ban me!

Which application of Transmission?

transmission-daemon

Which version of Transmission?

4.0.4

tearfur commented 7 months ago

Sounds like this usually happen to big files, which makes it difficult for you to provide samples to us.

Does this happen to a file that can be reproduced by us? (for example a file created using the dd command)

ghost commented 7 months ago

This hadn't occurred to me but it appears you are correct that this problem occurs mostly with files (or directories) that are above 2GB or so in size.

Is there anything else I can do to help pinpoint the source of this issue? If not, I'll try to find a random really large freely accessible (non-copyrighted) file that already exists online someplace, so all involved can directly download the same file to reproduce the behaviour.

tearfur commented 7 months ago

I'll try to find a random really large freely accessible (non-copyrighted) file that already exists online someplace, so all involved can directly download the same file to reproduce the behaviour.

If you can't read the code, this is the most straightforward thing to do.

ghost commented 7 months ago

Thank you, I did try to read through utils/create.cc, but after about an hour of squinting at the screen, I realized I was going nowhere fast. I also tried reviewing the changes made to that file in recent months, but even with that, there's quite a lot of missing knowledge that would be needed for me to be able to follow the code like that.

ghost commented 7 months ago

Okay here's an example. I used the VMware-TOOLCHAIN-ODP-70U3.iso file downloaded from the bottom of the following page: https://customerconnect.vmware.com/downloads/details?downloadGroup=ESXI70U3D_OSS&productId=974

Handily it also comes with the checksums displayed on the page in case your results are different from mine. Here's what happens when I try to create a torrent for this file:

root@OhWRT:~# transmission-create -p -t "https://mypersonalsite.com/tracker.php/47231234234dfsd/announce" -s 8MiB -o " /mnt/sda1/VMware-TOOLCHAIN-ODP-70U3.i
so.torrent" "/mnt/sda1/Private Torrents/VMware-TOOLCHAIN-ODP-70U3.iso"
Creating torrent " /mnt/sda1/VMware-TOOLCHAIN-ODP-70U3.iso.torrent"
1 file, 596.3 MB
72 pieces, 8.39 MB each
 done!

root@OhWRT:~# ls -l "/mnt/sda1/Private Torrents"
-rwxrw-rw-    1 root     root     9186191360 Nov 27 10:58 VMware-TOOLCHAIN-ODP-70U3.iso

So transmission-create is seeing this 8.55 GiB file as being only 596.3MB in size. Neither the tracker URL nor the piece size seem to matter for this issue.

tearfur commented 7 months ago

Hmm... Can't reproduce.

$ echo '3823a27b770ffe99b3b4cae8dc82bb9f5c5e73cb8b1e15297b3b91b1e0a72ec2 *VMware-TOOLCHAIN-ODP-70U3.iso' | sha256sum --check
VMware-TOOLCHAIN-ODP-70U3.iso: OK
$ transmission-create -p -t "https://mypersonalsite.com/tracker.php/47231234234dfsd/announce" -s 8MiB VMware-TOOLCHAIN-ODP-70U3.iso
Creating torrent "/test/VMware-TOOLCHAIN-ODP-70U3.iso.torrent"
1 file, 9.19 GB
1,096 pieces, 8.39 MB each
Piece 1081/1096 ... done!

On the most bottom level, the file size is determined by the system calls stat() or lstat(), could it be bugged on OpenWRT or maybe your filesystem is corrupted somehow?

https://github.com/transmission/transmission/blob/5d64c860ea27ac1b25fded7683df0feeac8d137d/libtransmission/file-posix.cc#L150-L157

siiky commented 7 months ago

This happens on my RBPi2 (32bit ARM) with v4.0.4. Needless to say, I never saw this happen in the past (with v3.x), so I believe it's new in v4.x.

I don't have a file to test this right now, I'm downloading Ubuntu 23.10 Desktop (64-bit) (from here). I'll comment again when I have news.

EDIT: btw the same file I first saw this bug happen with, transmission-create v4.0.2 worked correctly in an x64 laptop.

ghost commented 7 months ago

Apparently some others using the same distro as me (OpenWRT) are unable to reproduce the issue even with the same file as me.

Couple questions for you, @siiky :

(1) What filesystem are you using?

(2) Are you able to share the file that you saw this happen with?

siiky commented 7 months ago

@racingcirc (1) EXT4 and (2) not really, the file was of dubious origin (: but you can try the Ubuntu iso I linked above. When that's finished downloading (I have a very low download speed limit) I'll share the results on my end.

ghost commented 7 months ago

The plot thickens - I am also using ext4!

@tearfur What filesystem did you test this with, please?

ghost commented 7 months ago

Also adding an additional piece of info: The stat command shows the correct info for the file yet transmission-create still sees the incorrect file size:

root@OpenWRT:~# stat "/mnt/sda1/Private Torrents/VMware-TOOLCHAIN-ODP-70U3.iso"
  File: /mnt/sda1/Private Torrents/VMware-TOOLCHAIN-ODP-70U3.iso
  Size: 9186191360      Blocks: 17941792   IO Block: 4096   regular file
Device: 8,1     Inode: 104857604   Links: 1
Access: (0766/-rwxrw-rw-)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-11-30 23:31:42.116050311 -0500
Modify: 2023-11-27 10:58:46.285102400 -0500
Change: 2023-11-27 12:19:10.803382990 -0500
 Birth: -
ghost commented 7 months ago

Wondering whether this has anything to do with the change in musl 1.2.4 with legacy LFS64 API calls for large files? https://musl.libc.org/releases.html

tearfur commented 7 months ago

@tearfur What filesystem did you test this with, please?

I'm using a btrfs volume on x64 Fedora 39. I don't have an ARM device to test with, so I'm afraid I can't help much with that.

Wondering whether this has anything to do with the change in musl 1.2.4 with legacy LFS64 API calls for large files?

No, we never used the stat64() api.

tearfur commented 7 months ago

Needless to say, I never saw this happen in the past (with v3.x), so I believe it's new in v4.x.

If this is true, and compounded by the observation that the stat command is working properly, I'm guessing stat() and lstat() is not related here. We are using them in the exact same way in 3.00 as well.

The most probable breaking point I can see now is 5eb7f75010156a983773b6728cb3a6fa7eac9c52, where the function signature of tr_sys_path_get_info() was changed:

/* Before */
bool tr_sys_path_get_info(char const* path, int flags, tr_sys_path_info* info, tr_error** error)

/* After */
std::optional<tr_sys_path_info> tr_sys_path_get_info(std::string_view path, int flags, tr_error** error)

Having said that, I'm not sure if it is a problem on our part or the standard library. I can't see any problematic code from us so far.

@racingcirc @siiky If any of you can build from source, please try and A/B test before and after 5eb7f75010156a983773b6728cb3a6fa7eac9c52 to see if this is the culprit.

ghost commented 7 months ago

I can't build from source right now for a couple reasons, but will be looking into the possibilities over the next few weeks. For now, I've found that mktorrent seems to work fine for my distribution:

root@OpenWRT:~# mktorrent -p -v -a "https://mypersonalsite.com/tracker.php/47231234234dfsd/announce" -l 23 -n "Test Torrent made with VMWare 7.0 U3 Toolchain" -o "/mnt/sda1/VMware-TOOLCHAIN-ODP-70U3.iso.torrent" "/mnt/sda1/Private Torrents/VMware-TOOLCHAIN-ODP-70U3.iso"
mktorrent 1.1 (c) 2007, 2009 Emil Renner Berthing

Options:
  Announce URLs:
    1 : https://mypersonalsite.com/tracker.php/47231234234dfsd/announce
  Torrent name:  Test Torrent made with VMWare 7.0 U3 Toolchain
  Metafile:     /mnt/sda1/VMware-TOOLCHAIN-ODP-70U3.iso.torrent
  Piece length: 8388608
  Be verbose:   yes
  Write date:   yes
  Web Seed URL: none
  Comment:      none

9186191360d bytes in all.
That's 1096 pieces of 8388608 bytes each.

Hashing /mnt/sda1/Private Torrents/VMware-TOOLCHAIN-ODP-70U3.iso.
Writing metainfo file... done.

It looks like mktorrent also uses a similar stat call: https://github.com/pobrn/mktorrent/blob/de7d011b35458de1472665f50b96c9cf6c303f39/init.c#L165-L169

This hopefully confirms that the issue is specific to transmission, since it doesn't affect other packages on OpenWRT.

siiky commented 7 months ago

@tearfur I'll be very tight on time for the coming weeks, it's unlikely I'll be able to compare with/without that commit (dunno when I'll be able to upgrade to v4.0.5 either). But I took note, if I have a little spare time I'll try and report the results!

siiky commented 2 months ago

Hi all, just built and installed v4.0.5 and the issue seems to have been fixed, at least for a few torrents.

Specifically, every time those torrents reached 100%, if I asked to check local data, Transmission would report a ton of corrupt pieces. E.g., a 2.24GB torrent for which Transmission v4.0.4 downloaded 6.02GB.

I'll report back if I notice anything weird still happening.