.parts file not removed or shrunk when priority increased from 0 so pieces are copied out

arvidn / libtorrent

an efficient feature complete C++ bittorrent implementation

http://libtorrent.org

Other

5.17k stars 995 forks source link

.parts file not removed or shrunk when priority increased from 0 so pieces are copied out #7603

Open pcordes opened 7 months ago

pcordes commented 7 months ago

libtorrent version (or branch): 2.0.9-3 platform/architecture: x86-64 Arch GNU/Linux using package version 1:2.0.9-3

In Deluge-gtk (version 2.1.1-4 also from Arch distro packages), changing a file from "skip" to some other priority results in it appearing in the filesystem if it wasn't there already.

But the .parts file is still there with the same size as before, even if all the files now have non-zero priority so all pieces have been copied into the filesystem.

There might be a Deluge bug involved (with an earlier version, from 2023 before april) because I somehow had some torrents where a bunch of large files set to "skip" had actually been fully downloaded, so all their pieces were in large .parts files. I don't remember the details of how I added those torrents. I only recently figured out how to coax Deluge into letting me change priority on a "complete" torrent:

pause the torrent so useless downloading doesn't start in later steps.
rename one of the files that does exist in the filesystem and force-recheck
then the priority items won't be grayed out in the right-click menu, and setting them to something other than skip materializes the file in the filesystem.
rename the file back to its proper name and force-recheck again, so deluge knows the torrents is complete.

arvidn commented 7 months ago

there is no process to "defragment" or "compact" the parts file. The parts file is quite unsophisticated and really just a list of skipped blocks. When blocks are moved out of the part file, in order to truncate the file, other skipped blocks may need to be shifted down accordingly.

The case where all blocks are removed is an especially simple case that probably could be addressed quite easily. and maybe also when the blocks being removed happen to be at the end of the file, the file can also be truncated.

pcordes commented 6 months ago

Somehow Deluge managed to get into a state where all pieces of some large-ish files were downloaded but going to to the .parts file, so I had a couple multi-gigabyte files for the torrents I eventually got it to back to the FS. So lack of deletion was very noticeable in that corner case.

Even in normal cases, it would definitely be nice in general to avoid cluttering the FS with dot-files of a few MiB after deciding to download the rest of the files in a torrent after sampling a couple.

If you did ever want to implement collapsing of .parts files when a block is no longer needed, Linux fallocate(2) can punch holes in the file, making it sparse and freeing up the disk space. Or to logically defrag and shrink the file length with ftruncate(2), Linux ioctl(FICLONERANGE) can "copy" a block with only metadata I/O (on some filesystems)