search

qbittorrent / qBittorrent

qBittorrent BitTorrent client
https://www.qbittorrent.org
Other
28.69k stars 4.01k forks source link

Pre-allocate disk space for all files option does not work as intended #19410

Open superbonaci opened 1 year ago

superbonaci commented 1 year ago

qBittorrent & operating system versions

qBittorrent 4.5.4 (64-bit) Windows or 4.6.0beta2 (64-bit) Arch OS: Windows 11 64-bit or Linux 64-bit Qt: 6.4.3 Libtorrent: 2.0.9.0 Boost: 1.82.0 OpenSSL: 1.1.1u zlib: 1.2.13

What is the problem?

The option Preferences - Downloads - Pre-allocate disk space for all files (either if the desired outcome is enabled or disabled) doesn't work always as expected. Here are some use cases explaining the "bug". Basically it means you can't trust this option as is and the results vary.

superbonaci commented 1 year ago

HDD + WINDOWS + NTFS + NO-PREALLOCATE

Working as intended: YES

Steps to reproduce the issue:

  1. Do not start the download automatically: ticked
  2. Pre-allocate space for all files: not ticked
  3. Add Torrent File - Open
  4. Select None
  5. Download in sequential order: ticked
  6. Save at: HDD (mechanical spinning) formatted in NTFS
  7. OK
  8. Select the torrent from the list and in lower tab Content select (tick) only one file. It should keep as Paused.

The file does not take space in the filesystem and is allocated instantly. The file is created.

HDD + WINDOWS + NTFS + NO-PREALLOCATE

As torrent is Resumed, the file used size (Size on disk) grows according to the number of Pieces that have been downloaded. After a few pieces are downloaded Linux reads this information:

$ du -sh *
37M     HDD + WINDOWS + NTFS + NO-PREALLOCATE.mkv
$ du -h --apparent-size *
22G     HDD + WINDOWS + NTFS + NO-PREALLOCATE.mkv
$ stat *
  File: HDD + WINDOWS + NTFS + NO-PREALLOCATE.mkv
  Size: 23180943977     Blocks: 75264      IO Block: 4096   regular file
Device: 8,33    Inode: 294         Links: 1
Access: (0755/-rwxr-xr-x)  Uid: ( 1000/  user)   Gid: ( 1000/  user)
Access: 2023-08-05 00:57:42.486743600 +0000
Modify: 2023-08-05 00:57:42.486743600 +0000
Change: 2023-08-05 00:57:42.486743600 +0000
 Birth: 2023-08-05 00:46:54.031237800 +0000
$ sudo ntfsinfo -v -i 294 /dev/sdc1
Dumping Inode 294 (0x126)
Upd. Seq. Array Off.:    48 (0x30)
Upd. Seq. Array Count:   3 (0x3)
Upd. Seq. Number:        16 (0x10)
LogFile Seq. Number:     0x2012e38
MFT Record Seq. Numb.:   5 (0x5)
Number of Hard Links:    1 (0x1)
Attribute Offset:        56 (0x38)
MFT Record Flags:        IN_USE
Bytes Used:              448 (0x1c0) bytes
Bytes Allocated:         1024 (0x400) bytes
Next Attribute Instance: 4 (0x4)
MFT Padding:    00 00
Dumping attribute $STANDARD_INFORMATION (0x10) from mft record 294 (0x126)
        Attribute length:        96 (0x60)
        Resident:                Yes
        Name length:             0 (0x0)
        Name offset:             0 (0x0)
        Attribute flags:         0x0000
        Attribute instance:      0 (0x0)
        Data size:               72 (0x48)
        Data offset:             24 (0x18)
        Resident flags:          0x00
        ReservedR:               0 (0x0)
        File Creation Time:      Fri Aug  4 22:46:54 2023
        File Altered Time:       Fri Aug  4 22:57:42 2023
        MFT Changed Time:        Fri Aug  4 22:57:42 2023
        Last Accessed Time:      Fri Aug  4 22:57:42 2023
        File attributes:         ARCHIVE SPARSE_FILE (0x00000220)
        Maximum versions:        0
        Version number:          0
        Class ID:                0
        User ID:                 0 (0x0)
        Security ID:             538 (0x21a)
        Quota charged:           0 (0x0)
        Update Sequence Number:  2097127856 (0x7cffa1b0)
Dumping attribute $FILE_NAME (0x30) from mft record 294 (0x126)
        Attribute length:        176 (0xb0)
        Resident:                Yes
        Name length:             0 (0x0)
        Name offset:             0 (0x0)
        Attribute flags:         0x0000
        Attribute instance:      2 (0x2)
        Data size:               148 (0x94)
        Data offset:             24 (0x18)
        Resident flags:          0x01
        ReservedR:               0 (0x0)
        Parent directory:        293 (0x125)
        File Creation Time:      Fri Aug  4 22:46:54 2023
        File Altered Time:       Fri Aug  4 22:46:54 2023
        MFT Changed Time:        Fri Aug  4 22:46:54 2023
        Last Accessed Time:      Fri Aug  4 22:46:54 2023
        Allocated Size:          0 (0x0)
        Data Size:               0 (0x0)
        Filename Length:         41 (0x29)
        File attributes:         ARCHIVE (0x00000020)
        Namespace:               POSIX
        Filename:                'HDD + WINDOWS + NTFS + NO-PREALLOCATE.mkv'
Dumping attribute $DATA (0x80) from mft record 294 (0x126)
        Attribute length:        112 (0x70)
        Resident:                No
        Name length:             0 (0x0)
        Name offset:             0 (0x0)
        Attribute flags:         0x8000
        Attribute instance:      3 (0x3)
        Lowest VCN               0 (0x0)
        Highest VCN:             5659423 (0x565b1f)
        Mapping pairs offset:    72 (0x48)
        Compression unit:        4 (0x4)
        Data size:               23180943977 (0x565b12269)
        Allocated size:          23181000704 (0x565b20000)
        Initialized size:        58884096 (0x3828000)
        Compressed size:         38535168 (0x24c0000)
        Runlist:        VCN             LCN             Length
                        0x0             0xc8e00         0x30
                        0x30            0x377f          0xdb
                        0x10b           0xb68aa         0xf15
                        0x1020          <HOLE>          0x7e0
                        0x1800          0xb77bf         0x1470
                        0x2c70          <HOLE>          0xb90
                        0x3800          0xb8c2f         0x30
                        0x3830          <HOLE>          0x5622f0
End of inode reached
Total runs: 8 (fragments: 3)
superbonaci commented 1 year ago

HDD + WINDOWS + NTFS + DO-PREALLOCATE

Working as intended: NO

Steps to reproduce the issue:

  1. Do not start the download automatically: ticked
  2. Pre-allocate space for all files: ticked
  3. Add Torrent File - Open
  4. Select None
  5. Download in sequential order: ticked
  6. Save at: HDD (mechanical spinning) formatted in NTFS
  7. OK
  8. Select the torrent from the list and in lower tab Content select (tick) only one file. It should keep as Paused.

The file does not take space in the filesystem and is allocated instantly. The file is created.

HDD + WINDOWS + NTFS + DO-PREALLOCATE

As torrent is Resumed, the file used size (Size on disk) grows according to the number of Pieces that have been downloaded. After a few pieces are downloaded Linux reads this information:

$ du -sh *
55M     HDD + WINDOWS + NTFS + DO-PREALLOCATE.mkv
$ du -h --apparent-size *
22G     HDD + WINDOWS + NTFS + DO-PREALLOCATE.mkv
$ sudo ntfsinfo -v -i 289 /dev/sdc1
Dumping Inode 289 (0x121)
Upd. Seq. Array Off.:    48 (0x30)
Upd. Seq. Array Count:   3 (0x3)
Upd. Seq. Number:        6 (0x6)
LogFile Seq. Number:     0x2002ef1
MFT Record Seq. Numb.:   7 (0x7)
Number of Hard Links:    1 (0x1)
Attribute Offset:        56 (0x38)
MFT Record Flags:        IN_USE
Bytes Used:              448 (0x1c0) bytes
Bytes Allocated:         1024 (0x400) bytes
Next Attribute Instance: 4 (0x4)
MFT Padding:    00 00
Dumping attribute $STANDARD_INFORMATION (0x10) from mft record 289 (0x121)
        Attribute length:        96 (0x60)
        Resident:                Yes
        Name length:             0 (0x0)
        Name offset:             0 (0x0)
        Attribute flags:         0x0000
        Attribute instance:      0 (0x0)
        Data size:               72 (0x48)
        Data offset:             24 (0x18)
        Resident flags:          0x00
        ReservedR:               0 (0x0)
        File Creation Time:      Fri Aug  4 23:25:53 2023
        File Altered Time:       Fri Aug  4 23:32:47 2023
        MFT Changed Time:        Fri Aug  4 23:32:47 2023
        Last Accessed Time:      Fri Aug  4 23:32:47 2023
        File attributes:         ARCHIVE SPARSE_FILE (0x00000220)
        Maximum versions:        0
        Version number:          0
        Class ID:                0
        User ID:                 0 (0x0)
        Security ID:             538 (0x21a)
        Quota charged:           0 (0x0)
        Update Sequence Number:  2097130520 (0x7cffac18)
Dumping attribute $FILE_NAME (0x30) from mft record 289 (0x121)
        Attribute length:        176 (0xb0)
        Resident:                Yes
        Name length:             0 (0x0)
        Name offset:             0 (0x0)
        Attribute flags:         0x0000
        Attribute instance:      2 (0x2)
        Data size:               148 (0x94)
        Data offset:             24 (0x18)
        Resident flags:          0x01
        ReservedR:               0 (0x0)
        Parent directory:        288 (0x120)
        File Creation Time:      Fri Aug  4 23:25:53 2023
        File Altered Time:       Fri Aug  4 23:25:53 2023
        MFT Changed Time:        Fri Aug  4 23:25:53 2023
        Last Accessed Time:      Fri Aug  4 23:25:53 2023
        Allocated Size:          0 (0x0)
        Data Size:               0 (0x0)
        Filename Length:         41 (0x29)
        File attributes:         ARCHIVE (0x00000020)
        Namespace:               POSIX
        Filename:                'HDD + WINDOWS + NTFS + DO-PREALLOCATE.mkv'
Dumping attribute $DATA (0x80) from mft record 289 (0x121)
        Attribute length:        112 (0x70)
        Resident:                No
        Name length:             0 (0x0)
        Name offset:             0 (0x0)
        Attribute flags:         0x8000
        Attribute instance:      3 (0x3)
        Lowest VCN               0 (0x0)
        Highest VCN:             5659423 (0x565b1f)
        Mapping pairs offset:    72 (0x48)
        Compression unit:        4 (0x4)
        Data size:               23180943977 (0x565b12269)
        Allocated size:          23181000704 (0x565b20000)
        Initialized size:        157794304 (0x967c000)
        Compressed size:         57147392 (0x3680000)
        Runlist:        VCN             LCN             Length
                        0x0             0xb68aa         0x800
                        0x800           <HOLE>          0x4000
                        0x4800          0xb70aa         0x1000
                        0x5800          <HOLE>          0x1000
                        0x6800          0xb80aa         0x800
                        0x7000          <HOLE>          0x1000
                        0x8000          0xb88aa         0x1680
                        0x9680          <HOLE>          0x55c4a0
End of inode reached
Total runs: 8 (fragments: 1)
superbonaci commented 1 year ago

HDD + LINUX + NTFS + NO-PREALLOCATE

Working as intended: NO

Mount the NTFS partition on Linux, Dolphin automount uses this values:

/dev/sdc1 on /run/media/user/NTFS type ntfs3 (rw,nosuid,nodev,relatime,uid=1000,gid=1000,windows_names,iocharset=utf8,uhelper=udisks2)

Steps to reproduce the issue:

  1. Do not start the download automatically: ticked
  2. Pre-allocate space for all files: not ticked
  3. Add Torrent File - Open
  4. Select None
  5. Download in sequential order: ticked
  6. Save at: HDD (mechanical spinning) formatted in NTFS
  7. OK
  8. Select the torrent from the list and in lower tab Content select (tick) only one file. It should keep as Paused.

The file does take its full space in the filesystem and a takes a while to be allocated, basically maxing out the write speed of the drive. The file is created. After a few pieces are downloaded Linux reads this information:

$ du -sh *
2.4G    HDD + LINUX + NTFS + NO-PREALLOCATE.mkv
$ du -h --apparent-size *
2.4G    HDD + LINUX + NTFS + NO-PREALLOCATE.mkv
$ stat *
  File: HDD + LINUX + NTFS + NO-PREALLOCATE.mkv
  Size: 2470729058      Blocks: 4825648    IO Block: 4096   regular file
Device: 8,33    Inode: 308073      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/  user)   Gid: ( 1000/  user)
Access: 2023-08-05 01:50:46.835911500 +0000
Modify: 2023-08-05 01:50:46.835911500 +0000
Change: 2023-08-05 01:50:46.835911500 +0000
 Birth: 2023-08-05 01:50:46.835911500 +0000
$ sudo ntfsinfo -v -i 308073 /dev/sdc1
Dumping Inode 308073 (0x4b369)
Upd. Seq. Array Off.:    42 (0x2a)
Upd. Seq. Array Count:   3 (0x3)
Upd. Seq. Number:        1 (0x1)
LogFile Seq. Number:     0x0
MFT Record Seq. Numb.:   4 (0x4)
Number of Hard Links:    1 (0x1)
Attribute Offset:        48 (0x30)
MFT Record Flags:        IN_USE
Bytes Used:              520 (0x208) bytes
Bytes Allocated:         1024 (0x400) bytes
Next Attribute Instance: 5 (0x5)
MFT Padding:
Dumping attribute $STANDARD_INFORMATION (0x10) from mft record 308073 (0x4b369)
        Attribute length:        96 (0x60)
        Resident:                Yes
        Name length:             0 (0x0)
        Name offset:             0 (0x0)
        Attribute flags:         0x0000
        Attribute instance:      0 (0x0)
        Data size:               72 (0x48)
        Data offset:             24 (0x18)
        Resident flags:          0x00
        ReservedR:               0 (0x0)
        File Creation Time:      Fri Aug  4 23:50:46 2023
        File Altered Time:       Fri Aug  4 23:50:46 2023
        MFT Changed Time:        Fri Aug  4 23:50:46 2023
        Last Accessed Time:      Fri Aug  4 23:50:46 2023
        File attributes:         ARCHIVE (0x00000020)
        Maximum versions:        0
        Version number:          0
        Class ID:                0
        User ID:                 0 (0x0)
        Security ID:             291 (0x123)
        Quota charged:           0 (0x0)
        Update Sequence Number:  0 (0x0)
Dumping attribute $FILE_NAME (0x30) from mft record 308073 (0x4b369)
        Attribute length:        168 (0xa8)
        Resident:                Yes
        Name length:             0 (0x0)
        Name offset:             0 (0x0)
        Attribute flags:         0x0000
        Attribute instance:      1 (0x1)
        Data size:               144 (0x90)
        Data offset:             24 (0x18)
        Resident flags:          0x01
        ReservedR:               0 (0x0)
        Parent directory:        308072 (0x4b368)
        File Creation Time:      Fri Aug  4 23:50:46 2023
        File Altered Time:       Fri Aug  4 23:50:46 2023
        MFT Changed Time:        Fri Aug  4 23:50:46 2023
        Last Accessed Time:      Fri Aug  4 23:50:46 2023
        Allocated Size:          2470731776 (0x93446000)
        Data Size:               2470729058 (0x93445562)
        Filename Length:         39 (0x27)
        File attributes:         ARCHIVE (0x00000020)
        EA Length:               45 (0x2d)
        Namespace:               POSIX
        Filename:                'HDD + LINUX + NTFS + NO-PREALLOCATE.mkv'
Dumping attribute $DATA (0x80) from mft record 308073 (0x4b369)
        Attribute length:        80 (0x50)
        Resident:                No
        Name length:             0 (0x0)
        Name offset:             64 (0x40)
        Attribute flags:         0x0000
        Attribute instance:      2 (0x2)
        Lowest VCN               0 (0x0)
        Highest VCN:             603205 (0x93445)
        Mapping pairs offset:    64 (0x40)
        Compression unit:        0 (0x0)
        Data size:               2470729058 (0x93445562)
        Allocated size:          2470731776 (0x93446000)
        Initialized size:        2470729058 (0x93445562)
        Runlist:        VCN             LCN             Length
                        0x0             0x4dc3fc8               0x93446
Dumping attribute $EA_INFORMATION (0xd0) from mft record 308073 (0x4b369)
        Attribute length:        32 (0x20)
        Resident:                Yes
        Name length:             0 (0x0)
        Name offset:             24 (0x18)
        Attribute flags:         0x0000
        Attribute instance:      3 (0x3)
        Data size:               8 (0x8)
        Data offset:             24 (0x18)
        Resident flags:          0x00
        ReservedR:               0 (0x0)
        Packed EA length:        45 (0x2d)
        NEED_EA count:           0 (0x0)
        Unpacked EA length:      60 (0x3c)
Dumping attribute $EA (0xe0) from mft record 308073 (0x4b369)
        Attribute length:        88 (0x58)
        Resident:                Yes
        Name length:             0 (0x0)
        Name offset:             24 (0x18)
        Attribute flags:         0x0000
        Attribute instance:      4 (0x4)
        Data size:               60 (0x3c)
        Data offset:             24 (0x18)
        Resident flags:          0x00
        ReservedR:               0 (0x0)

        EA flags:                NONE
        Name length:     6 (0x6)
        Value length:    4 (0x4)
        Name:            '$LXUID'
        Value:           0xe8030000

        EA flags:                NONE
        Name length:     6 (0x6)
        Value length:    4 (0x4)
        Name:            '$LXGID'
        Value:           0xe8030000

        EA flags:                NONE
        Name length:     6 (0x6)
        Value length:    4 (0x4)
        Name:            '$LXMOD'
        Value:           0xa4810000
End of inode reached
Total runs: 1 (fragments: 1)
superbonaci commented 1 year ago

HDD + LINUX + NTFS + DO-PREALLOCATE

Working as intended: YES

  1. Pre-allocate space for all files: ticked

Same result as "HDD + LINUX + NTFS + NO-PREALLOCATE". Selected files are pre-allocated to full size and takes as much time as big the files are.

superbonaci commented 1 year ago

I could format the HDD to different filesystems (ETX4, APFS, ZFS, XFS, exFAT) and try too with SSD drives. Try too qBittorrent on MAC, there will be outcomes different from desired. This option needs lots of improvements, further explanations and expand the documentation to cover every case.

Juraj-Masiar commented 1 year ago

I've tried the exFAT on SSD and it will pre-allocate disk space in both Windows and Linux (with disabled pre-allocation).

superbonaci commented 1 year ago

I've tried the exFAT on SSD and it will pre-allocate disk space in both Windows and Linux (with disabled pre-allocation).

@Juraj-Masiar did your torrent auto-start? Make sure you've selected Do not start automatically both in Settings and when opening the torrent.

I think there's also a whole confusion between pre-allocating the file and creating the file. Creating the file means that the file "appears" in the filesystem. While pre-allocating means that it takes the whole space instead of 0-bytes, you have to compare du -sh file against du -h --apparent-size file to see if they give the same or not to really know if pre-allocation is working for you.

The documentation is not explaining this difference either.

For some reason my last post disappeared. I'll have to repeat it.

Juraj-Masiar commented 1 year ago

I'm downloading huge files (between 15GB and 50GB) and I monitor the SSD usage. So if I see SSD writing 500MB/s for first few minutes (while downloads drops to 0MB/s), I know it's pre-allocating the space and then I see the actual download begins (well, it first downloads a few MB to memory, which then tries to write to drive and that causes the whole pre-allocation which freezes the download until the allocaiton is finished).

EDIT: I forgot to mention - I can confirm your findings regarding working disabled pre-allocation: Windows + NTFS Linux + ext4

And a bonus experiment - if you start download in Windows and NTFS and let it create all files (without finishing download) and then move the drive to Linux and continue the download, it will first check the files and then when the download starts, with the first write operation it will again incorrectly "fills-up" the rest of the file.

superbonaci commented 1 year ago

And a bonus experiment - if you start download in Windows and NTFS and let it create all files (without finishing download) and then move the drive to Linux and continue the download, it will first check the files and then when the download starts, with the first write operation it will again incorrectly "fills-up" the rest of the file.

I'm assuming that what you did, is to add the torrent also in Linux and set the same destination to the same NTFS folder, and you did not tick Skip hash check. In case you had skipped the hash check, I assume it would set the download as 0% completed and overwrite it with empty data once started. I'm wondering if the NTFS driver for Linux does not support sparse files, or is a bug in another torrent related library, instead of qBittorrent itself.

I made another test with Windows + exFAT. It does not create any file if you select Options - Downloads - Do not start the download automatically. When adding the torrent make sure Start torrent is not ticked. You can tick all files to be downloaded in that dialog. It should not create any file file in the exFAT filesystem unless:

I've tried the exFAT on SSD and it will pre-allocate disk space in both Windows and Linux (with disabled pre-allocation).

exFAT does not support sparse files, so my take is that when the file is created it will always be pre-allocated to full size (take up all the space), not matter the operating system.

Juraj-Masiar commented 1 year ago

I'm wondering if the NTFS driver for Linux does not support sparse files, or is a bug in another torrent related library, instead of qBittorrent itself.

That is indeed a good question! The docs for ntfs-3g says it does support it. Anyway, it would be really nice to have it fixed. Since Windows can't read ext4, you can only use NTFS or exFAT if you want to use the drive on both systems.

And you are right, I've just checked that exFAT indeed doesn't support sparse files... So that leaves only NTFS.

My motivation here is to avoid writing 100GB data to SSD when downloading 50GB file. Plus the huge time reduction if you are using slower drive like flash drive or HDD.

glassez commented 1 year ago

libtorrent is responsible for discussed things so could you move this Issue to libtorrent side?

superbonaci commented 1 year ago

Issue created there, so from now on all comments go there. The last issue I've found, is that 0-byte files are always created when adding a new torrent, even if adding it as paused (if they are ticked during the import dialog). I guess they do it since you don't need to download anything, and the file name is already included in the torrent, so the file is created in the filesystem instantly. That's a different issue and is already been reported: https://github.com/qbittorrent/qBittorrent/issues/18859

r3a1d3a1 commented 6 months ago

~~As a side-note for the devs: Please mention in the GUI's tooltip for pre-allocation that it's: "good for avoiding outa space situations and improving speed on HDDs but bad for SSDs longevity", as compared to the current tooltip. (Unless maybe you could add an option to have it enabled only for HDDs so that the user with multiple types of storage wouldn't have to turn it on and off depending on the destination folder's storage type.)~~

r3a1d3a1 commented 6 months ago

Found out it uses 'falloc' only (no zeroing) and fails (miserably) on filesystems that don't support sparse files, e.g. exFat. On that basis, it shouldn't even be an option. Because if it were smart about when to use it, there'd be no downsides to it that a user might not be interested in. I.e. apply 'falloc' whenever you can, and don't do so when not supported.

It'd only make sense to have it as an option if other modes like zero'ing gets supported by libtorrent which some users may not be interested in due to extra time taken, or extra writes to a SSD.

glassez commented 6 months ago

@r3a1d3a1 https://github.com/qbittorrent/qBittorrent/issues/19410#issuecomment-1669798747

r3a1d3a1 commented 6 months ago

It's much harder to get things done in libtorrent than in QB, so I think QB can have a smart wrapper for the pre-alloc option.

HanabishiRecca commented 1 month ago

I actually discovered it independently in arvidn/libtorrent#7732.

It has nothing to do with filesystems etc. A simple bug in libtorrent on file prioriy changes. Already fixed by arvidn/libtorrent#7738, but not yet released.

thalieht commented 1 month ago

Already fixed by https://github.com/arvidn/libtorrent/pull/7738, but not yet released.

In Windows, the maintainer usually uses the latest commit for each libtorrent branch and if you check the git commit of libtorrent that is used in 5.0.1, it is included (https://www.qbittorrent.org/news, Library versions). I didn't check 5.0.0.

HanabishiRecca commented 1 month ago

if you check the git commit of libtorrent that is used in 5.0.1, it is included

Ok then, this issue for Windows users could be closed, I guess. We could keep my generic one #21306 until libtorrent 2.0.11.