moving storage starves other processes of data, can't watch movies

[cheater]

Please provide the following information

libtorrent version (or branch): I don't know (via qBittorrent v4.4.3.1, i.e. latest)

platform/architecture: Windows 10 Pro latest

compiler and compiler version: I don't know

please describe what symptom you see, what you would expect to see instead and how to reproduce it.

qBittorrent allows you to have a temporary folder and a finished folder.

I'll repost my whole report to qBittorrent, and then add remarks.

I have a scratch SSD for my temporary folder and a HDD for my completed folder. If I am watching e.g. a movie from the completed folder, and a large file (>5 GB) finishes downloading, the completion moving uses all available resources of the HDD and I cannot continue watching the movie until the moving completes several minutes later, which is annoying.

Solution: A) use some sort of operating system level QoS if that sort of thing exist. I don't know that it does.

B) in the settings, for every disk drive, add a max total read/write speed (two separate numbers), which will be respected by all operations, with a checkbox that says "except downloading" - in which case downloading is not limited. (seeding is rarely able to be fast enough to choke a hard drive, so no need for an exception here)

I am not sure that completion moving itself is done by libtorrent, can you confirm?

Thanks

There are two solutions - if A can be implemented, then it can happen transparently; if B is implemented, please could you allow configuration like described so that qBittorrent can set it.

It has been confirmed on the qBittorrent bug tracker that file moving happens via libtorrent.

Thanks and best regards!

[glassez]

I have a scratch SSD for my temporary folder and a HDD for my completed folder. If I am watching e.g. a movie from the completed folder, and a large file (>5 GB) finishes downloading, the completion moving uses all available resources of the HDD and I cannot continue watching the movie until the moving completes several minutes later, which is annoying.

I wonder what happens if you do the same thing with the exception that you manually (e.g. from command line or Explorer) start moving a large file from your SSD to HDD?

[cheater]

Similar, but not quite. The movie I tried is high bitrate (~3 MB/sec) and while it started dropping out (or rather, freezing for a little), it's not quite as bad as when qBittorrent is moving something over. I can see the transfer speed on the graph of the copy dialog dropping a little from its typical 145 MB/sec when I seek the movie in the player.

However, that's less of an issue, because I consciously elected to do a file copy at that time - whereas torrents happen in the background, so it would be better if they're not disruptive when that happens.

[cheater]

It occurs to me that this might also be detrimental for torrents being seeded, in which case throttling this just a little should increase swarm transfer speeds globally.

[the8472]

Windows supports IO priorities, have you tried setting qbt to a lower one? I think lowering the process CPU priority (can be done with the task manager or start.exe) also implicitly lowers the IO priority. In case I'm wrong you can try using process explorer which can set IO priority explicitly.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[cheater]

unstale, this is still an issue

[the8472]

have you tried setting IO priorities?

[cheater]

No, what would you set it to?

[the8472]

something lower than normal

[ghost]

Set aio_threads to 4 and hashing threads to 1.

[cheater]

the issue isn't with cpu contention - it is with io to a mechanical drive being fully saturated. do you still think this would be advisable, @summerqB ? and if so, why, what would it do? thanks

[ghost]

The fact that libtorrent is using multi threaded disk I/O to move files and the other apps are single threaded is the reason why libtorrent increases the I/O load. You should use less aio and hashing threads so that you have less disk I/O threads taking up all resources.

[cheater]

@summerqB ok but the disk saturation happens when merely one file is being moved. why does it matter how many threads libtorrent uses? will it somehow be using multiple threads for the file copy operation? i'm not sure about this.

[ghost]

If the app has multiple threads it could be using one for the file copy operation and another for reading or writing some other data to/from the disk. This increases overall I/O load of a disk. And I'm not sure if file copy operations are done using a single thread or not but you should give it a try. It won't bite.

[cheater]

no, this is specifically triggered by one file being copied.

[cheater]

in addition, i don't need to be seeding any files from that same drive for the issue to happen.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[cheater]

@arvidn could you please reopen this issue? it's still there, still very annoying.

[cheater]

thanks arvidn! just to push this forward a little bit, I wanted to propose a simple design:

• set a global configuration setting that limits the relocation transfers to a specific transfer rate
• when initiating a file move, pass in a transfer rate that can override this; 0 means global rate; negative number means no limit.
• when moving between the same file system, do not use a limit at all

[glassez]

@cheater How do you propose libtorrent to respect such a rate limit? Don't use system function to move files but "manually" perform data copying by some amount at a time?

[cheater]

yes, exactly that. mmap the files and go in 4M chunks.

[cheater]

@glassez CopyHandler is gpl, you could look at how they do things. https://fisheye.copyhandler.com/browse/CopyHandler/License.txt?hb=true

[cheater]

btw, regarding earlier discussion of "io priority", it should be said that that doesn't really exist on Windows. The only way to make this work is to throttle the transfer speed by hand.

[cheater]

btw... strongly suggest looking at CopyHandler source, as in their FAQ they have metnions of older versions having bugs that would result in data loss, so if they learned from those mistakes, you should learn from their mistakes as well!

[the8472]

btw, regarding earlier discussion of "io priority", it should be said that that doesn't really exist on Windows. The only way to make this work is to throttle the transfer speed by hand.

Setting a thread priority to THREAD_MODE_BACKGROUND_BEGIN will also lower its IO priority. Additionally an IO priority can be set on individual file handles

[cheater]

There's nothing in the documentation to indicate that "IO priority" will be lowered, or that it will limit transfer speeds when other things are using the same disk drive. Quotes because there's no mention of anything even similar io priority at all on that page either.

[cheater]

The closest thing to that is the following:

For threads that perform background work such as file I/O, network I/O, or data processing, it is not sufficient to adjust the CPU scheduling priority; even an idle CPU priority thread can easily interfere with system responsiveness when it uses the disk and memory. Threads that perform background work should use the THREAD_MODE_BACKGROUND_BEGIN and THREAD_MODE_BACKGROUND_END values to adjust their resource scheduling priorities; threads that interact with the user should not use THREAD_MODE_BACKGROUND_BEGIN.

but it isn't really said whether this actually changes some sort of IO priority or not.

[the8472]

CPU thread priorities also affect the default IO priorities, I assume it is done that way so that old programs that only set their thread priorities also automatically get their IO prio adjusted. https://bitsum.com//pl_io_priority.php

[cheater]

Interesting, if the copying got split off to another thread and the IO priority got set like that, then that would be worth a try without having to reimplement file copying. Thanks for bringing this up.