File checking speed for 4.4.0 is much slower than 4.3.9

[Coresi7]

qBittorrent & operating system versions

qBittorrent: 4.3.9 x64 Operating system: Windows server 2022 datacenter 21h1 (10.0.20348) libtorrent-rasterbar: 1.2.14 qt: 5.15.2

What is the problem?

qbittorrent v4.4.0 with qt6 file check speed is extremely slow, compare to qbittorrent v4.3.9. About 2x slower for hdd, or 10x slower for samba protocol. require for a fix, thanks!

Steps to reproduce

No response

Additional context

No response

Log(s) & preferences file(s)

No response

[microka]

qbittorrent_4.4.3.1_qt6_x64, default settings, hashing a big single file on SSD hashing threads set to 1, about 235MB/s & 20% CPU hashing threads set to 10, about 288MB/s & 75% CPU

but my qbittorrent_4.3.9_x64 hashing speed can be 650MB/s & 15% CPU.

So i can say 4.4.3.1 is still have this problem, 4.3.9 is still a good choice.

[ghost]

qbittorrent_4.4.3.1_qt6_x64, default settings, hashing a big single file on SSD hashing threads set to 1, about 235MB/s & 20% CPU hashing threads set to 10, about 288MB/s & 75% CPU

but my qbittorrent_4.3.9_x64 hashing speed can be 650MB/s & 15% CPU.

So i can say 4.4.3.1 is still have this problem, 4.3.9 is still a good choice.

How can hashing speed be in MB/s? The disk IO subsystem is completely different in 4.4.3.1. You should measure the time in seconds not disk read speeds!

[ESPGT]

Yes, in 4.4.3.1 qt5 and qt6 very slow speed creation torrent and also very slow speed checking

[Pentaphon]

So i can say 4.4.3.1 is still have this problem, 4.3.9 is still a good choice.

I really can't believe this issue was closed. This tells you a lot about how qBIttorrent is being managed. Adding more without actually fixing everything that needs to be fixed. So many people can see that 4.4.x has major issues.

[ESPGT]

utorrent create torrent file for 3 minute, but qbit need 1 hour and hour to check this! CPU very high also. Please fix this Video

[ghost]

Just to make everything clear, this is an upstream issue in the libtorrent library probably due to the use of memory mapped files. The issue is not reproducible with 4.4.3.1 built against libtorrent RC1_2. If anything needs fixing that has to be done on the library not here! The hashing thread setting was just compouding the issue for some users, but setting it to 1 does not solve it for all storage solutions and does not offer the same speed as RC1_2. Just keeping this open till it gets fixed upstream. FYI, POSIX compliant disk I/O subsystem can be toggled in master branch. You may try that meanwhile to see if that fixes the problem.

[arvidn]

for future reports on this issue, please include the libtorrent version. the most recent release made improvements. The most recent version in git (unreleased still) has made further improvements. On my ubuntu machine, using latest version from git, both checking and creation is on par with libtorrent 1.2.x (both on SSD and HDD). There are many variables involved though. checking or creating torrents on a network drive is likely still slower on libtorrent 2.0.x

[baccccccc]

In my case, the difference between 4.3.9 and 4.4.0 was not noticeable at all, so it might not be the same issue. However, starting with 4.4.2 (up to, and including 4.4.3.1) is night and day difference. 4.4.2+ have been painfully slow.

Up until, and including, 4.4.1, the bottleneck while checking has always been my disks (be it the SSD or HDD.) As a result, the CPU was mostly free, and the app overall was snappy and responding.

Now, starting with 4.4.2, the CPU is almost always at 100% during checking, and the disks are only moderately loaded. (Typically, under 70%.) As a result, the whole app is frequently unresponsive and unusable. (It does return to normal, once the checking is complete, so it's not the end of the word.)

I do not know the libtorrent version that corresponds to all those qBittorrent versions, but I'm only using official releases, so it should be known which version uses what.

I can also say that Qt version is out of the question, as I was always using Qt6 builds starting from 4.4.0 betas. The number of hashing threads has been set to 1 long ago (when I tried to minimize the disk load) so it does not fix the CPU problem for me.

The OS is the latest Windows 11 (have been the original version 21H2 until I upgaded to 22H2 very recently.) The filesystem is local, no network is involved.

[ghost]

In my case, the difference between 4.3.9 and 4.4.0 was not noticeable at all, so it might not be the same issue. However, starting with 4.4.2 (up to, and including 4.4.3.1) is night and day difference. 4.4.2+ have been painfully slow.

Up until, and including, 4.4.1, the bottleneck while checking has always been my disks (be it the SSD or HDD.) As a result, the CPU was mostly free, and the app overall was snappy and responding.

Now, starting with 4.4.2, the CPU is almost always at 100% during checking, and the disks are only moderately loaded. (Typically, under 70%.) As a result, the whole app is frequently unresponsive and unusable. (It does return to normal, once the checking is complete, so it's not the end of the word.)

I do not know the libtorrent version that corresponds to all those qBittorrent versions, but I'm only using official releases, so it should be known which version uses what.

I can also say that Qt version is out of the question, as I was always using Qt6 builds starting from 4.4.0 betas. The number of hashing threads has been set to 1 long ago (when I tried to minimize the disk load) so it does not fix the CPU problem for me.

The OS is the latest Windows 11 (have been the original version 21H2 until I upgaded to 22H2 very recently.) The filesystem is local, no network is involved.

Your issue could be related to working set size limit introduced in 4.4.2. When checking files qBt is most likely hitting this limit and causing GUI to become nonresponsive due to not having adequate free RAM. I’m wondering if the kernal prioritizes the checking file cache over the GUI cache and thus removing the GUI cache and not leaving anything for it.

@glassez is there anyway to ensure that the main process always has some reserved memory that doesn’t get flushed when the working set limit gets hit? What I can see is that when you’re downloading at high speed or checking files the main process memory usage becomes lower than it started with and the GUI becomes less responsive.

[ghost]

I think the working set size limit only fixed the high RAM usage issue in a bad way since libtorrent itself can’t impose any memory limit due to the memory mapped files so it leaves nothing for the main process to store in memory thus increasing the CPU usage.

[glassez]

What I can see is that when you’re downloading at high speed or checking files the main process memory usage becomes lower than it started with and the GUI becomes less responsive.

What is the "main process" you're talking about? AFAIK, qBittorrent is single process.

[ghost]

I mean the thread that paints/processes the GUI events.

[glassez]

I mean the thread that paints/processes the GUI events.

As another desperate attempt to fix something, we could increase the memory priority of the main thread.

[Pentaphon]

Has anybody tested libtorrent 2.0.7 yet?

[Safari77]

What I can see is that when you’re downloading at high speed or checking files the main process memory usage becomes lower than it started with and the GUI becomes less responsive.

What is the "main process" you're talking about? AFAIK, qBittorrent is single process.

$ grep ^Thr /proc/580146/status
Threads:    22

That said, when checking torrent after download is finished, GUI update frequency is two times per minute. rb_libtorrent 2.0.7 and qB latest git.

[Pentaphon]

As another desperate attempt to fix something, we could increase the memory priority of the main thread.

Or we can go back to using libtorrent 1.2.x for qBittorrent 4.x until qBittorrent 5.x is ready to go.

The userbase is clearly not seeing the benefit of using libtorrent 2.x with all these speed and responsiveness issues and many of them are going back to 4.3.9 because of it and we still have 4.5.x which will be another 2 years or more of dealing with a buggy libtorrent 2.x

libtorrent 2.x should go on an extended testing branch until we figure out all the remaining major issues with libtorrent, which probably won't be until 2.1.x as the rate things are going.

[linax3-genuine]

i dont plan to ever move from 4.3.9. i cant trust the new developers of qbt about anything and im very disappointed, by what i see (or do not) from them.

[Pentaphon]

i dont plan to ever move from 4.3.9. i cant trust the new developers of qbt about anything and im very disappointed, by what i see (or do not) from them.

A lot of people are saying the same thing in many different places across the web, especially people who have lots of torrents to manage.

If the upcoming 4.4.4 simply went back to using libtorrent 1.2.x, most of the issues would be solved for the time being and 2.x can be investigated by a smaller part of the community until it is ready for production releases.

[arvidn]

would anyone be interested in giving this a try? https://github.com/arvidn/libtorrent/pull/7013

it's in master because the disk code has diverged quite a bit from 2.0

[microka]

i dont plan to ever move from 4.3.9. i cant trust the new developers of qbt about anything and im very disappointed, by what i see (or do not) from them.

Totally agree with you.

[glassez]

would anyone be interested in giving this a try? arvidn/libtorrent#7013

:+1: Will do.

[ghost]

Please test v4.4.4, it should take care of the GUI stuttering issue.

[DanielSmedegaardBuus]

It seems there may have been some resolution on this issue? I'm experiencing something similar, except it's with file moving operations after doing an apt upgrade on my Linux Mint box.

Except here, libtorrent is 1.2.14+git20211025.eb4bbfd49c-1ppa1~20.04 and I'm running qbittorrent-nox at 4.3.9.99~202110311443-7435-01519b5e7~ubuntu20.04.1. But I feel like this started happening after I apt upgraded. I rarely upgrade (because increasingly, I find that when upgrading I get problems rather than fixes to problems 😄), but this upgrade also started some other bugs with qbittorrent, so I'm thinking this slow moving of torrents I'm now experiencing might be because of the upgrade.

I started the moving of a 117GB torrent this morning, from one USB drive to another. It had been in the moving state for a while when I went downstairs, did some cleaning, started a two-and-a-half-hour wash cycle, and after that completed, made lunch and ate it, then petted the cat and watched a bit of YouTube. I'd say somewhere between four and five hours passed, and when I returned upstairs, qbittorrent was still moving the torrent. It had moved around 84GB in this time. At the very best, that's still less than 6 MB/s.

I've shut down qbittorrent and started rsync'ing instead, and it's been running along at ~41MB/s since then, currently having copied around 44GB in the time it's taken me to search the qbittorrent bugs list and read most of this thread and writing this response.

I'm just wondering if there's any reason to believe that this behavior in moving slowness could perhaps be tied to whatever the problem was that caused this problem with checking slowness? I'd be interested to hear if there was some kind of fix I could apply here, even a downgrade if that's possible.

Cheers 😃

[DanielSmedegaardBuus]

@PriitUring

Well, I know these things, because I wrote them in my comment which is why you know them, and why you can regurgitate them back at me asking me if I'm aware of it, very redundantly and quite rudely 😆

I was simply asking if whichever discovery had been done might explain this other behavior as they seemed related.

But it doesn't actually matter, as I was able to downgrade to the previous version via downloaded debs and this issue is now resolved for me, and the new issues that I've been experiencing seem to have gone away as well. I'll just stick with that version.

Cheerio

[linax3-genuine]

as he reported an issue, about reduced speed of file operations, id think this is worthy of being investigated, as related to the issue here.

id remind my case was related with inappropriate usage of the storage device(HDD). and other people observed slowness on other types of storage devices. id also wonder, if the RAM is also used properly at this point, in which due its speed, is harder to observe slowdowns.

[DanielSmedegaardBuus]

But it doesn't actually matter, as I was able to downgrade to the previous version via downloaded debs and this issue is now resolved for me, and the new issues that I've been experiencing seem to have gone away as well. I'll just stick with that version.

Actually, in case someone encounters this same issue, it actually wasn't resolved by downgrading as I thought and stated above. I assumed so because I used the downgraded qBittorrent version to move the torrent in question to a different drive (One of 12 identical drives (same model), all with NTFS on Linux) and the speed here was normal again.

I'm now again seeing the same speed issues moving a huge torrent from an ext4 scratch CMR drive to another one of my NTFS SMR drives. It's been moving for about six hours and is only about 93GB into the operation. Again, copying the same torrent in terminal is magnitudes faster, so I'm currently starting to suspect that qBittorrent or libtorrent may be either using synchronous writes when moving (but the documentation I can find seems to state this is not the case), or doing some periodic flushing that is causing the drive to enter shingle RMW hell.

But, that's a discussion for a different thread. I just wanted to report back so that there's no lingering question of whether this check speed issue is related to my move issue. I'll stay out of this thread now.

Cheers 🍻

[linax3-genuine]

this is the only thing that i understood, after anything you have said: "I'll stay out of this thread now.". i hope the linux, or other people understood you better.

[DanielSmedegaardBuus]

this is the only thing that i understood, after anything you have said: "I'll stay out of this thread now.". i hope the linux, or other people understood you better.

Oh, wow. That's some next-level poor social skills there. You do realize that you're on a Github issues board, right? This is a place for developers (like myself) and users (also like myself in this case) to interact on work on issues with the code. So when you read something you don't understand, you don't have to tell everyone about it. Better to just ignore it. As someone once said, "Better to remain silent and be thought a fool than to open your mouth and remove all doubt."

As for the general attitude, cheers on adding to the thread's — and by extension the Issues section's — apparent toxicity. I hope you get happier and better at interacting, I really do. It'll be better for you and the people that come in contact with you.

Cheers

[baccccccc]

Please test v4.4.4, it should take care of the GUI stuttering issue.

Unfortunately, I don't see much improvement here. GUI performance still noticeably decrease over time, and I have to restart the app 1-2 times a day. If I don't, then after 2+ days it becomes completely unresponsive. In addition, all 8 cores of my CPU keep at almost 100% all the time. (Needless to say, that's not the case when qBitTorrent is not running.)

Is there any setting I have to change?

(64-bit qBittorrent v.4.4.4 official build, Qt 6.3.0 and Libtorrent 2.0.7.0. Windows 11 x64 on a VM, using 8 cores CPU and 14 GB dedicated RAM. Hashing threads option is set to 1.)

[glassez]

@qbittorrent/bug-handlers Could someone test https://github.com/arvidn/libtorrent/pull/7033#issuecomment-1230498096 with actively seeding a lot of data and unlimited working set (e.g. set it to maximum amount of memory you have)? I am interested to know about the responsiveness of the system when most of the memory is occupied by clean pages of mapped files. I guess this could still be a problem, albeit not the same as having dirty pages.

EDIT: How about create a separate Issue about testing new/changed disk I/O variants of libtorrent?

[Pentaphon]

Now that 4.4.5 went back to v1, how's everybody's performance on that release?

[mazzz1y]

Now that 4.4.5 went back to v1, how's everybody's performance on that release?

the same for me

my fault, I compilled it with libtorrent 2.0

[xinmans]

Same issue here， i use 4.5.0, but checking very slow, what happened?

qBittorrent v4.5.0 Web UI (64-bit) Qt: | 6.4.1 Libtorrent: | 2.0.8.0 Boost: | 1.81.0 OpenSSL: | 3.0.7 zlib: | 1.2.13

[Pentaphon]

i use 4.5.0, but checking very slow, what happened?

Use the libtorrent 1.2 version instead.

[xinmans]

i use 4.5.0, but checking very slow, what happened?

Use the libtorrent 1.2 version instead.

which docker image https://hub.docker.com/r/linuxserver/qbittorrent/tags?page=1 ？

this one：4.5.0-libtorrentv1 ？

[Pentaphon]

this one：4.5.0-libtorrentv1 ？

I think so? Anything labeled v1.2.18 instead of v2.0.8

[xinmans]

this one：4.5.0-libtorrentv1 ？

I think so? Anything labeled v1.2.18 instead of v2.0.8 Yes， but checking speed seems not faster

Qt: | 6.4.2 Libtorrent: | 1.2.18.0 Boost: | 1.81.0 OpenSSL: | 3.0.7 zlib: | 1.2.12.zlib-ng

[IRainman]

It's probably already fixed by added additional buffer to the hasher?

[Safari77]

It's probably already fixed by added additional buffer to the hasher?

All downloads are paused while the files are being checked. Maybe it needs a redesign.

[yueisme]

https://github.com/qbittorrent/qBittorrent/issues/16043#issuecomment-1112293856

change aio_threads can't feel effective, i recommend setting hashing_threads to 1 for HDD, higher for SSD.