Torrent with very small files locks up IO by filling up the hard drive queue length (inefficient IO handling?)

TomArrow commented 2 years ago

qBittorrent & operating system versions

qBitTorrent: 4.4.4 (newest update, updated today) OS: Win 7 Ultimate x64 (6.1.760x)

What is the problem?

Torrent file this happens with: problematic torrent.zip

Trying to download this torrent onto my external docking station with 2 HDDs ends up completely freezing anything relating to accessing my hard drives. It's an ~8GB torrent file with 40k+ files and a piece size of 512 KB. Some files are as small as 10 KB.

My hard drives stop responding to anything (opening folders or opening files) and in Resource Monitor I can see that the queue length is 50 and not budging. Only solution is to force kill the qbittorrent process AND THEN waiting a few minutes until the process actually disappears.

Now, I understand that this problem may not be entirely avoidable in principle, however what really made me create this bug report is that with rtorrent I was able to download this torrent without problems (although the GUI failed to show the list of files because tehy are too many I presume.

After downloading this entire torrent with rutorrent, I copied all the files over into my qbittorrent download folder, replacing the old attempt. However about 18 files failed to copy. They were all tiny files around 10 KB. So when I force rechecked the torrent (which worked fine by the way), it reached 99.9%. In the General tab, it showed that there was exactly one piece missing. Remember, each piece is 512 KB.

So I started the torrent again to download only the last missing piece. And it once again locked up my hard drive. This time I let it finish so it would be done with, and it took like 10 minutes until the torrent finally finished and my hard drives unfroze (plural hard drives since both the drives in the docking station freeze when one freezes).

During the 10 minutes or so that it took to finish, the queue length was once again up at the maximum of 50.

Now, it was about 18 files that were missing, all within a single 512 KB piece. The entire piece probably contains slightly above 50 files I'd presume. On occasion during the process, the drives would briefly unfreeze (queue length going down again), and then immediately this would start all over again, up to 50. What this makes me suspect is that every single time a single 10 KB file is written, the entire piece is rehashed or something? Causing an insane amount of read/write requests?

In my mind, downloading a single 512 KB piece and writing 50+ files around 10 KB size should NOT freeze up an entire hard drive due to being flooded with IO requests.

Steps to reproduce

I don't know tbh. Some hard drives/docking stations might be more tolerant than others and internal drives might not have the problem altogether, who knows.

Here's my settings in qbittorrent:

OS Caching is ON (but turning it off doesn't change this)
Async IO threads was limited to 1, no change
Hashing threads was set to 5, but I also tried 1, either way this seems to be for force rechecking which had no issues.

In any case, the problem gets worse as the torrent progresses (I turned it to sequential download). There are 3 subfolders in the torrent. The first and thrid managed to download somewhat okay, and then the middle one (just in the default sort order) finished to some percentage when this really started to become bad. I think that was the point it reached the section with the 10KB files presumably.

Additional context

No response

Log(s) & preferences file(s)

No response

stalkerok commented 2 years ago

Normally works. 4.4.4_RC_1_2_qt5_x64

fanjing-whu commented 1 year ago

I have the same issue. However, the reason is 'Validate HTTP tracker certificates' in a high latency network. Maybe, you can try this. Disable 'Validate HTTP tracker certificates' https://github.com/qbittorrent/qBittorrent/issues/15623

qbittorrent / qBittorrent