Having performance issues with 9k torrents & external HDD's

[AvangionQ]

qBittorrent & operating system versions

qBittorrent 4.6.3 on Windows 10

What is the problem?

I have nearly 9000 torrents running on qBittorrent. When I start qBittorrent, it starts taking up RAM, up to around 15GB after about an hour. When qBittorrent reaches 12 to 15GB RAM usage, the speed collapses. The disk cache is set to 23GB, so I have no idea why this is happening, but it's been occurring for a while. I've upgraded my RAM from 16GB to 64GB and added a new PCI/USB card with the hope that it would fix, but the problem remains. I have been intermittently manually restarting qBittorrent to fix the problem.

PS, first time posting one of these, not sure what settings files you want, to see if the problem is there.

Steps to reproduce

Load 9000 torrents, wait an hour.

Additional context

No response

Log(s) & preferences file(s)

No response

[AvangionQ]

I don't know if this might be useful, but:

[serpen7]

I have nearly 9000 torrents running on qBittorrent. When I start qBittorrent, it starts taking up RAM

Dude how do you think that even with inactive and stalled torrents when they starting to go active and etc will they demand separate resources? You are trying to be data center server with it's powers with such huge amount of torrents which demands resources. To be precise you will run out of resources if you will activate them all from "sleeping but ready to be active" mode if you will not limit the resources it could take up to.

[AvangionQ]

The problem isn't the memory usage, it's the sudden speed collapse after qBittorrent reaches 12 to 15 GB. It happens every time and takes about an hour, sometimes faster than that, before I have to restart the program. As soon as I restart qBittorrent, speed normalizes to 200mbps for ten minutes, then halves every ten minutes after that. When I see the speed is going less than 10mbps, I restart qBittorrent to temporarily fix it, but this is getting annoying. Yes, 45 active torrent transfers is about the usual, though I've seen it go as high as 100 active at a time. Having 9000+ torrents loaded doesn't mean all 9000 of them will be in use at the same time, that would be nuts.

My system has an SSD boot, three internal hard drives and seven external hard drives. None of the drives show any issue. USB cards indicate 3.0 and 3.1, but I wouldn't be able to get 200mbps transfers for ten minutes if they were the problem.

PS, I previously had 16GB and set qBittorrent to 10GB max usage. I recently upgraded to 64GB and set 23GB max usage.

qBittorrent.zip qbittorrent.log.zip

[AvangionQ]

Makes me wish I could figure out why the speed keeps crashing. Whenever I restart qBittorrent, I get a good ten minutes of 200mbps.

[AvangionQ]

Thank you for the effort. I've changed the specific settings according to your recommendations. Sadly, there's no change in speed crash in under an hour. I have no idea what's wrong here. One thing, how do you reset to default settings? I don't see that in the options menu.

[glassez]

As soon as I restart qBittorrent, speed normalizes to 200mbps for ten minutes, then halves every ten minutes after that

Judging by the symptoms described, the problem is the I/O speed of your disk, which contains the contents of the torrents. IIRC, you use USB-attached drive, right? It could be the case.

[AvangionQ]

Yes, that's right. SSD boot drive, three internal HDs, seven external HDs. Can't cram any more internal HDs. The case isn't big enough for them. All external HDs are connected via PCI USB3 card, see images above. I've even bought threaded USB cables to see if that would improve things. All drivers, apps & patches are up to date, as per IObit Driver Booster. I'm out of ideas for fixes. What else should I do about this problem?

[xavier2k6]

OSMemoryPriority=VeryLow you might want to change this too.

Are pagefile for drives managed by OS & what version/build of Windows 10?

[xavier2k6]

Can someone please atleast edit the issue title if there's no evidence of memory leak?

Changed it, can be changed again at later stage for something more fitting/if necessary.

[AvangionQ]

My apologies, I didn't know the difference between PCI vs PCIe. These are what I have: MZHOU 7 Port PCI Express Expansion Card, USB 3.0 7 Port Front Expansion Card, Connect 7 Devices Expanded https://www.amazon.com/gp/product/B09WYHRPTT/ ELUTENG PCIE USB 3.0 Card 7 Ports PCI Expree to USB Expansion Card Super Speed 5Gbps PCI-e USB3 Hub https://www.amazon.com/gp/product/B0BX5N22Z2/ If you think it's the PCI/USB card, let me know which one I should get. Changing these are in my budget.

The problem remains whether I put all my HDs on one USB/PCI card or mix them up 4 and 3. Yes, the page file is set by the system, though you reminded me that my smallest 2TB drive needs to be 8TB. Overheating is not possible, I use an adjacent 20-inch boxfan for ventilation. Everything's cool. I run IObit Smart Defrag once a month to optimize my hard drives. I'll go run it again right now. My hard drives are not USB 4 compatible. That's for next year's system when I finally get Windows 11. What benchmark tool would you suggest for testing the computer? I'll run it overnight tonight.

PS, thanks for the title change, it is more appropriate.

[LazyPajen]

I have had/have similar problem check this: https://www.reddit.com/r/techsupport/comments/bdz1il/seagate_hdd_st8000dm004_very_slow_write_speeds/?onetap_auto=true

[AvangionQ]

Pardon, I don't understand this program, but it says 99% good. What should I do with it? Weird, the drive letters are all out of sequence for some reason. Not alphabetical. No idea why.

I'll try CrystalDiskMark as a system test tomorrow night. Will reply back in roughly 44 hours.

EDIT: I checked all the drive letters by scrolling through each one. They all read as good.

[AvangionQ]

CrystalDiskMark crashed one of my external hard drives on test start. Upgrading 5TB to 8TB. Sucks that it wasn't the 2TB I had already planned on replacing (the reason for the day's delay). Now I gotta buy another one. $150 down, worth. Oh well, will retest after data transfers complete. I'll let you know how it goes in two days. This 4TB data transfer looks to take a while.

[ghost]

Yep looks like ISP thing. Restarting the client assigns new port?

[AvangionQ]

Yep looks like ISP thing. Restarting the client assigns new port?

Yes, have random port set to qBittorrent. I had considered it might be ISP throttling. I have Verizon 500mbps up/down. Is there a way to test if it's the ISP messing with it?

I have heard that VPN can bypass this. Any recommendations for fast inexpensive VPN?

I'm still waiting on the data transfer, bad HD to new HD. Will retest CrystalDiskMark afterward.

[AvangionQ]

Thank you for helping me find a damaged drive before I lost the data on that drive. I'm going to do some more networking testing to see if qBittorrent starts working normally. I'll get back to you in a few hours.

CrystalDiskMark results Drive, MB/s G: 39 H: 42 I: 113 J: 196 K: 146 L: 180 M: 38

[AvangionQ]

Yes, it looks like the speed problem is ISP throttling. I'm going to have to figure out a workaround.

[dawsonkm]

Yes, it looks like the speed problem is ISP throttling. I'm going to have to figure out a workaround.

I dunno. I have had fios for like 15 years. I've never had them throttle any downloads. 25+TB per month and nothing. Like was said above SMR hard disk are really problematic. There's a lot of info on them causing problems. I've have (Seagate) external drives that were SMR. You'd have to google you models to find out. I regularly have between 6-13K torrents running on one instance of Qbt and never had memory usage go so high. I don't get great stability when above 10K+. My suggestion would be to try the following if you can. There are two different libtorrent versions being used, 1.X and 2.X branch. I tried a 2.X version maybe 6 months ago and I wasn't getting the same performance as the 1.X. I've only been using QBT versions with the 1.X libtorrent branch.

[serpen7]

Yes, it looks like the speed problem is ISP throttling. I'm going to have to figure out a workaround.

try to turn on encryption it will demand however more from your CPU to manage torrenting but may help without some vpn if your ISP is just "easy throttling" you cause of torrenting. Also disable Local peer discovery cause it might be a one of things of how your ISP detecting you torrenting. See if it helps. If not then you would not have trouble reverting all back.

[ghost]

Some specific torrent files or number of torrent files can bug out torrent clients, had to start fresh sometimes and "reseed" some things. When I got to the problematic torrent file I had to reseed it with a separate client not to bug the main one.

[ghost]

@AvangionQ did you solve it ?

I also achieved higher stability by hardening security fundamentally.

[ghost]

@AvangionQ could it be before restarting the connections are TCP and after restarting the connections are uTP ?