search

qbittorrent / qBittorrent

qBittorrent BitTorrent client
https://www.qbittorrent.org
Other
27.75k stars 3.92k forks source link

High RAM usage (qBT v4.2.5) #12875

Closed wind77 closed 4 years ago

wind77 commented 4 years ago

qBittorrent version and Operating System

qBT v4.2.5 (64bit) Manjaro Linux 64 bit (kernel 5.6.11.1)

If on linux, libtorrent-rasterbar and Qt version

Qt 5.14.2 Libtorrent 1.2.6 Boost 1.72

What is the problem

10 torrents - 2 active, 2 seeding, 6 stalled, occupied >6GB RAM (total 16GB ram). Max number of upload slots per torrent is set at 2. Excessive RAM usage. The usage can go up till 15GB with 15 torrents.

What is the expected behavior

Low RAM usage. I have ever experienced a qBT version that shoot beyond 2GB - this is 1st time.

Steps to reproduce

Start qBT, load a few torrents, and RAM would just shoot up.

Extra info(if any)

Pls see attached. qBT memory

qBT torrent status

Libtorrent setting

FranciscoPombal commented 4 years ago

Any chance you can get a massif profile with qBittorrent+libtorrent debug symbols? Your issue could be related to this one, which affects some users when using HTTPS trackers: https://github.com/qbittorrent/qBittorrent/issues/12326

wind77 commented 4 years ago

Sorry, I'm relatively new to Linux. I can see a Massif-Visualizer in the repository, but I guess it doesn't collect the data you need? So which app or what command I need to use to generate the massif profile?

FranciscoPombal commented 4 years ago

@wind77 massif-visualizer is an optional program used to display the data in a nice GUI. The actual command you need to generate a massif profile is: valgrind --tool=massif --verbose --log-file=massif.log qbittorrent (of course first you need to install valgrind, which is available in the repositories).

Once you have the profile, you can just post it here.

wind77 commented 4 years ago

@FranciscoPombal Github doesn't seem to support the direct pasting of the massif file here. And Linux's Ark is unable to recognise massif file, and thus cannot compress it to zip. So I'm attaching the screenshots only.

Massif 4525 Massif 14608

These are 2 separate results from newly rebooted qBT, approx 10 mins after boot. The RAM usage of these 2 incidents, were about 1.4GB - 15 torrents (7 stalled, 4 completed, 3 seeding, 1 downloading), total dl speed <10kBps, uploading ~900kBps. If there is other data you need, do let me know!

FranciscoPombal commented 4 years ago

@wind77

@FranciscoPombal Github doesn't seem to support the direct pasting of the massif file here. And Linux's Ark is unable to recognise massif file, and thus cannot compress it to zip. So I'm attaching the screenshots only.

You can either:

These are 2 separate results from newly rebooted qBT, approx 10 mins after boot. The RAM usage of these 2 incidents, were about 1.4GB - 15 torrents (7 stalled, 4 completed, 3 seeding, 1 downloading), total dl speed <10kBps, uploading ~900kBps. If there is other data you need, do let me know!

Please do upload the 2 files corresponding to these runs, but in addition, it would also be helpful if you could upload another one from a run where you see really high RAM usage (like 4-5 GiB or greater).

wind77 commented 4 years ago

@FranciscoPombal Thanks for the guide! Here you go: the original 2 massif files. massif.out.4525.txt massif.out.14608.txt

I've deleted most torrents to reduce the memory issue. Once I got new torrents, I shall update on the RAM usage again.

FranciscoPombal commented 4 years ago

@wind77 Ok, this is very strange. These captures only show a few KiBs of memory, mostly associated with X11. They should look more like https://github.com/qbittorrent/qBittorrent/issues/12326#issuecomment-606296172.

Are you sure you are capturing the correct process? Are you using Wayland by any chance?

wind77 commented 4 years ago

@FranciscoPombal YUp, I run your command as it is: valgrind --tool=massif --verbose --log-file=massif.log qbittorrent

Regarding my OS, I'm using this:

Operating System: Manjaro Linux KDE Plasma Version: 5.18.5 KDE Frameworks Version: 5.70.0 Qt Version: 5.14.2 Kernel Version: 5.6.12-1-MANJARO OS Type: 64-bit Memory: 15.6 GiB of RAM

Since it is KDE, I presume it runs Wayland by default? But as I'm new to Linux, I'm not too sure. Anyway I can confirm this for u?

FranciscoPombal commented 4 years ago

@wind77 You can use loginctl to find out your SESSION, and then do loginctl show-session YOUR_SESSION_HERE -p Type

wind77 commented 4 years ago

@FranciscoPombal Thanks. Type is x11.

FranciscoPombal commented 4 years ago

@wind77 Well, thinking about it some more, I guess if you are running on some kind of compatibility layer, the session type could show up as X11 anyway. My thinking was that the weird stacktraces were due to something like that. I really don't know enough about wayland to be more helpful though. Maybe you could ask around some other Linux-related forum about why your massif traces look like that, and whether or not it could be related to wayland?

wind77 commented 4 years ago

@FranciscoPombal Well, if an app has an issue, who else could better advise on how to resolve it, but the app's own developer? FYI, after downloaded from repository, I merely changed the qBT settings - no change in advance setting like "Wayland" or "X11", which I'm totally unfamiliar with.

Anyway, here comes the new massif profile: Total torrents: 55

If there is other info which may help, do let me know.

qBTmemoryusage0524

qBTtorrentstatus

massif.out.61687.txt

FranciscoPombal commented 4 years ago

@wind77

Well, if an app has an issue, who else could better advise on how to resolve it, but the app's own developer?

qBittorrent is entirely developed by volunteers on their spare time, this is not a paid customer support channel. In my case, I've only "officially" joined the project a short time ago. I know my way around the codebase and the technologies involved, but I'm by no means the biggest expert.

Additionally, your setup is very different than mine, and you're having a separate problem related to massif profiles. I'm trying to figure out how to solve that so that we can get a useful profile and progress. However, I'm also not a massif expert, nor a Wayland expert (in fact I don't even use it) - and neither are you, since you don't even know whether or not you're using it (no offense meant here). That's why I'm recommending reaching out to other places that can better help you solving these ancillary problems first, otherwise we'll just be running around in circles.

My suggestion would be to go to some forum, IRC, Discord, or use google fu, read documentation, etc and ask around how to solve your massif profile issue, whether or not it might be related to Wayland vs X11, etc. Then once you get a usable massif profile (like those in the other thread I linked), we can go from there.

wind77 commented 4 years ago

@FranciscoPombal I do understand the support provided here is coming from expert volunteers, and every volunteer has his specialty areas. I dun mean u must solve this issue for me, also I appreciate the guides you have provided thus far. However, to solve an app issue, I think the qBT forum has the best chance - the volunteers/experts here have best knowledge on the app/codelines, as compared to Manjaro/KDE. Thus, if the issue i facing now is beyond ur specialty, i think my best chance is to wait for another expert who has time to take a look at the technical details. And no offense - I dun think we can expect an issue to behave exactly as we want, for us to solve it.

wind77 commented 4 years ago

More info:

This massif profile was captured after , without OS reboot or qBT reboot (ie. same old session), but the completed torrents have been removed.

Total torrents: 10 Memory usage: ~ 6GB

mem

qBTtorrentstatus0524 massif.out.76513.txt

FranciscoPombal commented 4 years ago

@wind77 look, the effort is appreciated, but these massif profiles are useless as is. You need to fix that first or find some other tools to make an actually useful memory profile. From a cursory look at massif's documentation, it seems you could try passing one or both of these flags: --pages-as-heap=yes, --keep-debuginfo=yes. Even if that works, I'm still not sure why it is not working now though.

wind77 commented 4 years ago

@FranciscoPombal Adding the flags that you requested. Can I also point out my observation: the RAM usage keeps increasing within same session. Whenever new torrent is added to the list, the RAM would increase. Even when the torrent has completed download and been removed, the RAM would not decrease. The only way to clear the RAM usage, is to exit qBT and restart it.

Total torrents: 7 Memory usage: > 7GB

DL: 0 Seeding: 0 Stalled: 7 Completed (Paused): 0

qBTmemoryusage0525

qBTstatus0525 massif.out.87392.txt

FranciscoPombal commented 4 years ago

@wind77

Is this latest capture from the 7 GiB run? If so, massif can only see 400 MiB of that (which is more reasonable than the few KiB of the previous captures), but still not very reasonable.

Once more: try to obtain a useful capture with massif first. You don't need me to tell you if it's good or not, just compare what you get to those in the other thread I linked above. In the allocation tree, you should see allocation trees from both libtorrent and Qt, and most likely some libtorrent branches that end in calls to the underlying crypto library (libcrypto, for example). Alternatively, try to find another program that yields useful profiles for you. Like I recommended before, you should seek help from other places and/or do your own research to accomplish this. Otherwise we'll get nowhere. Most likely, your memory leak is the same as https://github.com/qbittorrent/qBittorrent/issues/12326, but it would be best to confirm.

wind77 commented 4 years ago

@FranciscoPombal Useful capture? One that shows 7GB ram on massif? I have captured various profiles, some not uploaded, but none of them showed above 1GB usage. Is there any other program that you would recommend to use?

For #12326, is there any solution/workaround proposed which can tackle the issue? If so, maybe I could try that to confirm? (provided the solution is simple enough for layman like me to implement?)

I have restarted qBT session, as the RAM has exceeded 8GB and overall sys RAM usage was over 10GB ram. Right now the qBT is lingering around 128MB RAM, with the same 7 torrents as above (the same ones in 7GB RAM profile).

FranciscoPombal commented 4 years ago

@wind77

Useful capture? One that shows 7GB ram on massif?

Not only that, but that also shows useful allocation trees/traces, like I said above.

Is there any other program that you would recommend to use?

I'm only familiar with massif. You could search for alternatives or try to fix whatever problem is preventing you from obtaining useful captures with massif.

For #12326, is there any solution/workaround proposed which can tackle the issue? If so, maybe I could try that to confirm? (provided the solution is simple enough for layman like me to implement?)

Not right now. We don't yet know:

We just know that it has to do with the usage of HTTPS trackers.

I have restarted qBT session, as the RAM has exceeded 8GB and overall sys RAM usage was over 10GB ram. Right now the qBT is lingering around 128MB RAM, with the same 7 torrents as above (the same ones in 7GB RAM profile).

Your memory leak could either be related to the same thing or totally different. Right now the important thing is to determine whether or not it is due to the same HTTPS trackers problem.

wind77 commented 4 years ago

@FranciscoPombal qBT v4.2.1, doesn't have this issue. Is there any change in the codes between these 2 versions that are related to memory allocation?

I must admit that I'm a total stranger in programming (just an average user), so my suggestion might be laughable to u.

FranciscoPombal commented 4 years ago

I must admit that I'm a total stranger in programming (just an average user), so my suggestion might be laughable to u.

Not at all, it's good thinking actually.

@FranciscoPombal qBT v4.2.1, doesn't have this issue. Is there any change in the codes between these 2 versions that are related to memory allocation?

The problem here is that you're comparing two official Windows releases. Usually, more than one thing is different between 2 official qBIttorrent Windows releases besides the qBIttorrent code itself. Most notably, the version of libtorrent that is used.

In fact, if you look at the massif captures in the other threads, the HTTPS tracker-related leak does indeed seem to come from libtorrent, not qBIttorrent.

ghost commented 4 years ago

@wind77 can you try setting your disk cache to 128 MiB and reproduce the issue? The disk cache value was changed to -1 (auto) since 4.2.2.

wind77 commented 4 years ago

@an0n666 I did what u advised: After changing the setting, without rebooting, the issue remained throughout the 1hr normal usage. After changing the setting, with reboot, RAM is maintained at <700MB, throughout the 20 hrs of normal usage.

So is this a bug? Or the high RAM usage is intended with the auto disk cache setting?

ghost commented 4 years ago

I’ve always seen libtorrent based clients use up up to 8x of the set disk cache limits. There’s definitely some memory leaks going on. Also once the disk cache populates, it’s actually never evicted. So qbit continues to use up that RAM until you restart it. This will probably get fixed if libtorrent 2.0 gets released and if qbt adds support for it. Till then you just have to stick with lower cache settings.

wind77 commented 4 years ago

@an0n666 So if i set disk cache at "x", I would get "8x" usage in practice; and if it is "auto", it becomes "unlimited" RAM usage?

ghost commented 4 years ago

No. On a 16 GiB system auto would set disk cache to around ~800 MiB. Now 800 MiB x 8 = 6400 MiB. I think that perfectly fits the RAM usage that you’ve put in the issue ticket. I always divide it by 8 to get desired usage limit. I hope that helps.

wind77 commented 4 years ago

@an0n666 I see. Thanks for your clarification and help! I shall monitor how to set my cache properly then!

FranciscoPombal commented 4 years ago

@an0n666

I’ve always seen libtorrent based clients use up up to 8x of the set disk cache limits.

I've never seen this. Are you sure such a leak is due to the disk cache, and not something else that happens to grow up to 8x the disk cache? Can you reproduce this and open a new issue with a massif capture backing up this claim (either here or perhaps on the libtorrent issue tracker)?

Also once the disk cache populates, it’s actually never evicted.

This I've seen happen, though I'm not sure about the "never". But to be honest, I've never really bothered to look into it more thoroughly, but again, if you can reproduce this, you should probably open an issue in the libtorrent issue tracker about it, if you haven't done so already.

guillaumedsde commented 4 years ago

Hi, so I've had a (limited) look at this issue, I am also getting excessive memory usage.

I do not have a massif chart to contribute but here's a couple of RAM usage charts captured from my Alpine Linux, qBittorrent-nox + OpenVPN container taken on different days after multiple restarts:

Screenshot from 2020-06-10 03-01-42

Screenshot from 2020-06-10 03-02-02

Screenshot from 2020-06-11 22-54-05

Screenshot from 2020-06-11 22-57-34

A couple of things I have tried to no avail:

Here are a couple more notes :

edit: more info:

FranciscoPombal commented 4 years ago

@guillaumedsde depending on the number of torrents, this usage over these periods of 4 hours might not be unexpected. Although it does seem to be growing non-stop, it could very well be the case that it would stabilize a few hours or even minutes after those screenshots were taken. Have you actually observe it growing indefinitely, to the point that it exhausted your container's/system's memory completely?

Anyway, massif reports or equivalent profiling data is essential to track down the root cause of the bug.

guillaumedsde commented 4 years ago

@FranciscoPombal thanks for your reply :)

So I have 353 torrents total with 345 seeding. The number of active torrents is around 15 (+-5). From (vague) memory, it even filled up my 8GB of SWAP.

I have tried using the musl static binary from https://github.com/userdocs/qbittorrent-nox-static for the last couple of hours and it seems like the issue is gone.

I am building a "test" container, will try and get a massif report after running qbittorrent with a "proper load" (my 353 torrents)

FranciscoPombal commented 4 years ago

@guillaumedsde I see.

I have tried using the musl static binary from https://github.com/userdocs/qbittorrent-nox-static for the last couple of hours and it seems like the issue is gone.

Is that the only binary that doesn't leak? If so, it would also be quite interesting if you could also generate a massif report of that, to compare with the leaky one.

guillaumedsde commented 4 years ago

@FranciscoPombal I don't think so I'm unsure but I think my Ubuntu based container did not have this issue, I will try and test both the static binary and the binary from the alpine repository

guillaumedsde commented 4 years ago

Alright, so I have captured a massif report for a couple of hours on the latest alpine repo's release of qbittorrent-nox 4.2.5-r0 with a disk cache of 128MiB. During that time, I have not downloaded any torrents, I seeded roughly 20(+-5) torrents at a capped speed of 600KiBs (about a third of my upload bandwidth). I've attached the massif report below, a quick look shows the cache allocation is clearly visible at a roughly fixed 127MiB but on the other hand, it looks like some of the libtorrent SSL code could be leaking memory... If I read the dump correctly?

massif.out.276.txt

I will quickly rebuild a debug container for running valgrind on the static binary I linked above, I will advise back

guillaumedsde commented 4 years ago

hum, I tried twice to capture a massif log on the static binary but it yielded nothing twice, but let me know if you still need it once you've had a look at the previous one @FranciscoPombal :

desc: (none)         
cmd: qbittorrent-nox 
time_unit: i         
#-----------         
snapshot=0           
#-----------         
time=0               
mem_heap_B=0         
mem_heap_extra_B=0   
mem_stacks_B=0       
heap_tree=empty
FranciscoPombal commented 4 years ago

@guillaumedsde

it looks like some of the libtorrent SSL code could be leaking memory... If I read the dump correctly?

Yep. From your capture, your issue seems to be the same as https://github.com/qbittorrent/qBittorrent/issues/12326, so I would suggest posting your findings/continuing discussion there. If you can help with debugging this libtorrent issue, that would be most welcome. There are some comments of mine in that thread with cursory attempts at narrowing the problem down, feel free to take a look (the problem seems related to HTTP/HTTPS tracker connection management).

@wind77 This is another example of a good capture, providing the required information: https://github.com/qbittorrent/qBittorrent/issues/12875#issuecomment-643519239.

FranciscoPombal commented 4 years ago

Alright, I'm going to close this assuming it is the same as https://github.com/qbittorrent/qBittorrent/issues/12326. If not, please open a new issue with a massif capture or equivalent.