search

qbittorrent / qBittorrent

qBittorrent BitTorrent client
https://www.qbittorrent.org
Other
27.91k stars 3.94k forks source link

sync/maindata speedup via torrent changes history and the "t10k" problem #12093

Open FranciscoPombal opened 4 years ago

FranciscoPombal commented 4 years ago

Issue created to track progress on implementing the ideas discussed on this thread: https://github.com/qbittorrent/qBittorrent/issues/10999.

Though the original issue was solved, the sync/maindata endpoint can still be significantly sped up, by implementing a "torrent history" mechanism.

The idea is that each time a sync request is made, the API backend monitors the changes in the state (essentially libtorrent alerts) until the next request. When the next request does come, only the changes need to be serialized, and can be sent right away. This contrasts with the current method, whereby the whole state is serialized each request and the differences to be sent are computed by comparing two full serialization results (the previous one and the current one) each request as well.

The speed up resulting from implementing this change could finally enable qBittorrent to become a "t10k" client (meaning a client capable of reasonably handling 10s of thousands of torrents). Today, even on powerful machines, qBittorrent is not that usable with such high numbers of torrents loaded (talking about headless/WebAPI usage here).

hbh7 commented 4 years ago

This would be great. I've recently tried having around ~1500 torrents and it's slowing to a crawl and almost unusable. The api request to sync/maindata frequently times out with a net::ERR_CONTENT_LENGTH_MISMATCH 200 (OK) error, sometimes showing the red unreachable message at the bottom left.

namazso commented 3 years ago

I read through #10999 and I don't think those suggestions would fix my (#15067) problem. Full updates are still needed sometimes and they simply don't finish under 30 seconds. To me it sounds like full update performance needs to be fixed first, I don't think serializing some 100 kilobytes of json should take half a minute on any modern processor

beyondmeat commented 1 year ago

Any ideas / eta when this can be addressed. I hit 10K seeds and qbit webui is taking a huge long dump to do anything.

glassez commented 1 year ago

Any ideas / eta when this can be addressed.

18394

ftc2 commented 1 year ago

awesome, @glassez !!

i have AMD FX-8350 and a bit over 3k torrents, and when i add a few new torrents (and downloading starts happening), the webui is totally unusable until those downloads complete.

do you think it might come in 4.5.3? or is it still pretty early in testing? if you need another tester, i could try.

thalieht commented 1 year ago

do you think it might come in 4.5.3?

4.6.0. You can already test if you want https://github.com/qbittorrent/qBittorrent/actions/workflows/ci_windows.yaml?query=branch%3Amaster (first is latest commit).

ftc2 commented 1 year ago

nice

ok, i will give it a whirl. i'm trying this build script just because i think it's sexy:

docker run -it -w /root -v $HOME/qbt:/root alpine:edge /bin/ash -c 'apk update && apk add bash curl && curl -sL git.io/qbstatic | bash -s all --qbittorrent-master --optimize --icu'

:)

i think it's using libtorrent v2.0.9 because i forgot to configure that. hopefully i don't need master.

ftc2 commented 1 year ago

ok, here is my unscientific test. took me about an hour.

method

AMD FX-8350, >3k torrents seeding (different machine running browser accessing WebUI)

idle test

load test

i repeated both tests a couple of times for each setup and got pretty consistent results.

results

qBittorrent v4.3.9
Qt:     5.15.9
Libtorrent:     1.2.19.0
Boost:  1.82.0
OpenSSL:    3.1.0
zlib:   1.2.13.zlib-ng

idle (uploading @ 3MiB/s):

load @ t=60s:

CPU-starved to the point of the WebUI not even loading. couldn't get data.

qBittorrent v4.5.2
Qt:     6.5.0
Libtorrent:     1.2.19.0
Boost:  1.82.0
OpenSSL:    3.1.0
zlib:   1.2.13.zlib-ng

idle (uploading @ 3MiB/s):

load @ t=60s:

qBittorrent v4.6.0alpha1
Qt:     5.15.9
Libtorrent:     2.0.9.0 <--- note this
Boost:  1.82.0
OpenSSL:    3.1.0
zlib:   1.2.13.zlib-ng

idle (uploading @ 3MiB/s):

load @ t=60s:

amazing web api performance. piss-poor bt performance.

qBittorrent v4.6.0alpha1
Qt:     5.15.9
Libtorrent:     1.2.19.0
Boost:  1.82.0
OpenSSL:    3.1.0
zlib:   1.2.13.zlib-ng

idle (uploading @ 3MiB/s):

load @ t=60s:

conclusion

bt performance

libtorrent: v1.2.19.0 >> v2.0.9.0

qbittorrent: 4.5.2 > 4.6

maindata webapi performance (WebUI)

qbittorrent: 4.6 >> 4.5.2 > 4.3.9

btw, all versions tested take ~9s to initially load the webui

ftc2 commented 1 year ago

this 4.6 maindata speedup is easily the biggest improvement to qbittorrent in the ~3 years i've been using it. i love it!