Qbittorrent only using 1 CPU core at high download speed (1gbps), limiting the speed.

[GoogleCodeExporter]

What steps will reproduce the problem?
1. High speed internet connection (1gbps)
2. AMD FX 8350 CPU (probably others also)

What is the expected output? What do you see instead?

I would expect the torrent client to use more cores when downloading at high 
speed but instead it only  uses 1 core and when that core is ~100% the speed 
stops growing, if i OC my CPU more the speed will grow proportionally.

What version of the product are you using? On what operating system?

Qbittorrent V 3.1.3 (latest if newer one comes out in the mean time)

Windows 7 Ultimate x64

Please provide any additional information below.

When i download torrents at high speed Qbittorrent will eat up ~100% of CORE 1 
while almost not touching other cores as you can see in this screenshot 
http://s29.postimg.org/zbdpmg14l/01core.png 

If i OC my CPU more the speed will grow proportionally.

Why is it only using 1 core and not touching the others ?

Before you ask me if i am not limited by my HDD, internet connection or other 
stuff the answer is no, as previously mentioned if i OC the CPU higher, the 
speed will keep growing.

HDD I use for torrenting benchmark http://s7.postimg.org/iriw1rcjv/atto.jpg

I've also tested on SSD, no difference whatsoever.

I've initially posted in the QBT forum but was advised to post here.

Original issue reported on code.google.com by spam...@hotmail.ro on 13 Dec 2013 at 2:07

[GoogleCodeExporter]

unfortunately libtorrent is inherently single threaded. that is, there is only 
one _network_ thread. Disk I/O and GUI is handled by other threads, but at high 
loads like this, it's possible for the network thread to become the bottleneck.

There are a few things that are relatively low hanging fruit when it comes to 
improving this situation, mostly in terms of optimizing things, not improve 
scalability. Most of those things have been done in a future branch 
(branches/libtorrent_aio). This branch not only optimizes the network thread to 
make sure to utilize the CPU more efficiently, it also introduces more disk 
threads and hash threads, and even sending and receive threads (not all that 
well tested). Anyway, you may be able to improve the situation a little bit, 
say, maybe reach 100 MB/s byt tweaking some settings to deal with high 
throughput better. If you build libtorrent with statistics enabled, it will 
produce session_stats/ logs which then can be processed with 
parse_session_stats.py to produce nice graphs of different counters in 
libtorrent. (I'd be happy to take a look if you want to email me the logs).

Original comment by arvid.no...@gmail.com on 13 Dec 2013 at 4:55

[GoogleCodeExporter]

Thank you,

Sadly i have no idea how to do all that, i am just a standard user of 
Qbittorrent, don't have any developer skills that i know off when it comes to 
this stuff.

If you give me some nooby steps to follow in order to optimize performance i'll 
go for it though, what shall i tweak ?

Original comment by spam...@hotmail.ro on 13 Dec 2013 at 5:55

[GoogleCodeExporter]

Since, I was the one directing him here, I'll try to provide special qbt builds 
with libtorrent logging enabled. Expect them some time in the weekend. (If I am 
not really busy).

Original comment by hammered...@gmail.com on 13 Dec 2013 at 2:50

[GoogleCodeExporter]

Thanks guys.

Before i proceed running this build i have 2 questions:

1) What info is sent in the logs i will be providing, what is logged ?

2) Should i avoid using the modified code QBT version Sledgehammer will provide 
me on private trackers ? 
I am thinking maybe they will not recognize the client and ban it/me or similar 
stuff thats why i ask.

Original comment by spam...@hotmail.ro on 14 Dec 2013 at 12:55

[GoogleCodeExporter]

it will only provide performance counters. things like "the number of bytes 
queued to be written to disk", "the number of bytes queued to be written to a 
socket", "the number of downloading torrents", "the number of seeding 
torrents", "the number of peers uploading to", "the number of interested 
peers", "the number of unchoked peers", etc. no torrent specific or peer 
specific information.

In fact, feel free to inspect the logs, they are plain text, and just a bunch 
of columns with numbers in them (the column name is specified on the first 
line, but there are more column than typically fit on one screen)

Original comment by arvid.no...@gmail.com on 14 Dec 2013 at 1:45

[GoogleCodeExporter]

@hammered999, maybe you could also provide a build with 
"high_performance_seed()" set as the default settings. That preset is tweaked 
for high throughput environments (not just seeding).

Original comment by arvid.no...@gmail.com on 14 Dec 2013 at 1:46

[GoogleCodeExporter]

Here is a qbt build with statistics on. A folder named "session_stats" will be 
created in the same dir as qbittorrent.exe. The logs will be put there.

Also this is build with gcc 4.8.0 with full optimizations(O3), please report if 
you get more speed. (official builds are compiled with msvc2008).

Here is the link: 
http://builds.shiki.hu/temp/qbittorrent_3.2.0alpha_26122013_mingw_statistics_set
up.exe

Original comment by hammered...@gmail.com on 26 Dec 2013 at 11:28

[GoogleCodeExporter]

Same problem, no improvements.

I have uploaded the log here http://www.wikiupload.com/9R5LPW4Q39SCJQN

Screenshot: http://s30.postimg.org/qvvro9ukf/qbt.jpg (my speed is a bit lower 
than before cause i am OC'd to 4.5ghz, before i was at 4.7ghz)

Original comment by spam...@hotmail.ro on 26 Dec 2013 at 4:24

[GoogleCodeExporter]

that log is only 25 seconds long, and it doesn't have anything downloading at 
all, just one torrent being checked. could you run it a bit longer?
also, make sure to shut down gracefully to make sure the log file is flushed 
correctly (i.e. that the process isn't killed)

Original comment by arvid.no...@gmail.com on 27 Dec 2013 at 9:15

[GoogleCodeExporter]

Oh lol, that should not be, I've downloaded 4 torrents before uploading the log.

I will give it another try soon and post a new log, I didn't kill the process 
either, right clicked on task bar icon / exit.

Original comment by spam...@hotmail.ro on 27 Dec 2013 at 2:12

[GoogleCodeExporter]

>right clicked on task bar icon / exit.

If you have a lot torrent or a lot of data pending writing to disk wait a few 
seconds after hitting exit. Although the window disappears instantly qbt might 
stay in the background doing cleanup work.

Original comment by hammered...@gmail.com on 27 Dec 2013 at 2:54

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Done, i think it should be good now, has 2,12mb vs 13kb it had before.

Here it is http://www.wikiupload.com/BACPB6RVJJFXYQ7

I was able to hit 84mb this time for whatever reason.

Maybe it had something to do with me installing this new version of QBT in a 
different folder the first time ?

Original comment by spam...@hotmail.ro on 27 Dec 2013 at 3:08

[GoogleCodeExporter]

There's nothing obvious standing out. It does look like you should be able to 
do a bit better than what you are though. The only limit I can see being hit is 
the disk write queue. You may want to try increasing 
session_settings::max_queued_disk_bytes to, say, 10 MiB (it defaults to 1 MiB).

Original comment by arvid.no...@gmail.com on 27 Dec 2013 at 5:18

[GoogleCodeExporter]

>>You may want to try increasing session_settings::max_queued_disk_bytes to, 
say, 10 MiB (it defaults to 1 MiB).

If youre directing this to me, i am not able to find that setting anywhere in 
QBT, if youre directing it to Sledgehammer, i'll test it once he does the 
modification and post back the result :).

Original comment by spam...@hotmail.ro on 27 Dec 2013 at 7:38

[GoogleCodeExporter]

yeah, it's mostly directed to sledgehammer...

Original comment by arvid.no...@gmail.com on 28 Dec 2013 at 1:08

[GoogleCodeExporter]

Here are:

1. A build using max_queued_disk_bytes = 10 * 1024 * 1024
http://builds.shiki.hu/temp/qbittorrent_3.2.0alpha_28122013_mingw_statistics_max
queueddiskbytes_setup.exe

2. A build using high_performance_seed()
http://builds.shiki.hu/temp/qbittorrent_3.2.0alpha_28122013_mingw_statistics_hig
h_perf_setup.exe

Please use them in the order specified and note if one is better than the other.
Also these builds, will still create statistics logs, so you can upload them 
here for arvid.

Original comment by hammered...@gmail.com on 28 Dec 2013 at 10:48

[GoogleCodeExporter]

PS: I wonder if the linux network stack could squeeze a little more throughput 
than Windows in this kind of connection. What do you say arvid?

Original comment by hammered...@gmail.com on 28 Dec 2013 at 10:50

[GoogleCodeExporter]

I haven't run any benchmarks on win vs. linux, but windows does have a more 
thought through API for networking, which in theory allows network drivers to 
be more efficient (fewer copies)

Original comment by arvid.no...@gmail.com on 28 Dec 2013 at 3:20

[GoogleCodeExporter]

I've had some problems getting stats to log (a .log would not be created), had 
to install QBT multiple times before it worked, restarting the process itself 
would not help either.

Sadly i did not notice any improvement with this versions, they both start to 
throttle when hitting ~73mb (core 1 is at 100% when this happens), speed goes 
higher at times hitting ~85mb but on average it starts to throttle when hitting 
~73mb, the speed drops to ~40-50mb for 3-4 sec then starts growing again 
rapidly, it seems to be stable when its around 69mb, no throttling and cpu core 
is at 96% or 
so.

Log for 1: http://s000.tinyupload.com/?file_id=03456693065998566462
Log for 2: http://s000.tinyupload.com/?file_id=04347680177553528249

(Wikiupload was down)

I've downloaded 3 20.somethingGB torrents with each version, same torrents.

When testing the high_performance_seed QBT version at the end of downloading my 
3'rd torrent (70% complete) I've noticed that for whatever reason the speed 
went down to 30mb and would not grow anymore, i've restarted the client and 
everything went back to normal but a new log was not created.

Original comment by spam...@hotmail.ro on 28 Dec 2013 at 9:22

[GoogleCodeExporter]

While we're waiting for a reply from arvid and since you have 64bit OS, it 
would be interesting see if an 64bit qbt/libtorrent makes any measurable 
difference in this case.

So try using Dayman's 64bit builds(manually uninstall mine first): 
http://qbforums.shiki.hu/index.php/topic,1645.0.html

Original comment by hammered...@gmail.com on 31 Dec 2013 at 11:27

[GoogleCodeExporter]

to me it looks like the bottleneck is the fact that libtorrent reads 16 kiB at 
a time from the socket. unfortunately this has only been fixed in the 
branches/libtorrent_aio version of libtorrent (this is where most performance 
optimizations have gone in the last two years).

I'm currently working on making trunk releasable. part of that is to make sure 
qbittorrent builds using it. hopefully once this is done, using libtorrent_aio 
(which will then become trunk) has a lower threshold.

Original comment by arvid.no...@gmail.com on 31 Dec 2013 at 5:08

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

>part of that is to make sure qbittorrent builds using it

Are you testing against qbt? If not, you could ping me to do test builds once 
in a while.

Original comment by hammered...@gmail.com on 31 Dec 2013 at 5:14

[GoogleCodeExporter]

yeah, I'm trying to build qbt using libtorrent trunk. there are a few backwards 
compatibility issues I need to fix. but I also run into some other build errors 
that may be due to my setup. I was hoping you'de be online on IRC.

Original comment by arvid.no...@gmail.com on 31 Dec 2013 at 5:27

[GoogleCodeExporter]

No improvement with 64 bit version.

Original comment by spam...@hotmail.ro on 2 Jan 2014 at 6:18

[GoogleCodeExporter]

I would expect the main advantage of the 64 bit build to be that the disk cache 
can be > 2GB. That doesn't seem to be your bottleneck though.

If you feel up for it, you could make a release build with debug symbols (at 
least with debug symbols of libtorrent, but qbt might be useful as well), and 
run it through a profiler?

It's been quite a while I've developed on windows, but iirc, AMD's codeanalyst 
is free (but may require running on an actual AMD CPU). Intel VTune is good, 
but not free. Another option is VerySleepy ( 
http://www.codersnotes.com/sleepy/sleepy ). If you post profiler results, it 
may turn out to be something simpler to fix.

Original comment by arvid.no...@gmail.com on 2 Jan 2014 at 7:41