rTorrent reading 4x more data than sending

[ghost]

rTorrent seems to be reading 4-6x more data than it's seeding. In this case I'm seeing 100-160MB/s read from rTorrent while it's only seeding about 20MB/s worth of data.

--- IOTOP Total DISK READ : 161.06 M/s | Total DISK WRITE : 27.28 M/s Actual DISK READ: 161.06 M/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
42685 be/4 debian-d 161.06 M/s 27.28 M/s 0.00 % 70.79 % rtorrent

--- rTorrent [Throttle off/off KB] [Rate 21780.5/27685.2 KB] [Port: 65300] [U 357/0] [D 141/0] [H 0/4096] [S 21/700/65023] [F 5339/32768]

Thoughts?

[ghost]

I've tried playing with the preload features to see if that causes it, I've tried disabling preloading and that doesn't help, nor does enabling it or making the memory cache larger. rTorrent doesn't seem to utilize the piece cache anyways even though i have it set to like 16G and have 300 torrents all seeding with active peers.

[Speeddymon]

I see you don't have throttling enabled.

Can you enable throttling for both upload and download, and see if you can replicate it with throttling enabled. Your throttle should be 80% of your total maximum speed from your ISP.

So if your upload speed is 20MB/s then set the upload throttle to around 17MB/s, and if your download speed is 100MB/s then set the download throttle to 80MB/s.

If you are able to replicate it with throttling enabled, then please set the download throttle to something very low and see if the issue still occurs (uploads drop down to 1/4 of your throttled download speed)

[ghost]

I did try this with throttling too.

It seems to happen with lots of uploading slots and lots of torrents. I think what's happening is a slow client requests a piece, rTorrent loads the entire piece into memory, and attempts to send it but in smaller blocks. In the meantime many many torrents and clients are doing this causing an internal cache thrashing. So a piece is having to be reloaded every time a peer requests the next block because the piece gets kicked out of the cache due to other pieces being loaded. So instead of reading the entire piece from disk, how about just the block the peer requests? I know this is less efficient for most setups but an option to change the behavior on how data gets loaded from disk would be nice.

[Speeddymon]

Hmm..

Preload options might help here.

pieces.preload.min_size.set
pieces.preload.min_rate.set
pieces.preload.type.set

min_size is set to 131072 (or 128kb)

min_rate appears to be set to 5192

Sadly the documentation is lacking and I'm not an expert at reading this code, so I can't tell if that's 5192 bytes or a factor of time or what have you. Looks like valid values for type are 0 and 1, but no idea what either of those means.

Try lowering the min_size in your rtorrent config and see if that helps any.

[ghost]

I've also tried with and without preload, doesn't make a difference.

[Speeddymon]

You confirmed it happens even with just one active torrent and all others stopped?

If you could, please test with just one active torrent, and the rest stopped, assuming you haven't already. ;-)

If it doesn't happen there, then that will probably confirm the theory about slow clients, because you're then only dealing with a small subset of the number of peers you normally are interacting with.

[ghost]

Here's the result of downloading a SINGLE torrent. With 0 upload speed.

It's reading an insane amount of data for only downloading a torrent at 2MB/s. I mean what's the point of the freaking disk process? I'm using the 0.9.6 published build.

TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 24180 be/4 rtorrent 8.37 M/s 2.03 M/s 0.00 % 7.69 % rtorrent 24181 be/4 rtorrent 3.29 M/s 0.00 B/s 0.00 % 3.15 % rtorrent [rtorrent disk]

[Throttle 200/2048 KB] [Rate 2.3/2071.4 KB] [Port: 12345]

[ghost]

Using a completely stock rtorrent and stock debian install (no .rtorrent.rc file)... downloading a SINGLE torrent with ZERO UPLOAD.

TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 24861 be/4 rtorrent 10.21 M/s 2.99 M/s 0.00 % 7.71 % rtorrent

[Throttle off/off KB] [Rate 3.5/3091.0 KB] [Port: 6944]

This is a huge performance impact for rTorrent. This renders it unusable for me and is likely impacting the entire userbase and they don't even know it.

[Speeddymon]

I can see it reading that much with hashing enabled but you have it off.

Can you try to reproduce this with deluge?

It uses libtorrent also, and will help to narrow down whether its a bug in libtorrent or rtorrent itself.

[ghost]

Deluge uses a different libtorrent actually - libtorrent-rasterbar.

I tried builds all the way back to 0.9.0 and they all are affected. I haven't tried prior versions since they won't compile under newer versions of the C/C++ compilers.

Deluge does not have this same problem. It also performs significantly less write op/s too. I get maybe an average of 30 IOops every few seconds when the write cache is being flushed. When I was using rTorrentI was seeing a constant 300-400 IOops/s with rTorrent.

TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 15868 be/4 deluge 0.00 B/s 2.49 M/s 0.00 % 0.99 % python /usr/bin/deluged

[chros73]

I can see it reading that much with hashing enabled but you have it off.

check_hash is on by default (e.g. when you don't use a config file). Although it should mean "Check hash for finished torrents.", but who knows, can you try to disable it at first with:

pieces.hash.on_completion.set = no

This is a huge performance impact for rTorrent. This renders it unusable for me and is likely impacting the entire userbase and they don't even know it.

What OS is this exactly? Is it run on a physical device or in virtual environment? I have just checked on Ubuntu 14.04 laptop. These are the figures that I'm getting with sudo iotop -ok (interactive mode) checking different times and with this config (using my rtorrent-ps fork, based on 0.9.6, not on master):

[Rate 1697.0/1465.2 KB]
19812 be/4 chros73     990.25 K/s    2.53 M/s  0.00 %  2.06 % rtorrent
19812 be/4 chros73    1668.26 K/s 1144.28 K/s  0.00 %  1.87 % rtorrent
19812 be/4 chros73       2.79 M/s 1509.73 K/s  0.00 %  3.11 % rtorrent

[Rate 1469.0/2372.0 KB]
19812 be/4 chros73    1175.79 K/s    2.93 M/s  0.00 %  2.71 % rtorrent
19812 be/4 chros73       3.14 M/s 1419.14 K/s  0.00 %  4.33 % rtorrent
19812 be/4 chros73       2.18 M/s    2.71 M/s  0.00 %  2.94 % rtorrent
19812 be/4 chros73    1304.04 K/s 1591.24 K/s  0.00 %  1.71 % rtorrent

[Rate 1944.1/4556.1 KB]
19812 be/4 chros73     697.34 K/s    5.38 M/s  0.00 %  3.83 % rtorrent
19812 be/4 chros73    1716.96 K/s    4.05 M/s  0.00 %  0.10 % rtorrent
19812 be/4 chros73    1239.27 K/s    4.83 M/s  0.00 %  3.54 % rtorrent

[Rate 2071.3/  3.8 KB] 
19812 be/4 chros73    4846.16 K/s    0.00 K/s  0.00 % 16.97 % rtorrent
19812 be/4 chros73    7732.49 K/s    0.00 K/s  0.00 % 22.20 % rtorrent
19812 be/4 chros73    3456.99 K/s    0.00 K/s  0.00 %  7.55 % rtorrent
19812 be/4 chros73    3328.77 K/s    0.00 K/s  0.00 %  9.09 % rtorrent
19812 be/4 chros73    5752.25 K/s    0.00 K/s  0.00 % 27.22 % rtorrent

As you can see, I don't have that insane fluctuation like you do. The last readings are interesting , sometimes it can be almost 4x times of data that it reads. From the config I use these values (that can be interesting to this issue):

# Set the max amount of memory space used to mapping file chunks. This refers to memory mapping, not physical memory allocation. (max_memory_usage) This may also be set using ulimit -m where 3/4 will be allocated to file chunks.
pieces.memory.max.set = 2048M
# Adjust the send and receive buffer size for socket. Disabled by default (0), this means the default is used by OS (you have to modify the system wide settings!!!) (send_buffer_size, receive_buffer_size)
#   Increasing buffer sizes may help reduce disk seeking, connection polling as more data is buffered each time the socket is written to. It will result higher memory usage (not by rtorrent process!).
network.receive_buffer.size.set =  4M
network.send_buffer.size.set    = 12M
# Preloading a piece of a file. (Default: 0) Possible values: 0 (Off) , 1 (Madvise) , 2 (Direct paging). (https://github.com/rakshasa/rtorrent/issues/418)
pieces.preload.type.set = 2
# Check hash for finished torrents. (check_hash)
pieces.hash.on_completion.set = no

I mean what's the point of the freaking disk process?

I don't have rtorrent [rtorrent disk] process at all, just rtorrent. :)

Edit: I have to mention that I'm using modded 3.16.0-pf4-chros02 #4 SMP PREEMPT linux kernel with io tweaks, I don't know whether it matters or not.

[chros73]

downloading a SINGLE torrent with ZERO UPLOAD.

I've just tried it, just in case (with the above linked config): downloading only 1 with zero upload.

[Rate   4.5/4067.1 KB]
 9094 be/4 chros73       0.00 K/s 3968.74 K/s  0.00 %  0.00 % rtorrent

This part seems to be good on my system.

[chros73]

I made one more test:

• switch off pieces preloading (during runtime): pieces.preload.type.set = 0
• all other settings are the same as was before (network buffer size is still huge)
• just uploading with around 1130 KB (no download), and using sudo iotop -aokP accumulated output to see whats the sum after 20 sec:

[Rate 1125.2/  2.6 KB] 
 it should be around: 20x1130 = 22600.00 K
 sum is (reported by iotop) =56868.00 K

It's about 2.5x .

Can you try the similar test with deluge if it has any buffering option? I'm just curious.

[chros73]

And the last test, testing buffer sizes, setting them during runtime:

network.receive_buffer.size.set =  208K
network.send_buffer.size.set    = 208K
pieces.preload.type.set = 0

• all other settings are the same as was before
• just uploading with around 163 KB (no download), and using sudo iotop -aokP accumulated output to see what's the sum after 20 sec:

[Rate 162.4/  8.8 KB]
 it should be around: 20x163 = 3260.00 K
 sum is (reported by iotop) = 13292.00 K

It's about 4x .

So, it seems that bigger buffer sizes can help a bit with this but not really.

This is a huge performance impact for rTorrent. ... and is likely impacting the entire userbase and they don't even know it.

I tend to think that you're right :) Now, all we need is somebody who can fix this. :) Thanks for mentioning this bug as well.

[mrvn]

Why does it read ANYTHING during download? Unless you download more chunks in parallel than the system can cache all downloads should stay in memory until they are no longer needed.

Only when doing the final hash check when the download is finished I would expect reads, unless the downloaded file is still in cache (likely for a single download test smaller than the ram)..

So on that single download without upload what was the chunk size and how many chunks where downloaded in parallel? How much memory was there?

[ghost]

The system had 8G of RAM which was way more than the size of the torrent. Also the torrent, I don't remember the chunk size, there were a few dozen seeds the box was connected to. rTorrent was allowed to use 1G of memory for cache. There's no reason it should be reading anyways during downloading.

I'm wondering if rTorrent is trying to be so aggressive with it's caching that it's actually cache thrashing. Deluge does not have any of the same problems.

Debian 8 system, virtualized, all measurements were taken inside the VM.

[mrvn]

How does rtorrent caching work at all? If it's just mmap()ing the file and then uses that as buffer to recv() then every page will be read in before being overwritten with the download. That would explain as much read as download at least. I hope this isn't the case.

[chros73]

@mrvn As I mentioned before:

• I can not reproduce reading during downloading 1 torrent
• I can reproduce the 3x-4x reading issue during seeding

Can you also try it out?

[mrvn]

I have not benchmarked it but I have seen a lot more reads than my upload limit would suggest since basically forever.

[chros73]

Maybe this issue is related in this way: #409

[ryandaniels]

Reproduced downloading only 1 torrent with everything else stopped, on RaspberryPi 2 / 3 (running fresh install Raspbian/Debian Jessie). Tried all above config settings with no change. Issue seems to be related to non-local disks. Easy to detect, since downloading is being limited due to 100Mbit/s ethernet limitation of Pi. This really limits the RaspberryPi !

For reference, this is my up/down limit setting: download_rate = 3400 upload_rate = 2000

When downloading to local disk on RPi, no issue. Hit limit set in .rtorrent.rc config. iotop - 17:20:16 2877 be/4 rtorrent 0.00 B/s 3.39 M/s 0.00 % 0.00 % rtorrent ifstat - 17:20:16 28893.09 724.55

When downloading to NFS share, the issue happens. (Also tested with CIFS (samba) and still same issue). Tons of excessive reads when downloading 1 file! iotop - 17:18:42 2877 be/4 rtorrent 5.01 M/s 1774.55 K/s 0.00 % 77.63 % rtorrent iotop - 17:18:42 2879 be/4 rtorrent 2.36 M/s 0.00 B/s 0.00 % 37.71 % rtorrent [rtorrent disk] ifstat - 17:18:42 88057.85 16187.82

FYI, when copying file from local disk to NFS share, file transfer is fast. So not an issue with NFS share. (Hitting max of RPi ethernet). iotop - 17:57:31 8058 be/4 rtorrent 11.31 M/s 11.49 M/s 0.00 % 78.37 % cp -rp /NFSshare/ISO/ubuntu-16.04.1-desktop-amd64.iso /home/rtorrent ifstat - 17:57:31 94198.44 4390.52 iotop - 18:04:02 8197 be/4 rtorrent 12.60 M/s 12.37 M/s 0.00 % 86.25 % cp -rp /home/rtorrent/ubuntu-16.04.1-desktop-amd64.iso /NFSshare/ISO ifstat - 18:04:02 2599.28 97872.17

Note: ifstat is in bit per sec. iotop is in bytes per sec. Commands and command headers for reference: $ iotop -botqqq TIME TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 17:18:42 2877 be/4 rtorrent 5.01 M/s 1774.55 K/s 0.00 % 77.63 % rtorrent 17:18:42 2879 be/4 rtorrent 2.36 M/s 0.00 B/s 0.00 % 37.71 % rtorrent [rtorrent disk]

$ ifstat -b -t -i eth0 1 Time eth0 HH:MM:SS Kbps in Kbps out 17:18:42 88057.85 16187.82

[chros73]

Thanks for the detailed report. Can you also test 3x-4x seeding issue with local disk? That was the one that I could confirm. Thanks

[ryandaniels]

Upload limit 200KB. 1 torrent. About 2.75x when seeding from NFS disk: About 1.75x when seeding from local disk.

NFS disk: HH:MM:SS KB/s in KB/s out 08:50:27 464.41 208.26 08:50:28 628.22 277.58 08:50:29 628.28 214.16

TIME TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 08:50:27 11070 be/4 rtorrent 403.75 K/s 0.00 B/s 0.00 % 6.65 % rtorrent 08:50:28 11070 be/4 rtorrent 550.84 K/s 0.00 B/s 0.00 % 3.27 % rtorrent 08:50:29 11070 be/4 rtorrent 550.73 K/s 0.00 B/s 0.00 % 3.58 % rtorrent

Local disk: 09:20:25 7.04 229.83 09:20:26 6.51 218.40 09:20:27 4.59 240.26

09:20:25 11070 be/4 rtorrent 352.54 K/s 0.00 B/s 0.00 % 0.63 % rtorrent 09:20:26 11070 be/4 rtorrent 352.40 K/s 0.00 B/s 0.00 % 0.60 % rtorrent 09:20:27 11070 be/4 rtorrent 352.57 K/s 0.00 B/s 0.00 % 0.00 % rtorrent

Note: both iotop and ifstat in bytes

[chros73]

Thanks, that's indeed confirms this seeding bug with a local disk as well.

[chros73]

Can this be related? https://github.com/rakshasa/libtorrent/issues/83

[chros73]

Has anybody tried out this with master branch version?

[chros73]

I made test about the seeding issue again with local disk using 0.9.6: [Rate 2100.5 / 7500.6 KB] ... [U 65/300] pidstat -p $PID -dl 20 5 (run it 5 times, gives average at every 20 secs, and after the last run)

Note:

• downloading isn't affected
• ratio depends on the used upload slots: the more slot is used the more unnecessary read is made

I. if pieces.preload.type.set=0 or pieces.preload.type.set=2

19:09:31      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
19:09:51     1000     23536  11553.82   7841.68      0.00  /opt/rtorrent/bin/rtorrent
19:10:11     1000     23536   9645.20   7442.00      0.00  /opt/rtorrent/bin/rtorrent
19:10:31     1000     23536   9087.00   6860.80      0.00  /opt/rtorrent/bin/rtorrent
19:10:51     1000     23536   9763.00   7477.20      0.00  /opt/rtorrent/bin/rtorrent
19:11:11     1000     23536   9218.80   6956.40      0.00  /opt/rtorrent/bin/rtorrent
Average:     1000     23536   9853.73   7315.67      0.00  /opt/rtorrent/bin/rtorrent

It's 4.6x more then it's needed.

II. if pieces.preload.type.set=1

19:07:31      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
19:07:51     1000     23536  14749.60   6930.60      0.00  /opt/rtorrent/bin/rtorrent
19:08:11     1000     23536  15739.80   7864.20      0.00  /opt/rtorrent/bin/rtorrent
19:08:31     1000     23536  16461.80   7450.60      0.00  /opt/rtorrent/bin/rtorrent
19:08:51     1000     23536  15243.00   7157.60      0.00  /opt/rtorrent/bin/rtorrent
19:09:11     1000     23536  15412.80   7462.20      0.00  /opt/rtorrent/bin/rtorrent
Average:     1000     23536  15521.40   7373.04      0.00  /opt/rtorrent/bin/rtorrent

It's 7.4x more then it's needed.

How can we debug this?

[chros73]

Has anybody tried out this with master branch version?

I checked, it suffers from the same issue.

... ratio depends on the used upload slots ...

It doesn't, as it turned out: I checked the same instance at a different time and I got better results: [Rate 2100.5 / 7500.6 KB] ... [U 162/300]

12:31:59      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
12:32:19     1000     23536   3342.60   8174.60      0.00  /opt/rtorrent/bin/rtorrent
12:32:39     1000     23536   3854.00   7937.60      0.00  /opt/rtorrent/bin/rtorrent
12:32:59     1000     23536   2982.80   7713.40      0.00  /opt/rtorrent/bin/rtorrent
12:33:19     1000     23536   2559.00   7169.80      0.00  /opt/rtorrent/bin/rtorrent
12:33:39     1000     23536   2736.20   8156.40      0.00  /opt/rtorrent/bin/rtorrent
Average:     1000     23536   3094.92   7830.36      0.00  /opt/rtorrent/bin/rtorrent

What interesting is the output of dstat -ctdD sdb:

----total-cpu-usage---- ----system---- --dsk/sdb--
usr sys idl wai hiq siq|     time     | read  writ
  8   4  63  17   0   8|27-08 12:34:07|2928k   21M
  2   1   5  86   0   6|27-08 12:34:08| 256k   61M
  6   4  43  45   0   3|27-08 12:34:09|1444k   62M
  3   2  46  43   0   6|27-08 12:34:10|1104k   89M
 12   5  55  24   0   3|27-08 12:34:11|7080k   18M
  7   3  78   8   0   4|27-08 12:34:12|2188k  176k
  5   2  83   5   0   5|27-08 12:34:13|2304k    0
  6   1  74  15   0   4|27-08 12:34:14|2816k    0
  6   3  81   6   0   5|27-08 12:34:15|2436k    0
  6   2  83   5   0   5|27-08 12:34:16|2448k    0

As we can see, writing incoming data is buffered but not reading.

Somebody has to take a look at this issue who is smarter than me. :) @rakshasa ? :)

[ghost]

It sounds/looks like rTorrent is reading larger chunks than needed for each piece request from each seed slot. That would suggest the problem is actually in libtorrent.

[pyroscope]

It could also be kernel read-ahead. Someone with time + interest should look into that with sysdig.

[ghost]

I'll take a look into that, I haven't peeked back into this in awhile but I think I can dig into it.

[chros73]

@jmdevince Thank You for this!

[ghost]

Last night I spun up a fresh Debian 8 VM,

Compiled libtorrent and rtorrent from the git repo (using the master branch) and after letting it run overnight with about 30 torrents I'm currently seeding at about 6MB/s and I'm seeing about 6MB/s of disk reading.

My next step is to compile the latest stable release and look for the same behavior.

Here is the .rtorrent.rc file I'm using.

scgi_port = localhost:8050
directory.default.set = ./torrents
session.path.set = ./session
schedule2 = watch_directory,5,5,load.start=./watch/*.torrent
schedule2 = untied_directory,5,5,stop_untied=
network.port_range.set=65300-65300
throttle.max_downloads.global.set = 1000
throttle.max_uploads.global.set   = 1000
throttle.min_peers.normal.set = 500 
throttle.max_peers.normal.set = 1000
throttle.min_peers.seed.set = -1
throttle.max_peers.seed.set = -1
throttle.max_downloads.set = 100
throttle.max_uploads.set = 100
trackers.numwant.set = 100
pieces.memory.max.set = 2048M
network.max_open_sockets.set = 2048
network.max_open_files.set = 10240
network.http.max_open.set = 1024
network.receive_buffer.size.set =  4M
network.send_buffer.size.set    = 12M
pieces.preload.type.set = 2
network.http.ssl_verify_host.set = 0
network.http.ssl_verify_peer.set = 0
network.http.dns_cache_timeout.set = 25
network.xmlrpc.size_limit.set = 2M
schedule2 = session_save, 1200, 43200, ((session.save))
schedule2 = prune_file_status, 3600, 86400, ((system.file_status_cache.prune))
pieces.hash.on_completion.set = no
protocol.encryption.set = allow_incoming,enable_retry,prefer_plaintext

I'm also using the recommended sysctl changes

net.core.rmem_max = 16777216
# Maximum Socket Send Buffer. 16MB per socket - which sounds like a lot, but will virtually never consume that much. Default: 212992
net.core.wmem_max = 16777216
# Increase the write-buffer-space allocatable: min 4KB, def 12MB, max 16MB. Default: 4096 16384 4194304
net.ipv4.tcp_wmem = 4096 12582912 16777216
# Increase the read-buffer-space allocatable: min 4KB, def 12MB, max 16MB. Default: 4096 16384 4194304
net.ipv4.tcp_rmem = 4096 12582912 16777216

# Tells the system whether it should start at the default window size only for new TCP connections or also for existing TCP connections that have been idle for too long. Default: 1
net.ipv4.tcp_slow_start_after_idle = 0
# Allow reuse of sockets in TIME_WAIT state for new connections only when it is safe from the network stack’s perspective. Default: 0
net.ipv4.tcp_tw_reuse = 1
# Do not last the complete time_wait cycle. Default: 0
net.ipv4.tcp_tw_recycle = 1
# Minimum time a socket will stay in TIME_WAIT state (unusable after being used once). Default: 60
net.ipv4.tcp_fin_timeout = 30

[ghost]

Oddly enough, now that I've let it run just a little bit longer, the behavior we've seen is now occurring.

I'm seeing about 1.3x data being read as what is being seeded. I'm going to continue to let it run and see if it keeps getting worse as new torrents are added.

[asavah]

I can see this behavior too. OS: ubuntu 16.04 ext4 fs over LVM2

Latest (at the moment of writing) libtorrent + rtorrent + xmlrpc-c , everything built from git. I see 2x disk read.

93 torrents, 12 active, ul limited to 5MB/s, actual upload 5MB/s

Linux 4.4.0-45-generic (nas2)   08/11/16    _x86_64_    (4 CPU)

18:08:12      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
18:08:32      116      1194  10539.40      0.00      0.00     462  /opt/builds/rtorrent/bin/rtorrent -n -o import=/etc/rtorrent.d/nas2.rc 
18:08:52      116      1194  10124.20      0.00      0.00     435  /opt/builds/rtorrent/bin/rtorrent -n -o import=/etc/rtorrent.d/nas2.rc 
18:09:12      116      1194  10362.00      0.00      0.00     424  /opt/builds/rtorrent/bin/rtorrent -n -o import=/etc/rtorrent.d/nas2.rc 
18:09:32      116      1194  10036.20      0.00      0.00     442  /opt/builds/rtorrent/bin/rtorrent -n -o import=/etc/rtorrent.d/nas2.rc 
^C
Average:      116      1194  10265.45      0.00      0.00     441  /opt/builds/rtorrent/bin/rtorrent -n -o import=/etc/rtorrent.d/nas2.rc 

config:

scgi_port = 127.0.0.1:5000
directory = /media/nas/public
session = /media/nas/.rtorrent
network.port_range.set = 51413-51413
network.port_random.set = no
trackers.use_udp.set = yes
dht = auto
dht.port.set = 51414
protocol.pex.set = yes
system.file.allocate.set = yes
encoding_list = UTF-8
download_rate = 5120
upload_rate = 5120

system.umask.set = 0002
system.daemon.set = true

execute = {sh,-c,/usr/bin/php /media/nas/www/local/ruTorrent/php/initplugins.php &}

Note: /media/nas is local lvm mountpoint.

[devlo]

I am seeing 30-90 MB/s reads non-stop with 500 KB/s upload. Anyone working to fix this?

[devlo]

@rakshasa

Rtorrent is mmaping ~60 MB/s of files while client is sending 320 KB/s which is highly inefficient. In trace I can see mmaps for ex. sizes 4 MBs, 9 MBs or 12 MBs while those files have 1 KB/s upload speed on them. Maybe check the speed before mmap to determine what kind of size to set in mmap? Or make an option for that? Is this only problem with your custom version of libtorrent? Any plans to fix that?

[ghost]

Are these torrents with very large piece sizes?

On Dec 11, 2016, at 10:44 PM, devlo notifications@github.com wrote:

@rakshasa https://github.com/rakshasa Rtorrent is mmaping ~60 MB/s of files while client is sending 320 KB/s which is highly inefficient. In trace I can see mmaps for ex. sizes 4 MBs, 9 MBs or 12 MBs while those files have 1 KB/s upload speed on them. Maybe check the speed before mmap to determine what kind of size to set in mmap? Or make an option for that? Is this only problem with your custom version of libtorrent? Any plans to fix that?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rakshasa/rtorrent/issues/443#issuecomment-266342228, or mute the thread https://github.com/notifications/unsubscribe-auth/AAkw67psUB7V1Wqn0aInhxudQJ3IXMdjks5rHNEggaJpZM4I10xO.

[devlo]

Then for example rtorrent is sending couple of KBs of this 9.5 MB memory that he just mapped and couple of seconds later he is unmapping it...

[devlo]

@jmdevince

16 MB

[chros73]

Any news about this?

[devlo]

@chros73

As you can see I've asked @rakshasa 3 weeks ago about it, no answer to this day.

[JosiahWhite]

In regards to the bug of rtorrent reading from the disk at the same speed you are downloading, I was able to minimize this somewhat by forcing the MAP_POPULATE flag on the mmap call in libtorrent.

Before the change, when downloading many torrents (40-50) at a total rate of 80-100MB/s, rtorrent would be reading from the disk at almost exactly the same speed.

After the change, I can download at the same rate but disk read is at 5-10MB/s instead of the 90-100MB/s that I saw before.

I forced the flag by modifying this line and adding the MAP_POPULATE flag. From what I can see, rtorrent is trying to check every chunk as it is downloaded to confirm that it is not corrupt. I could be wrong but this is what ended up helping me somewhat. I assume the flag is causing the kernel to put the file in the filesystem cache so future reads and maps will be from memory but I have not tested this thoroughly

[chros73]

... reading from the disk at the same speed you are downloading ... rtorrent is trying to check every chunk as it is downloaded to confirm that it is not corrupt

As I mentioned it here I don't have this problem if there's no upload.

Does it also help with the seeding issue?

[ghost]

Has @rakshasa abandoned the issues section? I see him committing to this repo and others, but no chiming in on any issues.

[chros73]

@ls-dev749 : I have tried out this patch for libtorrent (based on your idea) on Ubuntu 14.04 , but dstat reported way worse results for seeding about 2MB/s: it reported 40-70 MB/s reading usage constantly (using git master branch version from 2016.10.10).

--- src/data/memory_chunk.cc    2014-05-13 13:40:01.843623816 +0100
+++ src/data/memory_chunk.cc    2017-01-05 18:15:21.843623816 +0100
@@ -65,6 +65,7 @@ const int MemoryChunk::prot_read;
 const int MemoryChunk::prot_write;
 const int MemoryChunk::prot_none;
 const int MemoryChunk::map_shared;
+const int MemoryChunk::map_populate;

 const int MemoryChunk::advice_normal;
 const int MemoryChunk::advice_random;
--- src/data/memory_chunk.h     2011-10-09 07:30:12.843623816 +0100
+++ src/data/memory_chunk.h     2017-01-05 18:16:40.843623816 +0100
@@ -54,6 +54,7 @@ class MemoryChunk {
   static const int prot_write             = PROT_WRITE;
   static const int prot_none              = PROT_NONE;
   static const int map_shared             = MAP_SHARED;
+  static const int map_populate           = MAP_POPULATE;

 #ifdef USE_MADVISE
   static const int advice_normal          = MADV_NORMAL;
--- src/torrent/data/file_list.cc       2015-08-08 16:01:32.000000000 +0100
+++ src/torrent/data/file_list.cc       2017-01-05 18:14:20.843623816 +0100
@@ -596,7 +596,7 @@ FileList::create_chunk_part(FileList::iterator itr, uint64_t offset, uint32_t le
   if (!(*itr)->prepare(prot))
     return MemoryChunk();

-  return SocketFile((*itr)->file_descriptor()).create_chunk(offset, length, prot, MemoryChunk::map_shared);
+  return SocketFile((*itr)->file_descriptor()).create_chunk(offset, length, prot, MemoryChunk::map_shared | MemoryChunk::map_populate);
 }

 Chunk*

[JosiahWhite]

It seems like the preloading helps during heavy downloading with a large number of torrents but not for seeding. Will look into the seeding issue and report back.

[chros73]

@ls-dev749 What OS, filesystem (local, remote, type), amount of RAM do you use?

[mrvn]

@devlo Mmaping a file (without MAP_POPULATE) does not cause disk IO. Only when you access the mapped memory does the data get read from disk. So if you map 1GB, read 4k of that and unmap then the kernel will only read 4k (+ read ahead) and not the full 1GB. So this should not be an issue.

@ls-dev749 MAP_POPULATE does force reading in all the data making this problem much worse. What you might see with heavy downloading is that you get a large spike of reads when the file gets mapped and then no reads while downloading. But you still get all the reds.

BUT does rtorrent WRITE to mmaped memory? That is a verry bad idea. Every time you write to a new page the page gets first load into memory and then overwritten. So you do get as much reads as writes. To make this realy bad and FS errors turn into segfaults. Rtorrent should really no write to mmaped files but write to the FD directly and use the mmaped memory only for reads.

[devlo]

@mrvn

As I wrote before, I did strace and the only IO syscalls were mmaps which means that rtorrent access a lot more than it should because I am getting reads from disk in magnitude of ~150-300x more (300 KB/s upload speed and 90 MB/s reads from disk).