search

arvidn / libtorrent

an efficient feature complete C++ bittorrent implementation
http://libtorrent.org
Other
5.16k stars 993 forks source link

Abnormally slow IO performance on CIFS network share #7480

Open constantins2001 opened 1 year ago

constantins2001 commented 1 year ago

Versions

libtorrent version (or branch): 2.0.9-r2 platform/architecture: ARM64

What is the problem?

IO on mounted CIFS (SMB) network share are abnormally low.

This indicates qBittorrent has drastically slower speeds when moving downloads/downloading to network shares compared to a manual move operation.

Steps to reproduce

  1. Mount CIFS share
    mount.cifs -o user=<XXX>,pass=<XXX>,cache=loose,actimeo=120,fsc //<URL>/<share> <mountpoint>
  2. Set qBittorent download/move location to <mountpoint>
  3. Download torrent
  4. Observe performance impact

Additional context

I'll be happy to provide a full testing environment, just contact me directly.

Related to https://github.com/qbittorrent/qBittorrent/issues/19407

nagyimre1980 commented 1 year ago

2.0px branch unusable, bad. don't bother with it! it will never be good

master255 commented 1 year ago

@nagyimre1980 Why do you think that?

master255 commented 1 year ago

@constantins2001 Do you have a comparison with version 1.2 of the library?

stalkerok commented 1 year ago

How about https://github.com/arvidn/libtorrent/issues/6561 https://github.com/qbittorrent/qBittorrent/issues/18983 https://github.com/qbittorrent/qBittorrent/issues/16043 and many more performance issues that I didn't comment on, so it's hard for me to track them down. Is that not enough to consider this branch bad?

master255 commented 1 year ago

@stalkerok Are these problems compared to the old version 1.2 of the library? Or are they general problems?

stalkerok commented 1 year ago

These are problems only 2.0.x

master255 commented 1 year ago

@arvidn Too many changes at one time (1.2->2). Is it possible to roll back the change of disk write type? (memory mapped files) This can increase disk write speed and improve the popularity of the library 2.0 version.

master255 commented 1 year ago

@stalkerok You need to test 1.2 torrents and 2.0 torrents on Url-seeded torrent files. This will rule out problems with other torrent mechanisms. This is pure write speed. Use this ready-made torrent file. 511.65-notebook-win10-win11-64bit-international-dch-whql.zip

constantins2001 commented 1 year ago

@constantins2001 Do you have a comparison with version 1.2 of the library?

https://github.com/qbittorrent/qBittorrent/issues/19407#issuecomment-1665324759

constantins2001 commented 1 year ago

Is it possible to roll back the change of disk write type? (memory mapped files)

I reverted back to version 2 due to the following issue with 1.2.19: https://github.com/qbittorrent/qBittorrent/issues/19407#issuecomment-1666140932

Better slowish write speeds than an unstable experience, but still I'd love to use the full potential of my platform...

stalkerok commented 1 year ago

https://user-images.githubusercontent.com/63958081/259389632-14476ece-989c-4394-b232-38e9da440111.png RC_2_0 is NOT ABLE to download/upload 100MB/s from HDD. Disk I/O type (POSIX-compliant) saves the situation a little, but these are far from the same results. Also unable to match RC_1_2 is this new disk I/O https://github.com/arvidn/libtorrent/pull/7013 So RC_1_2 remains the best.

master255 commented 1 year ago

@stalkerok Problem with the upload or download speed?

constantins2001 commented 1 year ago
  • Automatically moving to network share via qBittorrent results in about 5 MB/s speeds.

same issue on 1.2.19, only downloads directly to the share are fast

master255 commented 1 year ago

Looks like Arvin has abandoned library development. I don't know why. Maybe he has a creative crisis because of the wrong transition to version 2.+. Or maybe he's just tired of torrents.

It must be hard to develop a program and not see the results of its work.

paxter commented 1 year ago

Using 2.x with NFS share. Upload/download rates above 100 MBit/s are no problem. I think it's working fine with the right OS and/or kernel version.

constantins2001 commented 1 year ago

Using 2.x with NFS share

NFS and SMB seem to be very different in many regards

constantins2001 commented 1 year ago

Update: I moved to a non network share environment with 2x 8 TB HDDs (HGST HUH721008AL) in RAID 0 and now easily reach the limit of my network connection (1 Gbit/s). The issues many others seem to experience with HDDs don't show for me.

KyleSanderson commented 1 year ago

Update: I moved to a non network share environment with 2x 8 TB HDDs (HGST HUH721008AL) in RAID 0 and now easily reach the limit of my network connection (1 Gbit/s). The issues many others seem to experience with HDDs don't show for me.

Yeah, it's a bit silly but that's expected. Remember each time you're doing this you're incurring a network roundtrip which can be multiple ms at the very least. Writing sequentially is also a very different from writing "random" blocks, especially with that latency involved. I saw you turned your dirty to nothing which was obviously a mistake on your part.

But yeah, mmap is never going to be as performant as the classic C linux api which 1.2 used. The future remains io_uring here for linux. The backends are interchangeable, so 2.0 can recover from this (and be baller if the networking is moved as well). Just takes time.

constantins2001 commented 1 year ago

you're incurring a network roundtrip which can be multiple ms

Shouldn't matter, in combination with the write cache the 1 Gbit/s connection should easily be saturated (see OP).


Writing sequentially is also a very different from writing "random" blocks, especially with that latency involved.

Due to the same reasons as above I also don't think this is the issue. The move operation is just as slow as the download using random blocks (see OP).


I saw you turned your dirty to nothing which was obviously a mistake on your part.

Only on the old libtorrent, it crashed my system with the default settings. I think once it filled the write cache the system switched to sync operations and in combination with the network share that somehow locked up the whole system. The config I posted above was the only way for me to prevent the system from locking up. I also observed that behavior with Transmission (which prompted my switch to qBittorrent in the first place).

evulhotdog commented 1 year ago

Using 2.x with NFS share. Upload/download rates above 100 MBit/s are no problem. I think it's working fine with the right OS and/or kernel version.

@paxter Can you share your configuration with us?

arvidn commented 1 year ago

libtorrent 2.0 defaults to using memory mapped files for disk I/O. This turned out to be sub-optimal in a lot of common cases (perhaps most cases).

My top priority is to implement a new disk I/O subsystem in master of libtorrent, which is multi-threaded, cached and uses pread/pwrite to be thread-safe (and windows equivalents). Once this is ready I will cut libtorrent 2.1 which includes this.

In the meantime, you may want to try to posix disk I/O subsystem available in libtorrent 2.0. It's not multi-threaded nor cached, so it will likely not perform optimally, but for a network share, it might be better than mmap.

master255 commented 1 year ago

@arvidn It would be better to do DHT development (switch to tcp and broadcast traffic through other nodes (as in libp2p)). But if you'd rather work with the disk subsystem, then ok.

arvidn commented 1 year ago

It would be better to do DHT development

patches are welcome

master255 commented 1 year ago

It would be better to do DHT development

patches are welcome

I'm developing more important programs. Higher level. And it would be easier for me to do it if you improved the DHT.

master255 commented 1 year ago

It would be better to do DHT development

patches are welcome

Okay. I realize it's very complicated. So just help me on my tickets. I think that should be enough.

paxter commented 12 months ago

Using 2.x with NFS share. Upload/download rates above 100 MBit/s are no problem. I think it's working fine with the right OS and/or kernel version.

@paxter Can you share your configuration with us?

Which configuration? It's a default nfs share on bsd. Nothing special.

stalkerok commented 12 months ago

@arvidn, why can't the old disk I/O subsystem from 1.2 be used in 2.1, which works fine, if not by default, then at least as configurable parameters? You can't guarantee that the new disk subsystem will be better and there will be no problems with it.

KyleSanderson commented 12 months ago

@arvidn, why can't the old disk I/O subsystem from 1.2 be used in 2.1, which works fine, if not by default, then at least as configurable parameters? You can't guarantee that the new disk subsystem will be better and there will be no problems with it.

the p stands for position, which allows you to avoid the file pointer position and read and write to certain segments (as long as append isn't used). This is a general POSIX backend that was standardised in 2001 (not nearly as fast or efficient as io_uring or IoRing) - that should still be better if fully pipelined than the older one despite being 22 year-old technology. The risk here is the local cache, that's going to dump into the page cache, then finally spooled to disk.

People are having problems with the classic backend with multiple files not being committed to the page cache when cross-seeding (same data, different torrents). I'm unclear/concerned the local cache is going to still hamper these efforts, and the locking isn't going to be as tight as one would expect.

the real struggle for NVMe boxes right now (on 1.2) is the fact the network is single threaded. There's been custom TLS crypto and similar shoved in there that just kills the entire session if it's toggled on. io_uring "fixes" this as well by allowing you to do all that crypto for the next set of work between submissions (which could be threaded or not).

Anyway, it sounds like 2.1 is the next 1.2, and 2.0 will hopefully be forgotten in short order (it may even take calling it 3.0, but if there's a large problem when that's done it won't be possible to recover from this).

Smexhy commented 6 months ago

Versions

libtorrent version (or branch): 2.0.9-r2 platform/architecture: ARM64

What is the problem?

IO on mounted CIFS (SMB) network share are abnormally low.

  • Downloading to the network share directly results in ~5.4 MB/s speeds.
  • Downloading to local SSD results in speeds in excess of 26 MB/s
  • Automatically moving to network share via qBittorrent results in about 5 MB/s speeds.
  • Moving the file manually from local SSD to network share via the mv command results in speeds of ~35.7 MB/s.

This indicates qBittorrent has drastically slower speeds when moving downloads/downloading to network shares compared to a manual move operation.

Steps to reproduce

  1. Mount CIFS share mount.cifs -o user=<XXX>,pass=<XXX>,cache=loose,actimeo=120,fsc //<URL>/<share> <mountpoint>
  2. Set qBittorent download/move location to <mountpoint>
  3. Download torrent
  4. Observe performance impact

Additional context

I'll be happy to provide a full testing environment, just contact me directly.

Related to qbittorrent/qBittorrent#19407

This is driving me crazy as well. Literally tried everything, but the speeds are terrible. Wish I could just go around the NAS, but I can’t.

the-hotmann commented 6 days ago

I encounter the very same issue.

When I use qBittorrent via docker-compose and want it to access a SMB/CIFS share when everything is super super slow:

  torrent:
    image: lscr.io/linuxserver/qbittorrent:4.6.6
    container_name: torrent
    environment:
      - PUID=1000
      - PGID=33
    volumes:
      - "./torrent/config/:/config/:rw"
      - "./torrent/custom-services.d/:/custom-services.d/:ro"
      - "SMB_SHARE:/downloads/:rw"
    deploy:
      resources:
        limits:
          memory: 1G
    restart: unless-stopped

volumes:
  SMB_SHARE:
    name: "SMB_SHARE"
    driver: "local"
    driver_opts:
      type: "cifs"
      o: "seal,addr=###addr###,password=###pass###,rw,uid=1000,gid=33,file_mode=0644,dir_mode=0755"
      device: "//###device###/"

Did I do something wrong, or is it just qBittorrent not handeling SMB/CIFS Shares properly? Any recommendations are welcome!

https://github.com/qbittorrent/qBittorrent/issues/19407#issuecomment-1706120172

[...] this seems to be a quite niche use case.

I dont think it is a niche issue. Storage these days can be very cheap, or expensive. If you just buy/rent storage, it is cheap. If you buy a server with more ssd-raid storage it will be expensive, but you actually will never use the ssd-raid storage speed. So I think it would make a lot of sense to improve the handling/speed of SMB shares.

the-hotmann commented 6 days ago
  • Downloading to the network share directly results in ~5.4 MB/s speeds. [...]
  • Automatically moving to network share via qBittorrent results in about 5 MB/s speeds.

Yes, I see these 5MB/s limits aswell. When I move stuff on my own server I see about 500MB/s, but when to the SMB share it drops to 5MB/s.

Anyway, I can access the SMB share from my server with about 200-250MB/s. So the share is not the bottleneck!

HanabishiRecca commented 5 days ago

I am trying to solve this problem at qBittorrent side right now. qbittorrent/qBittorrent#21300 If someone wishes to test the patch and feed back with results, it would be great.

the-hotmann commented 4 days ago

How can one test this? I use a dockerised version of this image: lscr.io/linuxserver/qbittorrent Will there be a test/bets release with this patch?

HanabishiRecca commented 4 days ago

How can one test this?

  1. Build it yourself or grab a CI build artifact (x86 builds only, say you need qbittorrent-nox).
  2. Run this build, go to advanced settings and set Disk IO type (requires restart) to Force pread/pwrite.
  3. Don't forget to restart the client.

Will there be a test/bets release with this patch?

It's still in development and not even merged yet. So it's unknown when it will get to any kind of release.