rTorrent constantly churning / leaking memory

kthrow commented 10 months ago

Support guidelines

[X] I've read the support guidelines

I've found a bug and checked that ...

[X] ... the documentation does not mention anything about my problem
[X] ... there are no open or closed issues that are related to my problem

Description

rTorrent appears to be constantly allocating memory until reaching some limit, freeing pretty much all of it, and then reallocating constantly.

Expected behaviour

rTorrent uses a reasonable amount of memory. In the past, this has roughly been the max piece size, with some jumps to around ~10GB during large rechecking operations.

Actual behaviour

On my 64GB machine, it constantly allocates until hitting around 50GB of usage, and then resets itself.

This setup was working perfectly fine for a long time. I powered off my machine, ensuring I safely stopped rtorrent, yesterday to install some new fans. I also ran an apt update && apt upgrade. Once I turned it on, this new memory behaviour presented itself. This is the only container out of ~45 showing an issue.

You can see from this chart the behaviour before and after the restart: Both time periods had roughly the same number of active torrents (~20), and no new downloads happened during this time period at all.

I have tried both latest and edge and they have the same behaviour. I've also tried a fork I made of this image using jesec/rtorrent and jesec/libtorrent and the issue persists there as well, so it's presumably unrelated to the patches made in either version.

Zooming into the most recent five minutes, you can see it's allocated nearly 6GB.

Here's what it looks like when it frees -- generally it's accompanied with higher than usual CPU usage, where it drops from ~51GB to 2.3GB:

It seems to take 30-45 minutes to exhaust the memory "limit" and reset back to a reasonable number before climbing again.

Steps to reproduce

Have around ~2200 torrents loaded, total data of roughly ~39TB
Run the application as usual.

I don't really expect anyone to be able to reproduce this easily..

Docker info

Client: Docker Engine - Community
 Version:    25.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 54
  Running: 46
  Paused: 0
  Stopped: 8
 Images: 52
 Server Version: 25.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-14-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 58.74GiB
 Name: gus
 ID: 513370be-7fc3-44dc-afef-553f848ff236
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Docker Compose config

version: "3.2"
services:
  rutorrent:
    image: crazymax/rtorrent-rutorrent:edge
    container_name: rutorrent
    networks:
      - webaccess
    ports:
      - 6881:6881/udp
      - 58000:8000
      - 8001
      - 8080
      - 8081
      - 9000
      - 9001
      - 50000:50000
    environment:
      TZ: America/Toronto
      PUID: 1000
      PGID: 1000
      MEMORY_LIMIT: 1024M
      UPLOAD_MAX_SIZE: 64M
      XMLRPC_SIZE_LIMIT: 16M
      OPCACHE_MEM_SIZE: 512M
      MAX_FILE_UPLOADS: 50
      LOG_ACCESS: true
      XMLRPC_PORT: 8000
      RUTORRENT_PORT: 8080
      WEBDAV_PORT: 9000
      RT_INC_PORT: 50000
      RT_LOG_LEVEL: debug
      RT_LOG_EXECUTE: true
      RT_LOG_XMLRPC: false
      RT_SESSION_SAVE_SECONDS: 2400
    stop_grace_period: 30s
    volumes:
      - /home/censored/appdata/rtorrent/data:/data
      - /mnt/storage/downloads:/downloads
      - /home/censored/appdata/rtorrent/passwd:/passwd
    ulimits:
      nproc: 65535
      nofile:
        soft: 32000
        hard: 40000
    restart: unless-stopped

Logs

I can produce some, but (a) the file is ~8MB, and (b) I'd have to censor the announce URLs. I scanned through and didn't find anything useful, not sure if there's a more detailed `RT_LOG_LEVEL` than debug.

Additional info

top from inside the container:

stickz commented 10 months ago

Hello, your screenshot of top shows less than 7GB of memory is used. 32GB is in buffers and caches to reduce disk usage.

rTorrent does not physically allocate memory. It lets the Linux Kernel decide which files to put in buffers and caches. The process is currently using 1.5G approximately for physical allocation to run it's internal procedures.

I would recommend specifying pieces.memory.max.set = 4GB explicitly in your .rtorrent.rc file. This may further optimize your memory consumption. This diagnostic step will also eliminate possible avenues for a software memory leak to occur.

kthrow commented 10 months ago

Hello, your screenshot of top shows less than 7GB of memory is used. 32GB is in buffers and caches to reduce disk usage.

rTorrent does not physically allocate memory. It lets the Linux Kernel decide which files to put in buffers and caches. The process is currently using 1.5G approximately for physical allocation to run it's internal procedures.

I would recommend specifying pieces.memory.max.set = 4GB explicitly in your .rtorrent.rc file. This may further optimize your memory consumption. This diagnostic step will also eliminate possible avenues for a software memory leak to occur.

Thanks for the info!

I did already have pieces.memory.max.set = 6GB set while all this was going on. Though after a bunch of searching I did also arrive at the possibility a kernel update was the issue. I was running on 6.5.0-14 during the period of high allocations, and just downgraded back to 6.2.0-39. So far it seems like that's resolved the problem. I'm not sure how I'd determine what changed between these two versions but I guess I will just hold back updates and see if I can upgrade later.

Edit: Could be related? If I search "kernel 6.5.0 memory leak" this thread comes up: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2041668. Seems like it's bad metrics and not an actual leak..

stickz commented 10 months ago

Yes, a Linux Kernel can cause memory stability issues when running rTorrent. I'm not fimular with how the kernel layer works.

However, I understand their versioning system. You selected a version that was too "new". The last number is the number of bug fixes. For instance 6.2 has 39 bug fixes, 6.5 has 14 bug fixes.

I can see your host is Ubuntu 22.04 LTS. If the HWE (Hardware Enablement) kernel of 6.2 is not stable, it's sometimes required to go all the way back to the GA kernel of 5.19. Performance is great with newer releases, but memory stability is more important.

crazy-max / docker-rtorrent-rutorrent