Memory Leak through Web API

burritothief commented 1 year ago

qBittorrent & operating system versions

qBittorrent: 4.5.2 Operating system: Debian GNU/Linux 11 (bullseye) [Linux 5.10.0-19-amd64] Qt: 6.5.0 libtorrent-rasterbar: 2.0.8.0

What is the problem?

Problem Querying the qBittorrent Web API, especially if the client has a large number of torrents, permanently increases memory usage of the app. If you have external apps periodically querying the API, it will indefinitely grow memory usage until the app eventually crashes due to out of memory issues. The problem is really pronounced with large amounts of torrents.

Background In my personal situation I have several Docker containers running qBittorrent-nox instances. Each container has a few thousand torrents. I also have external apps like Radarr and Sonarr connected to my qBittorrent containers. My qBittorrent containers gradually grow in memory usage and typically crash due to OOM errors after 48 hours (on a 2GB RAM virtual machine). I initially thought the memory issues were related to mmap issues between libtorrent v1 and v2. However, I've tried running my qBittorrent containers all with v4.3.9, v.4.4.5 and v.4.5.2 (libtorrent v1 and v2), and still get OOM errors after a couple days.

In debugging, I found that disconnecting my Radarr/Sonarr instances from qBittorrent kept memory usage stable and I suspect that querying the web API can cause memory leaks.

Example To demonstrate the issue, I've created a simulated example where I create three new qBittorrent test instances. Then I loaded them with 10, 100, and 1000 torrents, respectively. Then I query each container's web API for a list of torrents one hundred times:

$ docker compose -f docker-compose.yml up -d
[+] Running 4/4
 ⠿ Network test-example_default  Created                                                                                                               
 ⠿ Container qbittorrent-1000      Started
 ⠿ Container qbittorrent-10        Started
 ⠿ Container qbittorrent-100       Started  
$ docker stats --no-stream       
CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
8e09d09ea340   qbittorrent-100    0.04%     12.96MiB / 1.936GiB   0.65%     3.66MB / 40.4kB   0B / 7.83MB       16
86e1ccdfa89c   qbittorrent-10     0.02%     16.06MiB / 1.936GiB   0.81%     3.66MB / 33.9kB   5.09MB / 7.83MB   16
59b407e1f51d   qbittorrent-1000   0.04%     12.84MiB / 1.936GiB   0.65%     3.67MB / 39.6kB   0B / 7.83MB       16                                                                                                              
$ python add-torrents.py 8081 10
$ python add-torrents.py 8082 100
$ python add-torrents.py 8083 1000
$ docker stats --no-stream 
CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O        PIDS
8e09d09ea340   qbittorrent-100    0.17%     24.28MiB / 1.936GiB   1.22%     3.78MB / 126kB    0B / 8.84MB      27
86e1ccdfa89c   qbittorrent-10     0.04%     25.45MiB / 1.936GiB   1.28%     3.75MB / 98.1kB   5.1MB / 7.96MB   27
59b407e1f51d   qbittorrent-1000   0.11%     41.53MiB / 1.936GiB   2.09%     4.03MB / 110kB    0B / 16.2MB      27
$ for n in {1..100}; do python fetch-torrents.py 8081; done
$ for n in {1..100}; do python fetch-torrents.py 8082; done
$ for n in {1..100}; do python fetch-torrents.py 8083; done
$ docker stats --no-stream 
CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O        PIDS
8e09d09ea340   qbittorrent-100    0.03%     70.79MiB / 1.936GiB   3.57%     3.96MB / 927kB    0B / 9.1MB       16
86e1ccdfa89c   qbittorrent-10     0.02%     30.14MiB / 1.936GiB   1.52%     3.93MB / 499kB    5.1MB / 7.97MB   16
59b407e1f51d   qbittorrent-1000   0.23%     508.9MiB / 1.936GiB   25.67%    4.22MB / 4.94MB   0B / 17MB        19

Note that a fresh container starts at 15MiB, but after querying the APIs, qbittorrent-1000 has 500MiB memory usage compared to qbittorrent-10 30MiB memory usage. I've included more details on how to reproduce similar results below. I've also tried to manually clear RAM cache by running echo 3 > /proc/sys/vm/drop_caches which does nothing. Continuing to run fetch-torrents.py will eventually cause the container to crash due to out of memory issues.

Steps to reproduce

docker-compose.yaml:

version: "2.1"
services:
  qbittorrent-10:
    image: lscr.io/linuxserver/qbittorrent:4.5.2
    container_name: qbittorrent-10
    volumes:
      - ./qbittorrent-10/config:/config
      - ./qbittorrent-10/downloads:/downloads
    environment:
      - WEBUI_PORT=8081
    ports:
      - 8081:8081
    restart: unless-stopped

  qbittorrent-100:
    image: lscr.io/linuxserver/qbittorrent:4.5.2
    container_name: qbittorrent-100
    volumes:
      - ./qbittorrent-100/config:/config
      - ./qbittorrent-100/downloads:/downloads
    environment:
      - WEBUI_PORT=8082
    ports:
      - 8082:8082
    restart: unless-stopped

  qbittorrent-1000:
    image: lscr.io/linuxserver/qbittorrent:4.5.2
    container_name: qbittorrent-1000
    volumes:
      - ./qbittorrent-1000/config:/config
      - ./qbittorrent-1000/downloads:/downloads
    environment:
      - WEBUI_PORT=8083
    ports:
      - 8083:8083
    restart: unless-stopped

fetch-torrents.py:

import sys
import qbittorrentapi

def fetch(port: int):
    client = qbittorrentapi.Client(
        host='localhost',
        port=port,
        username='admin',
        password='adminadmin'
     )
    client.auth_log_in()
    num_torrents = len(client.torrents_info())

if __name__ == '__main__':
    port = int(sys.argv[1])
    fetch(port)

add-torrents.py:

import sys
import os
import qbittorrentapi

def add_torrents(port: int, num_torrents: int):
    client = qbittorrentapi.Client(
        host='localhost',
        port=port,
        username='admin',
        password='adminadmin'
     )
    client.auth_log_in()

    torrent_files = []
    for torrent in os.listdir('./torrents'):
        filename = os.path.join('./torrents', torrent)
        torrent_files.append(filename)

    client.torrents_add(torrent_files=torrent_files[:num_torrents])

if __name__ == '__main__':
    port = int(sys.argv[1])
    num_torrents = int(sys.argv[2])
    add_torrents(port, num_torrents)

Full steps to reproduce. I am using two Python libraries torf-cli to locally edit .torrent files and qbittorrent-api to query the API. I am creating a .torrent file off a simple .txt file. I generate a thousand .torrent files by just modifying the source field in the metainfo field of the .torrent file.

$ pip install qbittorrent-api torf-cli
$ mkdir test-example
$ cd test-example
$ echo "Hello World!" > test.txt
$ torf test.txt -o base.torrent
$ mkdir torrents
$ for n in {1..01000}; do torf -i base.torrent -s "$n" -o "torrents/$n.torrent"; done > /dev/null
$ tree torrents/ | head -n 5
torrents/
├── 00001.torrent
├── 00002.torrent
├── 00003.torrent
├── 00004.torrent
├── 00005.torrent
$ mkdir qbittorrent-10 qbittorrent-100 qbittorrent-1000
$ mkdir qbittorrent-10/downloads qbittorrent-100/downloads qbittorrent-1000/downloads
$ cp test.txt qbittorrent-10/downloads/
$ cp test.txt qbittorrent-100/downloads/
$ cp test.txt qbittorrent-1000/downloads/
$ tree . | head -n 20
├── base.torrent
├── docker-compose.yml
├── fetch-torrents.py
├── add-torrents.py
├── qbittorrent-10
│   └── downloads
│       └── test.txt
├── qbittorrent-100
│   └── downloads
│       └── test.txt
├── qbittorrent-1000
│   └── downloads
│       └── test.txt
├── test.txt
└── torrents
    ├── 00001.torrent
    ├── 00002.torrent
    ├── 00003.torrent
    ├── 00004.torrent
    ├── 00005.torrent

$ docker compose -f docker-compose.yml up -d
[+] Running 4/4
 ⠿ Network test-example_default  Created               
 ⠿ Container qbittorrent-1000      Started                               
 ⠿ Container qbittorrent-10        Started       
 ⠿ Container qbittorrent-100       Started  
$ docker stats --no-stream       
CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
8e09d09ea340   qbittorrent-100    0.04%     12.96MiB / 1.936GiB   0.65%     3.66MB / 40.4kB   0B / 7.83MB       16
86e1ccdfa89c   qbittorrent-10     0.02%     16.06MiB / 1.936GiB   0.81%     3.66MB / 33.9kB   5.09MB / 7.83MB   16
59b407e1f51d   qbittorrent-1000   0.04%     12.84MiB / 1.936GiB   0.65%     3.67MB / 39.6kB   0B / 7.83MB       16                                                                                                              
$ python add-torrents.py 8081 10
$ python add-torrents.py 8082 100
$ python add-torrents.py 8083 1000
$ docker stats --no-stream 
CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O        PIDS
8e09d09ea340   qbittorrent-100    0.17%     24.28MiB / 1.936GiB   1.22%     3.78MB / 126kB    0B / 8.84MB      27
86e1ccdfa89c   qbittorrent-10     0.04%     25.45MiB / 1.936GiB   1.28%     3.75MB / 98.1kB   5.1MB / 7.96MB   27
59b407e1f51d   qbittorrent-1000   0.11%     41.53MiB / 1.936GiB   2.09%     4.03MB / 110kB    0B / 16.2MB      27
$ for n in {1..100}; do python fetch-torrents.py 8081; done
$ for n in {1..100}; do python fetch-torrents.py 8082; done
$ for n in {1..100}; do python fetch-torrents.py 8083; done
$ docker stats --no-stream 
CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O        PIDS
8e09d09ea340   qbittorrent-100    0.03%     70.79MiB / 1.936GiB   3.57%     3.96MB / 927kB    0B / 9.1MB       16
86e1ccdfa89c   qbittorrent-10     0.02%     30.14MiB / 1.936GiB   1.52%     3.93MB / 499kB    5.1MB / 7.97MB   16
59b407e1f51d   qbittorrent-1000   0.23%     508.9MiB / 1.936GiB   25.67%    4.22MB / 4.94MB   0B / 17MB        19
$ echo 3 > /proc/sys/vm/drop_caches
$ docker stats --no-stream
CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O        PIDS
8e09d09ea340   qbittorrent-100    0.03%     69.05MiB / 1.936GiB   3.48%     3.97MB / 930kB    0B / 9.1MB       16
86e1ccdfa89c   qbittorrent-10     0.02%     25.23MiB / 1.936GiB   1.27%     3.93MB / 502kB    5.1MB / 7.97MB   16
59b407e1f51d   qbittorrent-1000   0.07%     504MiB / 1.936GiB     25.42%    4.23MB / 4.94MB   336kB / 17.9MB   18

glassez commented 1 year ago

@jephdo IIRC, your test scripts log in into qBittorrent but never log out (unless qBittorrent-api does it by itself). Am I right?

burritothief commented 1 year ago

Ah thanks, I made the modification to explicitly log out and it completely frees the memory. You're correct.

Then my issue is that other clients don't do this either. As an example, you can search through the Radarr & Sonarr repos where they interact with qBittorrent and you can find a reference to /api/v2/auth/login, but there's no reference to /api/v2/auth/logout lol. So they are not logging out either which causes excessive memory usage

I guess I would just expect hitting the /api/v2/torrents/info endpoint to release the memory after the request-response cycle is over, rather than having to wait for the session to be over.

glassez commented 1 year ago

I guess I would just expect hitting the /api/v2/torrents/info endpoint to release the memory after the request-response cycle is over, rather than having to wait for the session to be over.

AFAIK, torrents/info does not store any data between calls. However, sessions themselves take up a certain amount of memory, so if you are constantly creating a new session, then memory consumption will still constantly grow. Of course, sessions have some inactive timeout, after which they are automatically deleted, but at a high rate of creating new sessions, it should be small enough to help. Unfortunately, qBittorrent has no limit on the number of simultaneously open sessions, which could better prevent problems like yours.

burritothief commented 1 year ago

I'll close this issue out since the best way to deal with this is to end the session by explicitly logging out

For anyone else, since some 3rd party clients don't implement the logout endpoint, my short term fix was to set the session timeout parameter to a much smaller value (I previously had it at 7 days), and I'm now using IP whitelisting to avoid logging in

qbittorrent / qBittorrent