Connectivity issue for rootless qbittorrent container using pasta driver

psyke83 commented 2 months ago

Issue Description

It seems that qbittorrent running rootless via the default pasta driver (specifically if launched via podman compose or a quadlet) has trouble acquiring an external IP, which breaks DHT connectivity. Reverting to the slirp4netns driver fixes the issue for compose/quadlet invocations, and running qbittorrent via command-line using otherwise identical arguments also seems to work OK.

Steps to reproduce the issue

After starting the qbittorrent container, navigate to web UI, then enable Logs in View menu. Click on Logs tab and check output for any line(s) matching Detected external IP: xxx.xxx.xxx.xxx. Additionally, watch the DHT node count at bottom of the web UI.

For testing the command-line invocation:

podman run -d \
--name=qbittorrent \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Etc/UTC \
-e WEBUI_PORT=8080 \
-e TORRENTING_PORT=6881 \
-p 8080:8080 \
-p 6881:6881 \
-p 6881:6881/udp \
-v ~/docker/qbittorrent/config:/config \
--restart unless-stopped \
lscr.io/linuxserver/qbittorrent:latest

Compose definition (same effect if converted to a quadlet):

services:
qbittorrent:
image: lscr.io/linuxserver/qbittorrent:latest
container_name: qbittorrent
environment:
  - PUID=1000
  - PGID=1000
  - TZ=Etc/UTC
  - WEBUI_PORT=8080
  - TORRENTING_PORT=6881
volumes:
  - ~/docker/qbittorrent/config:/config
ports:
  - 8080:8080
  - 6881:6881
  - 6881:6881/udp
restart: unless-stopped

Describe the results you received

When launched via command-line, external IP is resolved and DHT count will increase as normal. The pasta process will show these arguments:

/usr/bin/pasta --config-net -t 6881-6881:6881-6881 -t 8080-8080:8080-8080 -u 6881-6881:6881-6881 --dns-forward 169.254.0.1 -T none -U none --no-map-gw --quiet --netns /run/user/1000/netns/netns-5f81f8cc-b7b7-c851-af04-c24e80a4fddb

When launched via quadlet or compose with the default pasta driver, the external IP is not resolved in 95% of cases, and DHT count remains at 0. In some rare cases, the external IP is resolved after an extended delay, but the DHT count does not increase at the expected rate. Pasta process will show:

/usr/bin/pasta --config-net --pid /run/user/1000/containers/networks/rootless-netns/rootless-netns-conn.pid --dns-forward 169.254.0.1 -t none -u none -T none -U none --no-map-gw --quiet --netns /run/user/1000/containers/networks/rootless-netns/rootless-netns

When launched via quadlet/compose and default_rootless_network_cmd is set to slirp4netns, external IP resolution and DHT works correctly at all times.

Describe the results you expected

External IP and DHT connectivity should be working in all configurations.

podman info output

$ podman info
host:
  arch: amd64
  buildahVersion: 1.37.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-1:2.1.12-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: e8896631295ccb0bfdda4284f1751be19b483264'
  cpuUtilization:
    idlePercent: 87.55
    systemPercent: 4.06
    userPercent: 8.39
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2039
  hostname: hp255g7-arch
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.10.6-zen1-1-zen
  linkmode: dynamic
  logDriver: journald
  memFree: 684138496
  memTotal: 7173124096
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.12.1
    package: netavark-1.12.2-1
    path: /usr/lib/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.16.1-1
    path: /usr/bin/crun
    version: |-
      crun version 1.16.1
      commit: afa829ca0122bd5e1d67f1f38e6cc348027e3c32
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-2024_08_21.1d6142f-1
    version: |
      pasta 2024_08_21.1d6142f
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-1
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 2815983616
  swapTotal: 3586125824
  uptime: 10h 44m 20.00s (Approximately 0.42 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/conn/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 1
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/conn/.local/share/containers/storage
  graphRootAllocated: 392575868928
  graphRootUsed: 166278561792
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 11
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/conn/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 1724352649
  BuiltTime: Thu Aug 22 19:50:49 2024
  GitCommit: fcee48106a12dd531702d729d17f40f6e152027f
  GoVersion: go1.23.0
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2



### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

Yes

### Additional environment details

Arch Linux x86_64.

### Additional information

I can reproduce the exact same results with the same podman/pasta version on Arch Linux ARM aarch64 running on a Raspberry Pi 4.

sbrivio-rh commented 2 months ago

When launched via quadlet or compose with the default pasta driver [...] Pasta process will show:

/usr/bin/pasta --config-net --pid /run/user/1000/containers/networks/rootless-netns/rootless-netns-conn.pid --dns-forward 169.254.0.1 -t none -u none -T none -U none --no-map-gw --quiet --netns /run/user/1000/containers/networks/rootless-netns/rootless-netns

For some reason, ports are not forwarded at all (you get -t 6881-6881:6881-6881 -t 8080-8080:8080-8080 -u 6881-6881:6881-6881 -T none -U none when started by command line).

psyke83 commented 2 months ago

When launched via quadlet or compose with the default pasta driver [...] Pasta process will show:
/usr/bin/pasta --config-net --pid /run/user/1000/containers/networks/rootless-netns/rootless-netns-conn.pid --dns-forward 169.254.0.1 -t none -u none -T none -U none --no-map-gw --quiet --netns /run/user/1000/containers/networks/rootless-netns/rootless-netns
For some reason, ports are not forwarded at all (you get -t 6881-6881:6881-6881 -t 8080-8080:8080-8080 -u 6881-6881:6881-6881 -T none -U none when started by command line).

Yes, I noticed this discrepancy, but I'm not certain how pasta is supposed to operate in bridged mode (which seems to be the main networking difference between quadlet/compose vs command line invocation).

I tested the following:

pasta_options = ["-t", "6881-6881:6881-6881", "-t", "8080-8080:8080-8080", "-u", "6881-6881:6881-6881"] This results in the following error:

Error: rootlessport listen tcp 0.0.0.0:6881: bind: address already in use
Error: unable to start container "a1dcbf164f52ced90452666254b5d00005ce238e940fc1d228acf5b04034910d": rootlessport listen tcp 0.0.0.0:6881: bind: address already in use

pasta_options = ["-t", "auto", "-u", "auto"] Process spawns with the following arguments, but unfortunately does not improve the issue:

conn        4716  0.2  0.2  71932 15508 ?        Ss   15:24   0:00 /usr/bin/pasta --config-net -t auto -u auto --pid /run/user/1000/containers/networks/rootless-netns/rootless-netns-conn.pid --dns-forward 169.254.0.1 -T none -U none --no-map-gw --quiet --netns /run/user/1000/containers/networks/rootless-netns/rootless-netns

Luap99 commented 2 months ago

The bridge mode is special and never uses pasta for port forwarding, i.e. see https://github.com/containers/podman/discussions/22943#discussioncomment-9795883. But slirp4netns works in the same way so I do not see how this would cause any of this, i.e. port forwarding source ip is wrong (https://github.com/containers/podman/issues/8193).

How does qbittorrent actually try to figure out its external ip address?

psyke83 commented 2 months ago

I'm not well-versed on qBittorrent development, but my hunch is that it retrieves the ip address through libtorrent. So it would likely be this function which calls external_address() in libtorrent.

Although my test case doesn't apply (as I didn't set up an IPv6-enabled nework in the compose definition), after further testing I can see that rootless bridge mode has an issue resolving my IPv6 address even with slirp4netns.

In summary:

When network_mode is set to pasta or slirp4netns in the compose/quadlet definition, the external IPv4 and IPv6 addresses are retrieved correctly.
Rootless bridge with pasta driver cannot resolve any external addresses (5% of the time, an IPv4 address resolves after a delay, but DHT is still degraded)
Rootless bridge with slirp4netns driver only resolves the IPv4 external address.

github-actions[bot] commented 1 month ago

A friendly reminder that this issue had no activity for 30 days.

andsky-git commented 4 weeks ago

I have the same issue but for user created networks only. The default podman network works as expected. From what I can see pasta doesn't get the correct portmappings.


# default podman network
ps aux |grep -i pasta
user       3410  0.0  0.2  71940 12668 ?        Ss   10:18   0:00 /usr/bin/pasta --config-net -u 51413-51413:51413-51413 -t 9091-9091:9091-9091 -t 51413-51413:51413-51413 --dns-forward 169.254.0.1 -T none -U none --no-map-gw --quiet --netns /run/user/1000/netns/netns-1217136f-ebce-13d5-bb4c-ff59950580bd

# user created network
ps aux |grep -i pasta
user       4108  0.5  0.2  71936 12648 ?        Ss   10:43   0:00 /usr/bin/pasta --config-net --pid /run/user/1000/containers/networks/rootless-netns/rootless-netns-conn.pid --dns-forward 169.254.0.1 -t none -u none -T none -U none --no-map-gw --quiet --netns /run/user/1000/containers/networks/rootless-netns/rootless-netns

When i switch to slirp4netns it works like a charm.

Luap99 commented 3 weeks ago

user defined networks do not use pasta or slirp4netns for port forwarding, they use rootlessport: https://github.com/containers/podman/discussions/22943#discussioncomment-9795883

andsky-git commented 3 weeks ago

Thank you for the feedback. It is odd to me that changing to slirp4netns would make a difference if it's bypassed?

sbrivio-rh commented 3 weeks ago

It is odd to me that changing to slirp4netns would make a difference if it's bypassed?

It's probably due to a difference in the address that's reported to qbittorrent, see https://github.com/containers/podman/issues/23739#issuecomment-2310186061. I'm not sure how that affects things, more investigation is definitely needed.

andsky-git commented 3 weeks ago

To clarify I use another bittorrent client called transmission. I don't know if it uses the same libraries or not. Anyway I would be really happy if this issue could get resolved and I will gladly help out if you need more information.

sbrivio-rh commented 3 weeks ago

To clarify I use another bittorrent client called transmission. I don't know if it uses the same libraries or not. Anyway I would be really happy if this issue could get resolved and I will gladly help out if you need more information.

What would help here is some slightly deeper analysis of the issue (there's a start of it in https://github.com/containers/podman/issues/23739#issuecomment-2310186061): what do qbittorrent and Transmission expect to find as container address? How does the fact that slirp4netns does NAT, and the fact that pasta (by default) doesn't affect this? What happens if you specify options for pasta so that it behaves roughly like slirp4netns (look for pasta in man podman-run)? Which ones are actually needed?

containers / podman