Polling fails to run fping/fping6 due to lack of setcap support on Synology NAS (aufs storage backend)

silvester747 commented 1 year ago

Behaviour

The dispatcher cannot run fping and fping6 on Synology devices. For some reason the version of Docker for Synology NAS devices is hardcoded to run with the aufs storage backend. This backend does not support setcap. Instead it is required to use setuid to allow running fping and fping6. Would it be possible to adjust the images for this?

Steps to reproduce this issue

Run librenms on a Synology NAS, with the default Docker package from Synology.
Use the docker compose setup including rrdcached

Expected behaviour

Distributed polling should work.

Actual behaviour

Polling fails with a message that the host is not reachable. Validating the configuration of librenms indicates fping and fping6 cannot be executed without root.

Configuration

Docker version (type docker --version) : 20.10.3, build 55f0773
Docker compose version if applicable (type docker-compose --version) : 1.28.5, build 24fb474e
Platform (Debian 9, Ubuntu 18.04, ...) : Synology DSM 7.1.1-42962
System info (type uname -a) : Linux DS920 4.4.180+ #42962 SMP Tue Sep 20 22:35:56 CST 2022 x86_64 GNU/Linux synology_geminilake_920+
Include all necessary configuration files : docker-compose.yml, .env, ...

Docker info

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 9
  Running: 9
  Paused: 0
  Stopped: 0
 Images: 12
 Server Version: 20.10.3
 Storage Driver: aufs
  Root Dir: /volume1/@docker/aufs
  Backing Filesystem: extfs
  Dirs: 99
  Dirperm1 Supported: true
 Logging Driver: db
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs db fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3fa00912415f3e9c6f82dd72119179d599efd13b
 runc version: 31cc25f16f5eba4d0f53e35374532873744f4b31
 init version: ed96d00 (expected: de40ad0)
 Security Options:
  apparmor
 Kernel Version: 4.4.180+
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.678GiB
 Name: DS920
 ID: MBUX:VCM7:64RQ:UF2W:UJDO:MZCI:LWLN:4I6F:FHA5:FTSU:THC2:AMZ2
 Docker Root Dir: /volume1/@docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No kernel memory TCP limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No blkio weight support
WARNING: No blkio weight_device support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: the aufs storage-driver is deprecated, and will be removed in a future release.

Logs

Poller_0-17(WARNING):Polling device 8 unreachable, waiting 60s for retry
Poller_0-17(INFO):Completed poller run for 8 in 0.96s
Poller_0-18(WARNING):Polling device 2 unreachable, waiting 60s for retry
Poller_0-18(INFO):Completed poller run for 2 in 1.00s
Poller_0-19(WARNING):Polling device 1 unreachable, waiting 60s for retry
Poller_0-19(INFO):Completed poller run for 1 in 1.04s
Poller_0-3(WARNING):Polling device 6 unreachable, waiting 60s for retry
Poller_0-3(INFO):Completed poller run for 6 in 1.04s
Poller_0-12(WARNING):Polling device 7 unreachable, waiting 60s for retry
Poller_0-12(INFO):Completed poller run for 7 in 1.04s

silvester747 commented 1 year ago

Docker compose configuration:

name: librenms

services:
  db:
    image: mariadb:latest
    container_name: librenms_db
    command:
      - "mysqld"
      - "--innodb-file-per-table=1"
      - "--lower-case-table-names=0"
      - "--character-set-server=utf8mb4"
      - "--collation-server=utf8mb4_unicode_ci"
    volumes:
      - "${DB_VOLUME}:/var/lib/mysql"
    environment:
      - "TZ=${TZ}"
      - "MYSQL_ALLOW_EMPTY_PASSWORD=yes"
      - "MYSQL_DATABASE=${MYSQL_DATABASE}"
      - "MYSQL_USER=${MYSQL_USER}"
      - "MYSQL_PASSWORD=${MYSQL_PASSWORD}"
    restart: always

  redis:
    image: redis:5.0-alpine
    container_name: librenms_redis
    environment:
      - "TZ=${TZ}"
    restart: always

  rrdcached:
    image: crazymax/rrdcached
    container_name: librenms_rrdcached
    volumes:
      - "${RRD_VOLUME}:/data"
    environment:
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "LOG_LEVEL=LOG_INFO"
      - "WRITE_TIMEOUT=1800"
      - "WRITE_JITTER=1800"
      - "WRITE_THREADS=4"
      - "FLUSH_DEAD_DATA_INTERVAL=3600"
    restart: always

  msmtpd:
    image: crazymax/msmtpd:latest
    container_name: librenms_msmtpd
    env_file:
      - "stack.env"
    restart: always

  librenms:
    image: librenms/librenms:latest
    container_name: librenms
    hostname: librenms
    cap_add:
      - NET_ADMIN
      - NET_RAW
    ports:
      - target: 8000
        published: 8000
        protocol: tcp
    depends_on:
      - db
      - redis
      - rrdcached
      - msmtpd
    volumes:
      - "${DATA_VOLUME}:/data"
    env_file:
      - "stack.env"
    environment:
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "DB_HOST=db"
      - "DB_NAME=${MYSQL_DATABASE}"
      - "DB_USER=${MYSQL_USER}"
      - "DB_PASSWORD=${MYSQL_PASSWORD}"
      - "DB_TIMEOUT=60"
    restart: always

  dispatcher:
    image: librenms/librenms:latest
    container_name: librenms_dispatcher
    hostname: librenms-dispatcher
    cap_add:
      - NET_ADMIN
      - NET_RAW
    depends_on:
      - librenms
      - redis
    volumes:
      - "${DATA_VOLUME}:/data"
    env_file:
      - "stack.env"
    environment:
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "DB_HOST=db"
      - "DB_NAME=${MYSQL_DATABASE}"
      - "DB_USER=${MYSQL_USER}"
      - "DB_PASSWORD=${MYSQL_PASSWORD}"
      - "DB_TIMEOUT=60"
      - "DISPATCHER_NODE_ID=dispatcher1"
      - "SIDECAR_DISPATCHER=1"
    restart: always

  syslogng:
    image: librenms/librenms:latest
    container_name: librenms_syslogng
    hostname: librenms-syslogng
    cap_add:
      - NET_ADMIN
      - NET_RAW
    depends_on:
      - librenms
      - redis
    ports:
      - target: 514
        published: 514
        protocol: tcp
      - target: 514
        published: 514
        protocol: udp
    volumes:
      - "${DATA_VOLUME}:/data"
    env_file:
      - "stack.env"
    environment:
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "DB_HOST=db"
      - "DB_NAME=${MYSQL_DATABASE}"
      - "DB_USER=${MYSQL_USER}"
      - "DB_PASSWORD=${MYSQL_PASSWORD}"
      - "DB_TIMEOUT=60"
      - "SIDECAR_SYSLOGNG=1"
    restart: always

  snmptrapd:
    image: librenms/librenms:latest
    container_name: librenms_snmptrapd
    hostname: librenms-snmptrapd
    cap_add:
      - NET_ADMIN
      - NET_RAW
    depends_on:
      - librenms
      - redis
    ports:
      - target: 162
        published: 162
        protocol: tcp
      - target: 162
        published: 162
        protocol: udp
    volumes:
      - "${DATA_VOLUME}:/data"
    env_file:
      - "stack.env"
    environment:
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "DB_HOST=db"
      - "DB_NAME=${MYSQL_DATABASE}"
      - "DB_USER=${MYSQL_USER}"
      - "DB_PASSWORD=${MYSQL_PASSWORD}"
      - "DB_TIMEOUT=60"
      - "SIDECAR_SNMPTRAPD=1"
    restart: always

silvester747 commented 1 year ago

I have opened a support case with Synology as well to get aufs replaced with overlay2. In the meantime I hope we can come up with a fix for these docker images to allow running the required commands on aufs. The setuid approach works, but I am not sure whether that is acceptable for everybody.

silvester747 commented 1 year ago

Synology has responded that currently they do not support overlay2 yet, but my request has been passed on to the development team. There seems to be support for the btrfs backend, but that requires a DiskStation with btrfs support and a btrfs volume. Migrating my current ext4 volumes is not something I plan to do on the short term.

Would adding the setuid bits be acceptable for the Docker image? I can create a PR for it.

crazy-max commented 1 year ago

aufs storage driver is deprecated for almost 6 years and has been removed in recent version of Docker: https://github.com/moby/moby/pull/45342

librenms / docker