Container won't start on Fedora 39 Server

[bramvanoploo]

I'm trying to run this excellent solution on a server (Fedora 39 Server). It's a fresh and clean fedora installation. This image and portainer are the only containers installed. I've been using your image for at least 3 years without problems, on my Unraid server. But for some reason it just won't start on my new Fedora server. All the information I have is the log, which I'll include below. I'm by no means a docker expert, all though I'm comfortable with it and eager to learn. Can you point me in the right direction? Is this a bug, or my mistake? The configuration for the container is identical to the one on my Unraid server. I've also tried using macvlan instead of host, but that didn't change anything.

I've also manually added the ppd driver and AirPrint-Phaser-6280DN.service from my unraid server, to see if that would fix the problem. Clearly not. That's why these are shown in the log.

Can you please help me out?

+ '[' -z ******* ]
+ '[' -z ******* ]
+ grep -ci ******* /etc/shadow
+ '[' 1 -eq 0 ]
+ echo *******:*******
+ chpasswd
chpasswd: password for '*******' changed
+ mkdir -p /config/ppd
+ mkdir -p /services
+ rm -rf /etc/avahi/services/AirPrint-Phaser-6280DN.service
+ rm -rf /etc/cups/ppd
+ ln -s /config/ppd /etc/cups
+ ls -l /services/AirPrint-Phaser-6280DN.service
+ wc -l
+ '[' 1 -gt 0 ]
+ cp -f /services/AirPrint-Phaser-6280DN.service /etc/avahi/services/
+ ls -l /config/printers.conf
+ wc -l
+ '[' 1 -eq 0 ]
+ cp /config/printers.conf /etc/cups/printers.conf
+ ls -l /config/cupsd.conf
+ wc -l
+ '[' 0 -ne 0 ]
+ /usr/sbin/avahi-daemon --daemonize
Process 30 died: No such process; trying to remove PID file. (/run/avahi-daemon//pid)
+ exec /usr/sbin/cupsd -f
+ /root/printer-update.sh
Setting up watches.
Watches established.

[chuckcharlie]

Could you share your docker command or docker compose (without password)? Is this ARM64 or AMD64?

[bramvanoploo]

x64 (intel i3 from 2016).

Thanks you for your swift response! I'm using Portainer. no docker run or docker-compose in this case. But I can share the inspect object (credentials redacted). It contains all the information:

{
    "AppArmorProfile": "",
    "Args": [],
    "Config": {
        "AttachStderr": false,
        "AttachStdin": false,
        "AttachStdout": false,
        "Cmd": [
            "/root/run_cups.sh"
        ],
        "Domainname": "",
        "Entrypoint": null,
        "Env": [
            "CUPSADMIN=*****",
            "CUPSPASSWORD=*****",
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "ExposedPorts": {
            "631/tcp": {}
        },
        "Hostname": "nuc-server",
        "Image": "docker.io/chuckcharlie/cups-avahi-airprint:latest",
        "Labels": {
            "description": "AIRPRINT FROM SYNOLOGY DSM 7 (HP, SAMSUNG, ETC)",
            "maintainer": "znetwork@me.com",
            "org.opencontainers.image.ref.name": "ubuntu",
            "org.opencontainers.image.version": "22.04",
            "version": "0.3"
        },
        "OnBuild": null,
        "OpenStdin": true,
        "StdinOnce": false,
        "Tty": true,
        "User": "",
        "Volumes": {
            "/config": {},
            "/services": {}
        },
        "WorkingDir": ""
    },
    "Created": "2023-12-05T21:40:13.582945936Z",
    "Driver": "overlay2",
    "ExecIDs": null,
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/5b119329711594bba8170e11df22f2af18a13adb011271beee8094b9fe86cee9-init/diff:/var/lib/docker/overlay2/ec1e2d0e24d4bf774d69537892ff9682f901221c39ae687cf07ec6a8a1dad12f/diff:/var/lib/docker/overlay2/1f7261bf8ea40ce6a8fc7e74886f7276428fbc98679c4133d027391d47545a64/diff:/var/lib/docker/overlay2/89d4a38a15e4a31a6393e16265edde9f657ba3c817d65ea2835d389dbe61c219/diff:/var/lib/docker/overlay2/c65697f7fde750647e3d7acc727394a987cf7576324e4d32bfcddbe9aeaedede/diff:/var/lib/docker/overlay2/6c134b15c238bf1d158c23df8f2411c220eb4890632c3b18348ab4c79e171d56/diff",
            "MergedDir": "/var/lib/docker/overlay2/5b119329711594bba8170e11df22f2af18a13adb011271beee8094b9fe86cee9/merged",
            "UpperDir": "/var/lib/docker/overlay2/5b119329711594bba8170e11df22f2af18a13adb011271beee8094b9fe86cee9/diff",
            "WorkDir": "/var/lib/docker/overlay2/5b119329711594bba8170e11df22f2af18a13adb011271beee8094b9fe86cee9/work"
        },
        "Name": "overlay2"
    },
    "HostConfig": {
        "AutoRemove": false,
        "Binds": [
            "/docker/data/cups-avahi-airprint/config:/config",
            "/docker/data/cups-avahi-airprint/services:/services"
        ],
        "BlkioDeviceReadBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceWriteIOps": null,
        "BlkioWeight": 0,
        "BlkioWeightDevice": null,
        "CapAdd": [
            "AUDIT_WRITE",
            "CHOWN",
            "DAC_OVERRIDE",
            "FOWNER",
            "FSETID",
            "KILL",
            "MKNOD",
            "NET_BIND_SERVICE",
            "NET_RAW",
            "SETFCAP",
            "SETGID",
            "SETPCAP",
            "SETUID",
            "SYS_CHROOT"
        ],
        "CapDrop": [
            "AUDIT_CONTROL",
            "BLOCK_SUSPEND",
            "DAC_READ_SEARCH",
            "IPC_LOCK",
            "IPC_OWNER",
            "LEASE",
            "LINUX_IMMUTABLE",
            "MAC_ADMIN",
            "MAC_OVERRIDE",
            "NET_ADMIN",
            "NET_BROADCAST",
            "SYSLOG",
            "SYS_ADMIN",
            "SYS_BOOT",
            "SYS_MODULE",
            "SYS_NICE",
            "SYS_PACCT",
            "SYS_PTRACE",
            "SYS_RAWIO",
            "SYS_RESOURCE",
            "SYS_TIME",
            "SYS_TTY_CONFIG",
            "WAKE_ALARM"
        ],
        "Cgroup": "",
        "CgroupParent": "",
        "CgroupnsMode": "private",
        "ConsoleSize": [
            0,
            0
        ],
        "ContainerIDFile": "",
        "CpuCount": 0,
        "CpuPercent": 0,
        "CpuPeriod": 0,
        "CpuQuota": 0,
        "CpuRealtimePeriod": 0,
        "CpuRealtimeRuntime": 0,
        "CpuShares": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "DeviceCgroupRules": null,
        "DeviceRequests": [],
        "Devices": [],
        "Dns": [],
        "DnsOptions": [],
        "DnsSearch": [],
        "ExtraHosts": [],
        "GroupAdd": null,
        "IOMaximumBandwidth": 0,
        "IOMaximumIOps": 0,
        "Init": false,
        "IpcMode": "private",
        "Isolation": "",
        "Links": null,
        "LogConfig": {
            "Config": {},
            "Type": "json-file"
        },
        "MaskedPaths": [
            "/proc/asound",
            "/proc/acpi",
            "/proc/kcore",
            "/proc/keys",
            "/proc/latency_stats",
            "/proc/timer_list",
            "/proc/timer_stats",
            "/proc/sched_debug",
            "/proc/scsi",
            "/sys/firmware",
            "/sys/devices/virtual/powercap"
        ],
        "Memory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "MemorySwappiness": null,
        "NanoCpus": 0,
        "NetworkMode": "host",
        "OomKillDisable": null,
        "OomScoreAdj": 0,
        "PidMode": "",
        "PidsLimit": null,
        "PortBindings": {},
        "Privileged": false,
        "PublishAllPorts": false,
        "ReadonlyPaths": [
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ],
        "ReadonlyRootfs": false,
        "RestartPolicy": {
            "MaximumRetryCount": 0,
            "Name": "no"
        },
        "Runtime": "runc",
        "SecurityOpt": null,
        "ShmSize": 67108864,
        "UTSMode": "",
        "Ulimits": null,
        "UsernsMode": "",
        "VolumeDriver": "",
        "VolumesFrom": null
    },
    "HostnamePath": "/var/lib/docker/containers/7b4170b8add1c39f5fe23688d1095459c65261e45110f0c81c4cdddd2878fcb9/hostname",
    "HostsPath": "/var/lib/docker/containers/7b4170b8add1c39f5fe23688d1095459c65261e45110f0c81c4cdddd2878fcb9/hosts",
    "Id": "7b4170b8add1c39f5fe23688d1095459c65261e45110f0c81c4cdddd2878fcb9",
    "Image": "sha256:f44f20da747c4379469f10a83ad4b2ea06e55d4ada3e1e6659b10ef51fdefb00",
    "LogPath": "/var/lib/docker/containers/7b4170b8add1c39f5fe23688d1095459c65261e45110f0c81c4cdddd2878fcb9/7b4170b8add1c39f5fe23688d1095459c65261e45110f0c81c4cdddd2878fcb9-json.log",
    "MountLabel": "",
    "Mounts": [
        {
            "Destination": "/config",
            "Mode": "",
            "Propagation": "rprivate",
            "RW": true,
            "Source": "/docker/data/cups-avahi-airprint/config",
            "Type": "bind"
        },
        {
            "Destination": "/services",
            "Mode": "",
            "Propagation": "rprivate",
            "RW": true,
            "Source": "/docker/data/cups-avahi-airprint/services",
            "Type": "bind"
        }
    ],
    "Name": "/cups-avahi-airprint",
    "NetworkSettings": {
        "Bridge": "",
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "HairpinMode": false,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "MacAddress": "",
        "Networks": {
            "host": {
                "Aliases": null,
                "DriverOpts": null,
                "EndpointID": "",
                "Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAMConfig": {},
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "Links": null,
                "MacAddress": "",
                "NetworkID": "f4b98d592b110fa3c693051145cecac00cab8393bfafb67e6a72fa59fc701f01"
            }
        },
        "Ports": {},
        "SandboxID": "e6c50c1ef8b5c5e65e996ba93c21b0ab1338f62a73700777bc94fd67d2406403",
        "SandboxKey": "/var/run/docker/netns/default",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    },
    "Path": "/root/run_cups.sh",
    "Platform": "linux",
    "Portainer": {
        "ResourceControl": {
            "Id": 72,
            "ResourceId": "7b4170b8add1c39f5fe23688d1095459c65261e45110f0c81c4cdddd2878fcb9",
            "SubResourceIds": [],
            "Type": 1,
            "UserAccesses": [],
            "TeamAccesses": [],
            "Public": false,
            "AdministratorsOnly": true,
            "System": false
        }
    },
    "ProcessLabel": "",
    "ResolvConfPath": "/var/lib/docker/containers/7b4170b8add1c39f5fe23688d1095459c65261e45110f0c81c4cdddd2878fcb9/resolv.conf",
    "RestartCount": 0,
    "State": {
        "Dead": false,
        "Error": "",
        "ExitCode": 1,
        "FinishedAt": "2023-12-05T21:40:14.160883184Z",
        "OOMKilled": false,
        "Paused": false,
        "Pid": 0,
        "Restarting": false,
        "Running": false,
        "StartedAt": "2023-12-05T21:40:14.03560064Z",
        "Status": "exited"
    }
}

[dkisselev]

I'm also trying to get it running on Fedora server and running into the same issue.

My guess is that avahi is running into a port conflict with something that's running by default on Fedora (but not on rpi images) but the logs don't really show why it's crashing. I also tried opening an interactive shell into the container and manually running avahi-daemon, which it had no problem doing.

From here, I found their fix change where they removed -D on Avahi.

I played around with changing run_cups.sh to call /usr/sbin/avahi-daemon -s & and rebuilding the container but the error messages shift to Daemon already running

[...]
+ cp /config/printers.conf /etc/cups/printers.conf
+ ls -l /config/cupsd.conf
+ wc -l
+ '[' 0 -ne 0 ]
+ /usr/sbin/avahi-daemon -s
+ exec /usr/sbin/cupsd -f
+ /root/printer-update.sh
Setting up watches.
Daemon already running on PID 25
Watches established.

After which the container stops and attempts to start again.

[chuckcharlie]

Just wanted to let you know that I have reproduced the issue. Not sure why it is happening but I will dig into it more this week.

[chuckcharlie]

From my research, there have been a bunch of issues with containerd on Fedora 39. Assuming you are up to date, give this a try:

1. sudo systemctl stop docker

2. sudo vi /lib/systemd/system/containerd.service

Change LimitNOFILE value from infinity to 1048576:

#LimitNOFILE=infinity
LimitNOFILE=1048576

3. reload daemon and restart containerd

   sudo systemctl daemon-reload
   sudo systemctl restart containerd

4. sudo systemctl start docker

[chuckcharlie]

I was able to do this on a fresh install of Fedora Server 39 (using official Docker install method), and printed successfully. Let me know how it goes.

[dkisselev]

Thanks! That fixed it for me. Do you have a source for where you found the LimitNOFILE tweak?

[chuckcharlie]

No problem. Stumbled on the discussion here: https://github.com/containerd/containerd/pull/7566

[bramvanoploo]

Thank you very much @chuckcharlie ! Issues solved!