No workarounds for logging spam "docker does not support reopening container log files", when not using json-file logging driver

[benileo]

Expected Behavior

Mar 18 14:54:14 kubelet[35502]: E0318 14:54:14.509505   35502 container_log_manager.go:253] "Container log doesn't exist, reopen container log failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" 

Mar 18 14:54:14 kubelet[35502]: E0318 14:54:14.512587   35502 remote_runtime.go:751] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" 

The SPAM is real.... for a 10 node cluster we had over 200,000 log lines of this nature in 1 hour. There are no "workarounds" for users who do not use the 'json-file' logging driver. Suggested workaround, from https://github.com/Mirantis/cri-dockerd/issues/35

fixed when setting container-log-max-size to the same size as docker daemon max-size log opts.

Are users just subject to the pain? I saw in one thread that using a stub logging manager with cri-dockerd could work. Is there any documentation to help implement this?

This is our daemon configuration.

{
    "max-concurrent-downloads": 6,
    "exec-opts": [
        "native.cgroupdriver=systemd"
    ],
    "storage-driver": "overlay2",
    "live-restore": true,
    "log-driver": "fluentd",
    "log-opts": {
        ...
    }
}

Specifications

• Version: Kubernetes 1.29.2
• Platform: v0.3.11/cri-dockerd_0.3.11.3-0.ubuntu-jammy_amd64.deb

[benileo]

For any other poor souls out there. I configured syslog to drop the message matching that pattern. Not super happy with the workaround.

/etc/rsyslog.d/30-cri-spam.conf :msg, contains, "docker does not support reopening container log files" stop

[zarqman]

Not sure what version of docker this was added in, but current versions of docker default to dual-logging when using a remote logging driver (to facilitate the docker logs command, and likely kubectl logs). That is, they log to remote and local. This is pretty obscure in the docs and easy to miss as it's not well referenced from the remote logging driver pages. https://docs.docker.com/engine/logging/dual-logging/

Dual-logging can be disabled with the cache-disabled logging opt. Or, you can set cache-max-size (comparable to max-size when using local logging drivers, as referenced in #35), which will allow docker logs to still work.

One additional hint for anyone running across this: if setting the above in docker's daemon.json, you'll have to recreate all containers. The logging config for existing containers is not updated.