rpc error: code = Unknown desc = docker does not support reopening container log files

[dyrnq]

May 31 10:43:19 master1 kubelet[61217]: E0531 10:43:19.425540   61217 remote_runtime.go:1134] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:19 master1 kubelet[61217]: E0531 10:43:19.425653   61217 container_log_manager.go:259] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log\": failed to reopen container log \"449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:29 master1 kubelet[61217]: E0531 10:43:29.477431   61217 remote_runtime.go:1134] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:29 master1 kubelet[61217]: E0531 10:43:29.477512   61217 container_log_manager.go:259] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log\": failed to reopen container log \"449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:39 master1 kubelet[61217]: E0531 10:43:39.523688   61217 remote_runtime.go:1134] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:39 master1 kubelet[61217]: E0531 10:43:39.523790   61217 container_log_manager.go:259] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log\": failed to reopen container log \"449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:49 master1 kubelet[61217]: E0531 10:43:49.573024   61217 remote_runtime.go:1134] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:49 master1 kubelet[61217]: E0531 10:43:49.573139   61217 container_log_manager.go:259] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log\": failed to reopen container log \"449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:59 master1 kubelet[61217]: E0531 10:43:59.617391   61217 remote_runtime.go:1134] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:59 master1 kubelet[61217]: E0531 10:43:59.617466   61217 container_log_manager.go:259] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log\": failed to reopen container log \"449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"

[dyrnq]

Oct 13 10:37:44 master1 kubelet[33321]: E1013 10:37:44.923134   33321 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log\": failed to reopen container log \"2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:37:55 master1 kubelet[33321]: E1013 10:37:55.003191   33321 remote_runtime.go:1145] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:37:55 master1 kubelet[33321]: E1013 10:37:55.005104   33321 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log\": failed to reopen container log \"2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:05 master1 kubelet[33321]: E1013 10:38:05.053292   33321 remote_runtime.go:1145] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:05 master1 kubelet[33321]: E1013 10:38:05.053382   33321 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log\": failed to reopen container log \"2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:15 master1 kubelet[33321]: E1013 10:38:15.154938   33321 remote_runtime.go:1145] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:15 master1 kubelet[33321]: E1013 10:38:15.155004   33321 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log\": failed to reopen container log \"2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:25 master1 kubelet[33321]: E1013 10:38:25.245699   33321 remote_runtime.go:1145] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:25 master1 kubelet[33321]: E1013 10:38:25.245759   33321 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log\": failed to reopen container log \"2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:35 master1 kubelet[33321]: E1013 10:38:35.296500   33321 remote_runtime.go:1145] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:35 master1 kubelet[33321]: E1013 10:38:35.297509   33321 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log\": failed to reopen container log \"2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"

[dyrnq]

xref https://github.com/Mirantis/cri-dockerd/issues/35

[dyrnq]

du -ch /var/lib/docker/containers/*/*-json.log
8.0K    /var/lib/docker/containers/054dd623cd6bf7adafe2ef798f52c3784211a57037e98377c0e3831e5eb43a8c/054dd623cd6bf7adafe2ef798f52c3784211a57037e98377c0e3831e5eb43a8c-json.log
0   /var/lib/docker/containers/06c5f7e3816b0ca0b4c320fd888fc5104fc039bd7627bf856e513bb20b7374eb/06c5f7e3816b0ca0b4c320fd888fc5104fc039bd7627bf856e513bb20b7374eb-json.log
8.0K    /var/lib/docker/containers/19eeca71f931c6c0f05fe4a3be7935e419dc1b17fff50fb9ad034faebf77bcd7/19eeca71f931c6c0f05fe4a3be7935e419dc1b17fff50fb9ad034faebf77bcd7-json.log
4.0K    /var/lib/docker/containers/1d13a725e32e9570ed9d183c1cffb010c9e2dba3728b9b8999e29ad461d6679b/1d13a725e32e9570ed9d183c1cffb010c9e2dba3728b9b8999e29ad461d6679b-json.log
8.0K    /var/lib/docker/containers/222330711eb536ea5a2497bb660331a0afcfb5d2141e1a088f8243b3fccde1da/222330711eb536ea5a2497bb660331a0afcfb5d2141e1a088f8243b3fccde1da-json.log
0   /var/lib/docker/containers/27a488e2adcebf74573c4fc7f2450bf3dc4954921e094323744d2e13585ff3bd/27a488e2adcebf74573c4fc7f2450bf3dc4954921e094323744d2e13585ff3bd-json.log
0   /var/lib/docker/containers/28ea42f81748823f91c30979bddd9717593a751dabc8454986561f903ef6e532/28ea42f81748823f91c30979bddd9717593a751dabc8454986561f903ef6e532-json.log
49M /var/lib/docker/containers/2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1/2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1-json.log
4.0K    /var/lib/docker/containers/3408e75fb16fab729b57a3ebbd593c62e1ee1b86b564d141033dc894458a7743/3408e75fb16fab729b57a3ebbd593c62e1ee1b86b564d141033dc894458a7743-json.log
0   /var/lib/docker/containers/3da81688306de6afb19af07f02a1bb0860fbaa3b62c938e55554789a99d98a19/3da81688306de6afb19af07f02a1bb0860fbaa3b62c938e55554789a99d98a19-json.log
72K /var/lib/docker/containers/498638d10c26bc0447e290655e814de4c32d259ec497bd06a656a910516fc86f/498638d10c26bc0447e290655e814de4c32d259ec497bd06a656a910516fc86f-json.log
0   /var/lib/docker/containers/598f7a4d0aa1741891b2bdf1dca80fc25f78b96543bb7e78e095915b86e80511/598f7a4d0aa1741891b2bdf1dca80fc25f78b96543bb7e78e095915b86e80511-json.log
4.0K    /var/lib/docker/containers/718384359f6e03cbe58a1ad02261aae691d6459dd42608814136f5ace6622ead/718384359f6e03cbe58a1ad02261aae691d6459dd42608814136f5ace6622ead-json.log
0   /var/lib/docker/containers/9495438663cd2bbb77f6f58dc816b79d2a6db7ae32e0aef43ba21d86d844238a/9495438663cd2bbb77f6f58dc816b79d2a6db7ae32e0aef43ba21d86d844238a-json.log
0   /var/lib/docker/containers/a720299627230f7361c4362d7807c744820ec3dd76108de7315137a325bc9d25/a720299627230f7361c4362d7807c744820ec3dd76108de7315137a325bc9d25-json.log
0   /var/lib/docker/containers/b36a9e80278c5556c460b09db4928ec2e487d6bd2514b52f8317b05adae7ff33/b36a9e80278c5556c460b09db4928ec2e487d6bd2514b52f8317b05adae7ff33-json.log
328K    /var/lib/docker/containers/bc55d8f8d66b7de59790043a4ba23ff7093c5e20b350c06f08e7e4a5e65cb0b9/bc55d8f8d66b7de59790043a4ba23ff7093c5e20b350c06f08e7e4a5e65cb0b9-json.log
276K    /var/lib/docker/containers/c81d8adbc8c6584764787b5b49fc1296265eab305a6fa57ce178eafc8ed12364/c81d8adbc8c6584764787b5b49fc1296265eab305a6fa57ce178eafc8ed12364-json.log
0   /var/lib/docker/containers/cb54179e8d951108eb0b9122b7eca28ba6e8a14fb775f1c7bdc91cb88fe6bdb3/cb54179e8d951108eb0b9122b7eca28ba6e8a14fb775f1c7bdc91cb88fe6bdb3-json.log
0   /var/lib/docker/containers/e01d8a93392dc19d2cddc3bd225a01b243710e0a44bffc1d8d03ac12d4d1ff8d/e01d8a93392dc19d2cddc3bd225a01b243710e0a44bffc1d8d03ac12d4d1ff8d-json.log
6.8M    /var/lib/docker/containers/e82d09e2764e826849ff7827c4572085bb5157270a228b300563b1524bc0e330/e82d09e2764e826849ff7827c4572085bb5157270a228b300563b1524bc0e330-json.log
57M total

[dyrnq]

docker version
Client:
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 22:56:42 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:45 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309f
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

cri-dockerd --version
cri-dockerd 0.2.6 (d8accf7)

kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.6", GitCommit:"b39bf148cd654599a52e867485c02c4f9d28b312", GitTreeState:"clean", BuildDate:"2022-09-21T13:19:24Z", GoVersion:"go1.18.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.6", GitCommit:"b39bf148cd654599a52e867485c02c4f9d28b312", GitTreeState:"clean", BuildDate:"2022-09-21T13:12:04Z", GoVersion:"go1.18.6", Compiler:"gc", Platform:"linux/amd64"}

[dyrnq]

maybe https://github.com/kubernetes/kubernetes/pull/59898 https://github.com/kubernetes/kubernetes/issues/110630

[dyrnq]

conflict --container-log-max-size with log-opts in daemon.json https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

--container-log-max-size string     Default: 10Mi

cat /etc/docker/daemon.json 
{
    "log-level": "info",
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m",
    "max-file": "5"
    }
}

[dyrnq]

Final solution pass --container-log-max-size=100Mi to kubelet args and systemctl daemon-reload && systemctl restart kubelet . let --container-log-max-size same with max-size in log-opts of /etc/docker/daemon.json !

[dyrnq]

Maybe use --feature-gates=CRIContainerLogRotation=false to close this feature. But CRIContainerLogRotation has removed in https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates-removed/

[dyrnq]

xref https://github.com/containerd/containerd/issues/4830

[dyrnq]

docker(conf log-opts in daemon.json) ---> containerd (no need config)

kubelet(conf containerLogMaxSize and containerLogMaxFiles in KubeletConfiguration ) ---> containerd (no need config)

nerdctl run (pass --log-opt) ---> containerd (no need config) eg

nerdctl run --log-driver=json-file --log-opt=max-size=100m --log-opt=max-file=10 -d --name mynginx nginx