joes
commented
2 years ago I "fixed" the issue by draining, restarting server, and uncordoning all nodes.
An "interesting" (or totally expected thing) is that microk8s ctr c info [id] no longer provides any information about the containers that failed to create previously.
sudo journalctl --since "10 minutes ago" \
> | grep "Failed to create existing container" \
> | grep "task" \
> | awk -F ': task ' '{print $2}' \
> | awk -F ' not found: ' '{print $1}' \
> | sort -u \
> | xargs -I {} -n 1 bash -c "echo Failed to create existing container {}; microk8s ctr c list | grep {}; echo ''"Failed to create existing container 1da2b0e7abac49f5b0a59f625410d02e5d09b23588f5a8136eba980971e7a772
Failed to create existing container 286ebbf78e48b36dd32f01a7324dcebb424c2a09109eaea273b321d27310801e
Failed to create existing container 37bb5ef545f621987209d3abe1e877edbb1ff6243ba4c5eeb3bc2fbb7affeb15
Failed to create existing container 4b48154390c73e14da20f37624dd37b15449276639f98c8ac2eb822914fbf115
Failed to create existing container 5b1d0ac2dbd040707934b772722a4ffc777480c60576f0bf11be300e2e3e567a
Failed to create existing container 5e5558e9f2a4b3c12a08114945c747ec3254ced3cd811d68c6cb6d22bdda67c3
Failed to create existing container 6ced94e2753519017411d4b0d05cc7f51576b15d63eb1024483ef7e571ba458c
Failed to create existing container 7a999c211e5643861782b80a0d4369bac8db1e5fa80eb1b08d1b123ac7c12799
Failed to create existing container 8a5a50caef481ad5f099b4c7d18daae34caf97220e8cc8702f50d7833f515866
Failed to create existing container 9bc948c868d2e39cba5d3169a2590b10f4cb5bd0c5860a5504ea2e8649f1090b
Failed to create existing container c96c113358b1370733c3ae41af416045292d7580887c542dcd00d7d04ef124e9
Failed to create existing container d6f1b4c1e7f6597f72a8c54c2ba7b89e51356bdc52263e99c457e39480465b7b
Failed to create existing container ec631ea825dbce4496ca8be2c12a89b5f55a0489f456344128d84c2d972e104a
Failed to create existing container edd488456797599d06c16aefcd2e92ac5c0c6e827b9eafe7ca01f1145ce20e4aSo something (possibly out-dated) regarding containers seems to have been cleared/reset here after the restart.
Azbesciak
I am getting a lot of "Failed to create existing container" errors in the logs.
$ sudo journalctl --since "1 minute ago" | grep "Failed to create existing container"If I parse the logs and feed the ID:s to
microk8s ctr c infoI find these containers and related images:Output:
What might be the cause of this? Any help or pointers are much appreciated.
$ microk8s kubectl version$ microk8s status$ microk8s kubectl get nodes