Open stevefan1999-personal opened 5 years ago
Well, I was able to partially recover by desperately deleting all container storages in /var/lib/docker/containers
for usual people (in my case it is not), although all my containers are lost, I was lucky enough to have compose files remained, but I'm still curious why was this triggered so I want this to remain open. Was this correlated to Docker Swarm networking?
This solved my issue today. The dockerd was failing to start and deleting the container folder made it work.
here are the logs
time="2023-02-09T13:19:24.011614000+03:00" level=info msg="libcontainerd: new containerd process, pid: 12753"
time="2023-02-09T13:19:25.081397000+03:00" level=info msg="[graphdriver] using prior storage driver: btrfs"
time="2023-02-09T13:19:26.518112000+03:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2023-02-09T13:19:26.518735000+03:00" level=warning msg="Your kernel does not support cgroup cpu shares"
time="2023-02-09T13:19:26.518810000+03:00" level=warning msg="Your kernel does not support cgroup cfs period"
time="2023-02-09T13:19:26.518864000+03:00" level=warning msg="Your kernel does not support cgroup cfs quotas"
time="2023-02-09T13:19:26.518913000+03:00" level=warning msg="Your kernel does not support cgroup rt period"
time="2023-02-09T13:19:26.518986000+03:00" level=warning msg="Your kernel does not support cgroup rt runtime"
time="2023-02-09T13:19:26.520431000+03:00" level=info msg="Loading containers: start."
time="2023-02-09T13:19:27.841848000+03:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (9cbc59e0e70f75548a25cdb28f245f573681c2e2248f026ad45f3b3e73b8e40e)."
time="2023-02-09T13:19:27.842918000+03:00" level=warning msg="libcontainerd: failed to retrieve container 9cbc59e0e70f75548a25cdb28f245f573681c2e2248f026ad45f3b3e73b8e40e state: rpc error: code = Unknown desc = containerd: container not found"
time="2023-02-09T13:19:28.396435000+03:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (a35e29d7d7a3e52e6ffc506d83838e0ca43c07ace595e97c1e3fb9f81f2f4878)."
time="2023-02-09T13:19:28.397310000+03:00" level=warning msg="libcontainerd: failed to retrieve container a35e29d7d7a3e52e6ffc506d83838e0ca43c07ace595e97c1e3fb9f81f2f4878 state: rpc error: code = Unknown desc = containerd: container not found"
time="2023-02-09T13:19:28.853721000+03:00" level=warning msg="Running modprobe nf_nat failed with message: ``, error: exit status 255"
time="2023-02-09T13:19:28.857584000+03:00" level=warning msg="Running modprobe xt_conntrack failed with message: ``, error: exit status 255"
time="2023-02-09T13:19:29.243959000+03:00" level=warning msg="Could not load necessary modules for IPSEC rules: Running modprobe xfrm_user failed with message: ``, error: exit status 255"
time="2023-02-09T13:19:29.667642000+03:00" level=info msg="Removing stale sandbox ddec5b6a1b82c06937def42d7309151e965ee4ac4e731af60a10679f406bb6a4 (9cbc59e0e70f75548a25cdb28f245f573681c2e2248f026ad45f3b3e73b8e40e)"
time="2023-02-09T13:19:31.263461000+03:00" level=error msg="getNetworkFromStore for nid dc97692c1acf66b31c007cb3c5128273af4cc4d8941ab72bdf5e48daef0ca498 failed while trying to build sandbox for cleanup: network dc97692c1acf66b31c007cb3c5128273af4cc4d8941ab72bdf5e48daef0ca498 not found"
time="2023-02-09T13:19:31.263693000+03:00" level=error msg="getNetworkFromStore for nid 4ba1ee99bb14ac5b8af74fa4a930bca0ae2640f72dac2e61c6a0e25ad5e0fb8c failed while trying to build sandbox for cleanup: network 4ba1ee99bb14ac5b8af74fa4a930bca0ae2640f72dac2e61c6a0e25ad5e0fb8c not found"
time="2023-02-09T13:19:31.263816000+03:00" level=error msg="getNetworkFromStore for nid 19ef5e0dc49f99a0e5943e274e53ce599d1815e30face41ca9ff8e0c10fdab5a failed while trying to build sandbox for cleanup: network 19ef5e0dc49f99a0e5943e274e53ce599d1815e30face41ca9ff8e0c10fdab5a not found"
time="2023-02-09T13:19:31.263914000+03:00" level=error msg="getNetworkFromStore for nid 733fd239aea303ad2fe620872c0154e80dc57efb0480d5b28c5ece6274bbedc4 failed while trying to build sandbox for cleanup: network 733fd239aea303ad2fe620872c0154e80dc57efb0480d5b28c5ece6274bbedc4 not found"
time="2023-02-09T13:19:31.264007000+03:00" level=info msg="Removing stale sandbox 635b527878465b8a88ee95a417554431473284205dcb2874d991a8cd20712e23 (f6f3bd530c83ea610cfaf6417d314a11bc9bcd68e31e68028a54d4437993897c)"
time="2023-02-09T13:19:31.264112000+03:00" level=warning msg="Failed getting network for ep 4c5b7863bbcb7eb73d79eb1330005137c2f845ef77b3670a255965ee9c631145 during sandbox 635b527878465b8a88ee95a417554431473284205dcb2874d991a8cd20712e23 delete: network dc97692c1acf66b31c007cb3c5128273af4cc4d8941ab72bdf5e48daef0ca498 not found"
time="2023-02-09T13:19:31.264240000+03:00" level=warning msg="Failed getting network for ep 950fa6c773aa9c5de8d9108e237573e886a3d20b9f485e69162282e8b6d65b13 during sandbox 635b527878465b8a88ee95a417554431473284205dcb2874d991a8cd20712e23 delete: network 4ba1ee99bb14ac5b8af74fa4a930bca0ae2640f72dac2e61c6a0e25ad5e0fb8c not found"
time="2023-02-09T13:19:31.264350000+03:00" level=warning msg="Failed getting network for ep e3c919fdf89c328b5d782890593d640e4eaf0b8f77eaf407b19963c0f6fc8f0f during sandbox 635b527878465b8a88ee95a417554431473284205dcb2874d991a8cd20712e23 delete: network 19ef5e0dc49f99a0e5943e274e53ce599d1815e30face41ca9ff8e0c10fdab5a not found"
time="2023-02-09T13:19:31.264455000+03:00" level=warning msg="Failed getting network for ep 42ab6750127879852b959586e6c338e14e4002516549860fab1ce78ad80f7aa4 during sandbox 635b527878465b8a88ee95a417554431473284205dcb2874d991a8cd20712e23 delete: network 733fd239aea303ad2fe620872c0154e80dc57efb0480d5b28c5ece6274bbedc4 not found"
time="2023-02-09T13:19:31.264531000+03:00" level=error msg="Failed to delete sandbox 635b527878465b8a88ee95a417554431473284205dcb2874d991a8cd20712e23 while trying to cleanup: could not cleanup all the endpoints in container f6f3bd530c83ea610cfaf6417d314a11bc9bcd68e31e68028a54d4437993897c / sandbox 635b527878465b8a88ee95a417554431473284205dcb2874d991a8cd20712e23"
time="2023-02-09T13:19:31.469131000+03:00" level=info msg="Removing stale sandbox 8e702c874246c557f6cc45c7c347273f0a548e7fcfa5b4d91ab07fef9cc80424 (a35e29d7d7a3e52e6ffc506d83838e0ca43c07ace595e97c1e3fb9f81f2f4878)"
time="2023-02-09T13:19:33.126291000+03:00" level=error msg="getNetworkFromStore for nid dc97692c1acf66b31c007cb3c5128273af4cc4d8941ab72bdf5e48daef0ca498 failed while trying to build sandbox for cleanup: network dc97692c1acf66b31c007cb3c5128273af4cc4d8941ab72bdf5e48daef0ca498 not found"
time="2023-02-09T13:19:33.126502000+03:00" level=error msg="getNetworkFromStore for nid 0c5085b1b048cd809cab3f7c930d0a3d6d3204be6f50f6dd97831936597114dd failed while trying to build sandbox for cleanup: network 0c5085b1b048cd809cab3f7c930d0a3d6d3204be6f50f6dd97831936597114dd not found"
time="2023-02-09T13:19:33.126783000+03:00" level=info msg="Removing stale sandbox 9b539f767ae0793949bf6214fca5bb588a1d1e102c857920c4810dc4152f7ebd (8840fc4cf353614867786a8e907363085684e5245a02451d62a145d798d8382b)"
time="2023-02-09T13:19:33.126878000+03:00" level=warning msg="Failed getting network for ep e4b994e3382d4ac15a94580b19e00040d62da2c1fe14888716131191845a425b during sandbox 9b539f767ae0793949bf6214fca5bb588a1d1e102c857920c4810dc4152f7ebd delete: network dc97692c1acf66b31c007cb3c5128273af4cc4d8941ab72bdf5e48daef0ca498 not found"
time="2023-02-09T13:19:33.126984000+03:00" level=warning msg="Failed getting network for ep 441892459cff0e33162245531f2cc92c34c62d5d9636f21cfd842319a89818c4 during sandbox 9b539f767ae0793949bf6214fca5bb588a1d1e102c857920c4810dc4152f7ebd delete: network 0c5085b1b048cd809cab3f7c930d0a3d6d3204be6f50f6dd97831936597114dd not found"
time="2023-02-09T13:19:36.889028000+03:00" level=error msg="Failed to delete sandbox 9b539f767ae0793949bf6214fca5bb588a1d1e102c857920c4810dc4152f7ebd while trying to cleanup: could not cleanup all the endpoints in container 8840fc4cf353614867786a8e907363085684e5245a02451d62a145d798d8382b / sandbox 9b539f767ae0793949bf6214fca5bb588a1d1e102c857920c4810dc4152f7ebd"
time="2023-02-09T13:19:38.883576000+03:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2023-02-09T13:19:44.111321000+03:00" level=warning msg="Failed getting network for ep e4b994e3382d4ac15a94580b19e00040d62da2c1fe14888716131191845a425b during sandbox 9b539f767ae0793949bf6214fca5bb588a1d1e102c857920c4810dc4152f7ebd delete: network dc97692c1acf66b31c007cb3c5128273af4cc4d8941ab72bdf5e48daef0ca498 not found"
time="2023-02-09T13:19:44.112926000+03:00" level=warning msg="Failed getting network for ep 441892459cff0e33162245531f2cc92c34c62d5d9636f21cfd842319a89818c4 during sandbox 9b539f767ae0793949bf6214fca5bb588a1d1e102c857920c4810dc4152f7ebd delete: network 0c5085b1b048cd809cab3f7c930d0a3d6d3204be6f50f6dd97831936597114dd not found"
time="2023-02-09T13:19:44.112987000+03:00" level=warning msg="Failed getting network for ep 4c5b7863bbcb7eb73d79eb1330005137c2f845ef77b3670a255965ee9c631145 during sandbox 635b527878465b8a88ee95a417554431473284205dcb2874d991a8cd20712e23 delete: network dc97692c1acf66b31c007cb3c5128273af4cc4d8941ab72bdf5e48daef0ca498 not found"
time="2023-02-09T13:19:44.113090000+03:00" level=error msg="failed to cleanup up stale network sandbox for container 8840fc4cf353614867786a8e907363085684e5245a02451d62a145d798d8382b"
time="2023-02-09T13:19:44.113131000+03:00" level=warning msg="Failed getting network for ep 950fa6c773aa9c5de8d9108e237573e886a3d20b9f485e69162282e8b6d65b13 during sandbox 635b527878465b8a88ee95a417554431473284205dcb2874d991a8cd20712e23 delete: network 4ba1ee99bb14ac5b8af74fa4a930bca0ae2640f72dac2e61c6a0e25ad5e0fb8c not found"
time="2023-02-09T13:19:44.754239000+03:00" level=warning msg="Failed getting network for ep e3c919fdf89c328b5d782890593d640e4eaf0b8f77eaf407b19963c0f6fc8f0f during sandbox 635b527878465b8a88ee95a417554431473284205dcb2874d991a8cd20712e23 delete: network 19ef5e0dc49f99a0e5943e274e53ce599d1815e30face41ca9ff8e0c10fdab5a not found"
time="2023-02-09T13:19:44.754420000+03:00" level=warning msg="Failed getting network for ep 42ab6750127879852b959586e6c338e14e4002516549860fab1ce78ad80f7aa4 during sandbox 635b527878465b8a88ee95a417554431473284205dcb2874d991a8cd20712e23 delete: network 733fd239aea303ad2fe620872c0154e80dc57efb0480d5b28c5ece6274bbedc4 not found"
time="2023-02-09T13:19:44.754503000+03:00" level=error msg="failed to cleanup up stale network sandbox for container f6f3bd530c83ea610cfaf6417d314a11bc9bcd68e31e68028a54d4437993897c"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xab97e8]
goroutine 172 [running]:
panic(0x17fef00, 0x4420012070)
/usr/lib/go-1.7/src/runtime/panic.go:500 +0x390
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*endpoint).addServiceInfoToCluster(0x44200774a0, 0x44208d0960, 0x0, 0x0)
/code/docker/docker/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:587 +0x88
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*sandbox).EnableService(0x44208d0960, 0x0, 0x0)
/code/docker/docker/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox.go:677 +0x144
github.com/docker/docker/daemon.(*Daemon).ActivateContainerServiceBinding(0x4420338400, 0x44202118d0, 0xb, 0x0, 0x0)
/code/docker/docker/.gopath/src/github.com/docker/docker/daemon/container_operations.go:1070 +0x13c
github.com/docker/docker/daemon.(*Daemon).connectToNetwork(0x4420338400, 0x44202946c0, 0x1a48819, 0x6, 0x442030f800, 0x0, 0x0, 0x0)
/code/docker/docker/.gopath/src/github.com/docker/docker/daemon/container_operations.go:782 +0xa48
github.com/docker/docker/daemon.(*Daemon).allocateNetwork(0x4420338400, 0x44202946c0, 0x0, 0x0)
/code/docker/docker/.gopath/src/github.com/docker/docker/daemon/container_operations.go:526 +0x350
github.com/docker/docker/daemon.(*Daemon).initializeNetworking(0x4420338400, 0x44202946c0, 0x0, 0x0)
/code/docker/docker/.gopath/src/github.com/docker/docker/daemon/container_operations.go:903 +0x370
github.com/docker/docker/daemon.(*Daemon).containerStart(0x4420338400, 0x44202946c0, 0x0, 0x0, 0x0, 0x0, 0x2020202020202001, 0x0, 0x0)
/code/docker/docker/.gopath/src/github.com/docker/docker/daemon/start.go:140 +0x1b4
github.com/docker/docker/daemon.(*Daemon).restore.func2(0x44208d26f0, 0x4420338400, 0x44202a1a70, 0x44202946c0, 0x442005d020)
/code/docker/docker/.gopath/src/github.com/docker/docker/daemon/daemon.go:323 +0x274
created by github.com/docker/docker/daemon.(*Daemon).restore
/code/docker/docker/.gopath/src/github.com/docker/docker/daemon/daemon.go:327 +0xd54
I was fuzzing about configurating Docker Swarm overlay network to electing multiple managers and this happened.
My main manager now ends up now in a deadloop because of this.
I only had a very little hand in Golang, and I don't know what to do.