Open mitar opened 7 years ago
Fixed with https://github.com/tozd/salt/commit/4594ecab327857204ab89a8867cf4febb1579a5b.
I redeployed containers on server3 with the new restart policy.
I still had to restart it manually
After power outage, docker ps
hangs. Output of cat syslog | grep docker
:
Sep 29 10:46:06 server3 kernel: [ 220.837419] audit: type=1400 audit(1569779166.039:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=975 comm="apparmor_parser"
Sep 29 11:14:20 server3 dockerd[3322]: time="2019-09-29T11:14:20.645348694-07:00" level=info msg="libcontainerd: new containerd process, pid: 3329"
Sep 29 11:14:22 server3 kernel: [ 1916.701963] audit: type=1400 audit(1569780862.150:9): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="docker-default" pid=3342 comm="apparmor_parser"
Sep 29 11:15:34 server3 dockerd[3322]: time="2019-09-29T11:15:34.309096926-07:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Sep 29 11:15:34 server3 dockerd[3322]: time="2019-09-29T11:15:34.309491833-07:00" level=warning msg="Your kernel does not support swap memory limit."
Sep 29 11:15:34 server3 dockerd[3322]: time="2019-09-29T11:15:34.310262644-07:00" level=info msg="Loading containers: start."
Sep 29 11:15:35 server3 dockerd[3322]: ............................time="2019-09-29T11:15:35.240805555-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (96491bfe0c9cff2454d2565f7bfd99159fa2720fb417004cbc49f13f2e498a0e)."
Sep 29 11:15:35 server3 dockerd[3322]: time="2019-09-29T11:15:35.241460676-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers 96491bfe0c9cff2454d2565f7bfd99159fa2720fb417004cbc49f13f2e498a0e/shm: invalid argument"
Sep 29 11:17:10 server3 dockerd[3322]: time="2019-09-29T11:17:10.811812666-07:00" level=info msg="Processing signal 'terminated'"
Sep 29 11:17:20 server3 dockerd[3322]: time="2019-09-29T11:17:20.954444523-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (d4c0a7c04d8e50283ab3555eb534f974f1fd36bfa10c922de151adbc2f05db52)."
Sep 29 11:17:20 server3 dockerd[3322]: time="2019-09-29T11:17:20.955368704-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/d4c0a7c04d8e50283ab3555eb534f974f1fd36bfa10c922de151adbc2f05db52/shm: invalid argument"
Sep 29 11:17:21 server3 dockerd[3322]: time="2019-09-29T11:17:21.299657097-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (68d7d76d552a8974b366730f97beaa752b26fdeeec5e197bf7392efa5c16f853)."
Sep 29 11:17:21 server3 dockerd[3322]: time="2019-09-29T11:17:21.300564525-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/68d7d76d552a8974b366730f97beaa752b26fdeeec5e197bf7392efa5c16f853/shm: invalid argument"
Sep 29 11:17:21 server3 dockerd[3322]: time="2019-09-29T11:17:21.489469578-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (2f3bad9727ab14654dea1247ff1b0a236dd83e423816805123712a05b5c4884b)."
Sep 29 11:17:21 server3 dockerd[3322]: time="2019-09-29T11:17:21.490086369-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/2f3bad9727ab14654dea1247ff1b0a236dd83e423816805123712a05b5c4884b/shm: invalid argument"
Sep 29 11:17:21 server3 dockerd[3322]: time="2019-09-29T11:17:21.657160860-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (c774112b6544330babdb58d5a445cd1fd103776defcbde338b3e973c145ee60b)."
Sep 29 11:17:21 server3 dockerd[3322]: time="2019-09-29T11:17:21.657908245-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/c774112b6544330babdb58d5a445cd1fd103776defcbde338b3e973c145ee60b/shm: invalid argument"
Sep 29 11:17:22 server3 dockerd[3322]: time="2019-09-29T11:17:22.402786336-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (df188824b0f7323a80b97b7bebeeca757fdbe3836f963029fae086d902d5da21)."
Sep 29 11:17:22 server3 dockerd[3322]: time="2019-09-29T11:17:22.403641552-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/df188824b0f7323a80b97b7bebeeca757fdbe3836f963029fae086d902d5da21/shm: invalid argument"
Sep 29 11:17:23 server3 dockerd[3322]: time="2019-09-29T11:17:23.148499532-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (888f4ad8e6fe5747d1b051a2bbb3e941680c6160fba4eb4722c2901cc891a671)."
Sep 29 11:17:23 server3 dockerd[3322]: time="2019-09-29T11:17:23.153025606-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/888f4ad8e6fe5747d1b051a2bbb3e941680c6160fba4eb4722c2901cc891a671/shm: invalid argument"
Sep 29 11:17:23 server3 dockerd[3322]: time="2019-09-29T11:17:23.326681631-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (635df72c69c2a94fba6fc41c08b48e08b46be8c515350d410089199d2d871404)."
Sep 29 11:17:23 server3 dockerd[3322]: time="2019-09-29T11:17:23.327626754-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/635df72c69c2a94fba6fc41c08b48e08b46be8c515350d410089199d2d871404/shm: invalid argument"
Sep 29 11:17:23 server3 dockerd[3322]: time="2019-09-29T11:17:23.594646547-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:23 server3 kernel: [ 2098.306289] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
Sep 29 11:17:24 server3 dockerd[3322]: time="2019-09-29T11:17:24.138296872-07:00" level=info msg="Removing stale sandbox b529a7fb5226c7126df9b7cfc4887eafb8bf9f11dcca6e4885ff4d3dc05a2617 (888f4ad8e6fe5747d1b051a2bbb3e941680c6160fba4eb4722c2901cc891a671)"
Sep 29 11:17:25 server3 dockerd[3322]: time="2019-09-29T11:17:25.228154698-07:00" level=info msg="Removing stale sandbox d659e1885345f922af8f2b2daba1fc08d6c2fe8f640a2cdd07f37befbef6700c (df188824b0f7323a80b97b7bebeeca757fdbe3836f963029fae086d902d5da21)"
Sep 29 11:17:25 server3 dockerd[3322]: time="2019-09-29T11:17:25.783207780-07:00" level=info msg="Removing stale sandbox 0d494be82acd7c90411b60f3d8977889b2284ca1a821004ded833bc3da13f73b (635df72c69c2a94fba6fc41c08b48e08b46be8c515350d410089199d2d871404)"
Sep 29 11:17:26 server3 dockerd[3322]: time="2019-09-29T11:17:26.170928661-07:00" level=info msg="Removing stale sandbox 1e5c49cadd7838950e953a9bb1d9fd6c2aad68aa96977775c3cfb5b4ce0046eb (2f3bad9727ab14654dea1247ff1b0a236dd83e423816805123712a05b5c4884b)"
Sep 29 11:17:26 server3 dockerd[3322]: time="2019-09-29T11:17:26.531080976-07:00" level=info msg="Removing stale sandbox 454dc89aa9e94352d3359d9fe3a62807883d38b1666fc23272bdbbb65e4ffe8b (d4c0a7c04d8e50283ab3555eb534f974f1fd36bfa10c922de151adbc2f05db52)"
Sep 29 11:17:26 server3 dockerd[3322]: time="2019-09-29T11:17:26.869025267-07:00" level=info msg="Removing stale sandbox 584b842cd64271c9b079943d638f64300c1a8dd72b13babd89edf2d944dea808 (96491bfe0c9cff2454d2565f7bfd99159fa2720fb417004cbc49f13f2e498a0e)"
Sep 29 11:17:27 server3 dockerd[3322]: time="2019-09-29T11:17:27.170912300-07:00" level=info msg="Removing stale sandbox 84146f3c236d6c8880a707db9e924d9c7e092c554250f9c4d7ec698ab9b84d1e (c774112b6544330babdb58d5a445cd1fd103776defcbde338b3e973c145ee60b)"
Sep 29 11:17:27 server3 dockerd[3322]: time="2019-09-29T11:17:27.647156587-07:00" level=info msg="Removing stale sandbox 8f84fe1dfcca7b8cbc94a2fe652b3fddf0cc9ea1b937588cb9feecf4b853a94c (68d7d76d552a8974b366730f97beaa752b26fdeeec5e197bf7392efa5c16f853)"
Sep 29 11:17:27 server3 dockerd[3322]: time="2019-09-29T11:17:27.988341341-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Sep 29 11:17:36 server3 dockerd[3322]: time="2019-09-29T11:17:36-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:36 server3 dockerd[3322]: time="2019-09-29T11:17:36-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:37 server3 dockerd[3322]: time="2019-09-29T11:17:37-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:42 server3 dockerd[3322]: time="2019-09-29T11:17:42-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:42 server3 dockerd[3322]: time="2019-09-29T11:17:42-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:48 server3 dockerd[3322]: time="2019-09-29T11:17:48-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:55 server3 dockerd[3322]: time="2019-09-29T11:17:55.972804177-07:00" level=info msg="Loading containers: done."
Sep 29 11:17:56 server3 dockerd[3322]: time="2019-09-29T11:17:56.031355165-07:00" level=info msg="Daemon has completed initialization"
Sep 29 11:17:56 server3 dockerd[3322]: time="2019-09-29T11:17:56.031415687-07:00" level=info msg="Docker daemon" commit=7392c3b graphdriver=overlay2 version=1.12.5
Sep 29 11:17:56 server3 dockerd[3322]: time="2019-09-29T11:17:56.045097929-07:00" level=info msg="API listen on /var/run/docker.sock"
Running service docker restart
fixed it, after a delay.
This time it did eventually start, but something is wrong with rocketchat's dockergen setup so it's not starting automatically.
It seems the issue is that we have restart policy set to
unless-stopped
, which is not supported on Docker < 1.9. We have currently Docker 1.8.3 on server3.