search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.91k stars 1.95k forks source link

"no such image" - unrecoverable error #2154

Closed kak-tus closed 7 years ago

kak-tus commented 7 years ago

Nomad version

0.5.2

Issue

There were many fixes from 0.4 version to fix container start. Now, in 0.5.2 I have same issue. Sometimes, very rarely (may be docker can't download image) and nomad mark this error as non recoverable.

Nomad Client logs (if appropriate)

01/04/17 17:12:00 MSK  Not Restarting  Error was unrecoverable
01/04/17 17:11:59 MSK  Driver Failure  failed to start task 'daemon' for alloc 'c5b52fc0-7b04-f24c-1dfe-e92f798daf85': Failed to create container: no such image
dadgar commented 7 years ago

@kak-tus Do you have the logs from the clients (hopefully debug)?

kak-tus commented 7 years ago

@dadgar From client - no. I have logs (usual logs, not debug) from server for nomad and docker.

kak-tus commented 7 years ago

Oh, I understand. Problem is not on the client side. Sorry for incomplete description. Job was started and worked for a long time. But by some reasons (vault token revoke/create) nomad were tried to restart job. And new job allocation was started without success.

dadgar commented 7 years ago

Could you paste the logs you have and if. you have more of the event output that would be very useful

kak-tus commented 7 years ago

Sorry, here is some a little bit more logs, than needed. But it shows a problem - some containers were restarted successful, and one of them - not.

nomad

2017-01-04 17:11:27 c1 [nomad 16723]      2017/01/04 17:11:27.278709 [ERR] client: failed to update allocations: rpc error: EOF
2017-01-04 17:11:27 c1 [nomad 16723]      2017/01/04 17:11:27.278753 [ERR] worker: failed to dequeue evaluation: rpc error: EOF
2017-01-04 17:11:27 c1 [nomad 16723]      2017/01/04 17:11:27.278767 [ERR] worker: failed to dequeue evaluation: rpc error: EOF
2017-01-04 17:11:27 c1 [nomad 16723]  client: failed to update allocations: rpc error: EOF
2017-01-04 17:11:27 c1 [nomad 16723]  worker: failed to dequeue evaluation: rpc error: EOF
2017-01-04 17:11:27 c1 [nomad 16723]  worker: failed to dequeue evaluation: rpc error: EOF
2017-01-04 17:11:27 c1 [nomad 16723]      2017/01/04 17:11:27.292811 [ERR] http: Request /v1/status/peers, error: rpc error: stream closed
2017-01-04 17:11:27 c1 [nomad 16723]  http: Request /v1/status/peers, error: rpc error: stream closed
2017-01-04 17:11:27 c1 [nomad 16723]      2017/01/04 17:11:27.983399 [INFO] driver.docker: stopped container 76d5716471de9276243efa6789a2ac7447fadc2c438d0c071c93173097176b2c
2017-01-04 17:11:27 c1 [nomad 16723]  driver.docker: stopped container 76d5716471de9276243efa6789a2ac7447fadc2c438d0c071c93173097176b2c
2017-01-04 17:11:28 c1 [nomad 16723]      2017/01/04 17:11:28.005848 [INFO] driver.docker: stopped container 55d2ba767d44a673245102313a969b749214cb06c59266412afe08c94a29db79
2017-01-04 17:11:28 c1 [nomad 16723]  driver.docker: stopped container 55d2ba767d44a673245102313a969b749214cb06c59266412afe08c94a29db79
2017-01-04 17:11:28 c1 [nomad 16723]      2017/01/04 17:11:28.228609 [INFO] driver.docker: stopped container 4a3e3ba91583362bb6d5e5982e1aeb4aeb5a8c5ad266b81b547d1d9a3e1d2242
2017-01-04 17:11:28 c1 [nomad 16723]  driver.docker: stopped container 4a3e3ba91583362bb6d5e5982e1aeb4aeb5a8c5ad266b81b547d1d9a3e1d2242
2017-01-04 17:11:32 c1 [nomad 16723]      2017/01/04 17:11:32.768420 [INFO] driver.docker: stopped container 49386194222ad4579372f5326a6fbb0ad0f9c443c4845ebe335812bd877e4a2b
2017-01-04 17:11:32 c1 [nomad 16723]  driver.docker: stopped container 49386194222ad4579372f5326a6fbb0ad0f9c443c4845ebe335812bd877e4a2b
2017-01-04 17:11:33 c1 [nomad 16723]      2017/01/04 17:11:33.121078 [INFO] driver.docker: stopped container 2a0f9205d093dc3185c12d4749641b5a11d31988a2793d19c97572fe3b8c9b4e
2017-01-04 17:11:33 c1 [nomad 16723]  driver.docker: stopped container 2a0f9205d093dc3185c12d4749641b5a11d31988a2793d19c97572fe3b8c9b4e
2017-01-04 17:11:33 c1 [nomad 16723]      2017/01/04 17:11:33.305245 [INFO] driver.docker: created container 620cd3ff956f5b7b9d5e83488025364aaab8724d31847fd3eb31c9a397633396
2017-01-04 17:11:33 c1 [nomad 16723]  driver.docker: created container 620cd3ff956f5b7b9d5e83488025364aaab8724d31847fd3eb31c9a397633396
2017-01-04 17:11:33 c1 [nomad 16723]      2017/01/04 17:11:33.307596 [INFO] driver.docker: stopped container 9cc5bbae2b44ee43ac6d457567513c11d605004aea911d7ca1f0e3bfa2ba010d
2017-01-04 17:11:33 c1 [nomad 16723]  driver.docker: stopped container 9cc5bbae2b44ee43ac6d457567513c11d605004aea911d7ca1f0e3bfa2ba010d
2017-01-04 17:11:33 c1 [nomad 16723]      2017/01/04 17:11:33.370003 [INFO] driver.docker: created container a75e9422d89bbc690a46060ada9c5a2868f087f63a342b2ace9d38d61d744a49
2017-01-04 17:11:33 c1 [nomad 16723]  driver.docker: created container a75e9422d89bbc690a46060ada9c5a2868f087f63a342b2ace9d38d61d744a49
2017-01-04 17:11:33 c1 [nomad 16723]      2017/01/04 17:11:33.441488 [INFO] driver.docker: stopped container 9d40c30a72b70816f02638f8c36d017bc6cde0f323473a727d3de4ba0777ac5a
2017-01-04 17:11:33 c1 [nomad 16723]  driver.docker: stopped container 9d40c30a72b70816f02638f8c36d017bc6cde0f323473a727d3de4ba0777ac5a
2017-01-04 17:11:33 c1 [nomad 16723]      2017/01/04 17:11:33.519906 [INFO] driver.docker: stopped container 11a6ab590236b02f006ed400719a924fef0d106cd6a0253e195a35ba72e7fc53
2017-01-04 17:11:33 c1 [nomad 16723]  driver.docker: stopped container 11a6ab590236b02f006ed400719a924fef0d106cd6a0253e195a35ba72e7fc53
2017-01-04 17:11:33 c1 [nomad 16723]      2017/01/04 17:11:33.592626 [INFO] driver.docker: stopped container 42a34404113bd702c9c166df6a549c038828d6969217b52225b21d8fc4b2bc7b
2017-01-04 17:11:33 c1 [nomad 16723]  driver.docker: stopped container 42a34404113bd702c9c166df6a549c038828d6969217b52225b21d8fc4b2bc7b
2017-01-04 17:11:33 c1 [nomad 16723]      2017/01/04 17:11:33.633735 [INFO] driver.docker: stopped container 6d87d7190be8184bafcc29d1b912163e6f5fe37bf3314b7161733efe02ffc640
2017-01-04 17:11:33 c1 [nomad 16723]  driver.docker: stopped container 6d87d7190be8184bafcc29d1b912163e6f5fe37bf3314b7161733efe02ffc640
2017-01-04 17:11:33 c1 [nomad 16723]      2017/01/04 17:11:33.709354 [INFO] driver.docker: created container 8eae6fd84e9c9279cd8177997fbb16494373a3656f865d02b9e9e6c0b4fd264f
2017-01-04 17:11:33 c1 [nomad 16723]  driver.docker: created container 8eae6fd84e9c9279cd8177997fbb16494373a3656f865d02b9e9e6c0b4fd264f
2017-01-04 17:11:33 c1 [nomad 16723]      2017/01/04 17:11:33.911732 [INFO] driver.docker: created container 8bc6beee5c2eef034df8c7847d44c55f62419956baff0c617fa71cbd4d8b7835
2017-01-04 17:11:33 c1 [nomad 16723]  driver.docker: created container 8bc6beee5c2eef034df8c7847d44c55f62419956baff0c617fa71cbd4d8b7835
2017-01-04 17:11:33 c1 [nomad 16723]      2017/01/04 17:11:33.924781 [INFO] driver.docker: started container 620cd3ff956f5b7b9d5e83488025364aaab8724d31847fd3eb31c9a397633396
2017-01-04 17:11:33 c1 [nomad 16723]  driver.docker: started container 620cd3ff956f5b7b9d5e83488025364aaab8724d31847fd3eb31c9a397633396
2017-01-04 17:11:33 c1 [nomad 16723]      2017/01/04 17:11:33.927465 [WARN] client: error fetching stats of task nginx: stats collection hasn't started yet
2017-01-04 17:11:33 c1 [nomad 16723]  client: error fetching stats of task nginx: stats collection hasn't started yet
2017-01-04 17:11:34 c1 [nomad 16723]      2017/01/04 17:11:34.091675 [INFO] driver.docker: created container 65069ed3401bb431246e6ba671c16e257fcc004450daf05daadecebead32dc52
2017-01-04 17:11:34 c1 [nomad 16723]  driver.docker: created container 65069ed3401bb431246e6ba671c16e257fcc004450daf05daadecebead32dc52
2017-01-04 17:11:34 c1 [nomad 16723]      2017/01/04 17:11:34.293763 [INFO] driver.docker: created container 0629bdbaf4ba0b5255ec7e1afb4458592fbf9a409604372ae2b318c433064f66
2017-01-04 17:11:34 c1 [nomad 16723]  driver.docker: created container 0629bdbaf4ba0b5255ec7e1afb4458592fbf9a409604372ae2b318c433064f66
2017-01-04 17:11:34 c1 [nomad 16723]      2017/01/04 17:11:34.334449 [INFO] driver.docker: created container 35c583a7027eec590a82ca6b7e4854583dd5a7a1c13edc364970f5b65feaf61f
2017-01-04 17:11:34 c1 [nomad 16723]  driver.docker: created container 35c583a7027eec590a82ca6b7e4854583dd5a7a1c13edc364970f5b65feaf61f
2017-01-04 17:11:34 c1 [nomad 16723]      2017/01/04 17:11:34.992298 [INFO] driver.docker: started container 8bc6beee5c2eef034df8c7847d44c55f62419956baff0c617fa71cbd4d8b7835
2017-01-04 17:11:34 c1 [nomad 16723]  driver.docker: started container 8bc6beee5c2eef034df8c7847d44c55f62419956baff0c617fa71cbd4d8b7835
2017-01-04 17:11:34 c1 [nomad 16723]      2017/01/04 17:11:34.994300 [WARN] client: error fetching stats of task logs: stats collection hasn't started yet
2017-01-04 17:11:34 c1 [nomad 16723]  client: error fetching stats of task logs: stats collection hasn't started yet
2017-01-04 17:11:35 c1 [nomad 16723]      2017/01/04 17:11:35.082248 [INFO] driver.docker: started container 0629bdbaf4ba0b5255ec7e1afb4458592fbf9a409604372ae2b318c433064f66
2017-01-04 17:11:35 c1 [nomad 16723]  driver.docker: started container 0629bdbaf4ba0b5255ec7e1afb4458592fbf9a409604372ae2b318c433064f66
2017-01-04 17:11:35 c1 [nomad 16723]      2017/01/04 17:11:35.084684 [WARN] client: error fetching stats of task nginx: stats collection hasn't started yet
2017-01-04 17:11:35 c1 [nomad 16723]  client: error fetching stats of task nginx: stats collection hasn't started yet
2017-01-04 17:11:35 c1 [nomad 16723]      2017/01/04 17:11:35.949441 [INFO] driver.docker: started container a75e9422d89bbc690a46060ada9c5a2868f087f63a342b2ace9d38d61d744a49
2017-01-04 17:11:35 c1 [nomad 16723]  driver.docker: started container a75e9422d89bbc690a46060ada9c5a2868f087f63a342b2ace9d38d61d744a49
2017-01-04 17:11:35 c1 [nomad 16723]      2017/01/04 17:11:35.952766 [WARN] client: error fetching stats of task syncthing: stats collection hasn't started yet
2017-01-04 17:11:35 c1 [nomad 16723]  client: error fetching stats of task syncthing: stats collection hasn't started yet
2017-01-04 17:11:35 c1 [nomad 16723]      2017/01/04 17:11:35.994935 [WARN] client: error fetching stats of task logs: stats collection hasn't started yet
2017-01-04 17:11:35 c1 [nomad 16723]  client: error fetching stats of task logs: stats collection hasn't started yet
2017-01-04 17:11:36 c1 [nomad 16723]      2017/01/04 17:11:36.082120 [INFO] driver.docker: started container 8eae6fd84e9c9279cd8177997fbb16494373a3656f865d02b9e9e6c0b4fd264f
2017-01-04 17:11:36 c1 [nomad 16723]  driver.docker: started container 8eae6fd84e9c9279cd8177997fbb16494373a3656f865d02b9e9e6c0b4fd264f
2017-01-04 17:11:36 c1 [nomad 16723]      2017/01/04 17:11:36.086048 [WARN] client: error fetching stats of task mta: stats collection hasn't started yet
2017-01-04 17:11:36 c1 [nomad 16723]  client: error fetching stats of task mta: stats collection hasn't started yet
2017-01-04 17:11:36 c1 [nomad 16723]      2017/01/04 17:11:36.302370 [INFO] driver.docker: started container 65069ed3401bb431246e6ba671c16e257fcc004450daf05daadecebead32dc52
2017-01-04 17:11:36 c1 [nomad 16723]  driver.docker: started container 65069ed3401bb431246e6ba671c16e257fcc004450daf05daadecebead32dc52
2017-01-04 17:11:36 c1 [nomad 16723]      2017/01/04 17:11:36.303647 [WARN] client: error fetching stats of task cron: stats collection hasn't started yet
2017-01-04 17:11:36 c1 [nomad 16723]  client: error fetching stats of task cron: stats collection hasn't started yet
2017-01-04 17:11:36 c1 [nomad 16723]      2017/01/04 17:11:36.385427 [INFO] driver.docker: started container 35c583a7027eec590a82ca6b7e4854583dd5a7a1c13edc364970f5b65feaf61f
2017-01-04 17:11:36 c1 [nomad 16723]  driver.docker: started container 35c583a7027eec590a82ca6b7e4854583dd5a7a1c13edc364970f5b65feaf61f
2017-01-04 17:11:36 c1 [nomad 16723]      2017/01/04 17:11:36.388632 [WARN] client: error fetching stats of task daemon: stats collection hasn't started yet
2017-01-04 17:11:36 c1 [nomad 16723]  client: error fetching stats of task daemon: stats collection hasn't started yet
2017-01-04 17:11:37 c1 [nomad 16723]      2017/01/04 17:11:37.592769 [INFO] driver.docker: created container e3c434aff1dd624bab47272a53a77f390472f61c690a57588c368b9a2df30f70
2017-01-04 17:11:37 c1 [nomad 16723]  driver.docker: created container e3c434aff1dd624bab47272a53a77f390472f61c690a57588c368b9a2df30f70
2017-01-04 17:11:38 c1 [nomad 16723]      2017/01/04 17:11:38 [ERR] raft-net: Failed to decode incoming command: read tcp 185.75.47.37:4647->185.12.94.137:60946: read: connection reset by peer
2017-01-04 17:11:38 c1 [nomad 16723]  raft-net: Failed to decode incoming command: read tcp 185.75.47.37:4647->185.12.94.137:60946: read: connection reset by peer
2017-01-04 17:11:39 c1 [nomad 16723]      2017/01/04 17:11:39.470744 [INFO] driver.docker: started container e3c434aff1dd624bab47272a53a77f390472f61c690a57588c368b9a2df30f70
2017-01-04 17:11:39 c1 [nomad 16723]  driver.docker: started container e3c434aff1dd624bab47272a53a77f390472f61c690a57588c368b9a2df30f70
2017-01-04 17:11:39 c1 [nomad 16723]      2017/01/04 17:11:39.473256 [WARN] client: error fetching stats of task redis: stats collection hasn't started yet
2017-01-04 17:11:39 c1 [nomad 16723]  client: error fetching stats of task redis: stats collection hasn't started yet
2017-01-04 17:11:42 c1 [nomad 16723]      2017/01/04 17:11:42.014772 [ERR] driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:11:42 c1 [nomad 16723]      2017/01/04 17:11:42.014845 [INFO] client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:11:42 c1 [nomad 16723]      2017/01/04 17:11:42.015130 [INFO] client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 17.497664836s
2017-01-04 17:11:42 c1 [nomad 16723]  driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:11:42 c1 [nomad 16723]  client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:11:42 c1 [nomad 16723]  client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 17.497664836s
2017-01-04 17:11:46 c1 [nomad 16723]      2017/01/04 17:11:46.352004 [INFO] driver.docker: created container ba4fbaae4ad3c50b229ed3097361de5e91a8f2bf9f0bdc3c1187b403207c8008
2017-01-04 17:11:46 c1 [nomad 16723]  driver.docker: created container ba4fbaae4ad3c50b229ed3097361de5e91a8f2bf9f0bdc3c1187b403207c8008
2017-01-04 17:11:46 c1 [nomad 16723]      2017/01/04 17:11:46.787217 [INFO] driver.docker: started container ba4fbaae4ad3c50b229ed3097361de5e91a8f2bf9f0bdc3c1187b403207c8008
2017-01-04 17:11:46 c1 [nomad 16723]  driver.docker: started container ba4fbaae4ad3c50b229ed3097361de5e91a8f2bf9f0bdc3c1187b403207c8008
2017-01-04 17:11:46 c1 [nomad 16723]      2017/01/04 17:11:46.789204 [WARN] client: error fetching stats of task frontend: stats collection hasn't started yet
2017-01-04 17:11:46 c1 [nomad 16723]  client: error fetching stats of task frontend: stats collection hasn't started yet
2017-01-04 17:11:58 c1 [nomad 16723]      2017/01/04 17:11:58.573390 [INFO] driver.docker: stopped container ffc48fe2f9924d7ae93e4a4a2e7e44181fd272357a83c243e7bbb127410aef70
2017-01-04 17:11:58 c1 [nomad 16723]  driver.docker: stopped container ffc48fe2f9924d7ae93e4a4a2e7e44181fd272357a83c243e7bbb127410aef70
2017-01-04 17:11:58 c1 [nomad 16723]      2017/01/04 17:11:58.801303 [INFO] driver.docker: stopped container 511e9603b1b3ae1f0be6166e4d52f10899d605bd7e13262e2896b4c31e01b119
2017-01-04 17:11:58 c1 [nomad 16723]  driver.docker: stopped container 511e9603b1b3ae1f0be6166e4d52f10899d605bd7e13262e2896b4c31e01b119
2017-01-04 17:11:58 c1 [nomad 16723]  2017/01/04 17:11:58 [ERR] yamux: Failed to write header: write unix @->/tmp/plugin728464039: write: broken pipe
2017-01-04 17:11:58 c1 [nomad 16723]  ==> Caught signal: broken pipe
2017-01-04 17:11:58 c1 [nomad 16723]      2017/01/04 17:11:58.815372 [INFO] driver.docker: stopped container 3c86a0db96bf247eeaad3e8d32caf727b7434a7e1da0a46f8bec6a5a5024fb12
2017-01-04 17:11:58 c1 [nomad 16723]  driver.docker: stopped container 3c86a0db96bf247eeaad3e8d32caf727b7434a7e1da0a46f8bec6a5a5024fb12
2017-01-04 17:11:58 c1 [nomad 16723]      2017/01/04 17:11:58.889959 [INFO] driver.docker: stopped container 782183b6d4546d029afca992738f1201fe7addafb33cde4713c5d1bb78ad9e3e
2017-01-04 17:11:58 c1 [nomad 16723]  driver.docker: stopped container 782183b6d4546d029afca992738f1201fe7addafb33cde4713c5d1bb78ad9e3e
2017-01-04 17:11:59 c1 [nomad 16723]      2017/01/04 17:11:59.029875 [INFO] driver.docker: stopped container 06805b46389a8f61e7430e6ffcf1741ec3bef04630c2ae03aaf27b5296973152
2017-01-04 17:11:59 c1 [nomad 16723]  driver.docker: stopped container 06805b46389a8f61e7430e6ffcf1741ec3bef04630c2ae03aaf27b5296973152
2017-01-04 17:11:59 c1 [nomad 16723]      2017/01/04 17:11:59.117736 [INFO] driver.docker: stopped container e3e67774e9cb087c8085a2cbc4d32d41f676e385fb275372217ab0d6cf898c7b
2017-01-04 17:11:59 c1 [nomad 16723]  driver.docker: stopped container e3e67774e9cb087c8085a2cbc4d32d41f676e385fb275372217ab0d6cf898c7b
2017-01-04 17:11:59 c1 [nomad 16723]      2017/01/04 17:11:59.184857 [ERR] driver.docker: failed to create container: no such image
2017-01-04 17:11:59 c1 [nomad 16723]  driver.docker: failed to create container: no such image
2017-01-04 17:11:59 c1 [nomad 16723]      2017/01/04 17:11:59.207695 [INFO] client: failed to start task 'cron' for alloc 'c5b52fc0-7b04-f24c-1dfe-e92f798daf85': Failed to create container: no such image
2017-01-04 17:11:59 c1 [nomad 16723]  client: failed to start task 'cron' for alloc 'c5b52fc0-7b04-f24c-1dfe-e92f798daf85': Failed to create container: no such image
2017-01-04 17:11:59 c1 [nomad 16723]      2017/01/04 17:11:59.215108 [INFO] client: Not restarting task: cron for alloc: c5b52fc0-7b04-f24c-1dfe-e92f798daf85
2017-01-04 17:11:59 c1 [nomad 16723]  client: Not restarting task: cron for alloc: c5b52fc0-7b04-f24c-1dfe-e92f798daf85
2017-01-04 17:11:59 c1 [nomad 16723]      2017/01/04 17:11:59.980965 [ERR] driver.docker: failed to create container: no such image
2017-01-04 17:11:59 c1 [nomad 16723]      2017/01/04 17:11:59.981094 [ERR] driver.docker: failed to create container: no such image
2017-01-04 17:11:59 c1 [nomad 16723]  driver.docker: failed to create container: no such image
2017-01-04 17:11:59 c1 [nomad 16723]  driver.docker: failed to create container: no such image
2017-01-04 17:11:59 c1 [nomad 16723]      2017/01/04 17:11:59.983053 [INFO] client: failed to start task 'minion' for alloc 'c5b52fc0-7b04-f24c-1dfe-e92f798daf85': Failed to create container: no such image
2017-01-04 17:11:59 c1 [nomad 16723]  client: failed to start task 'minion' for alloc 'c5b52fc0-7b04-f24c-1dfe-e92f798daf85': Failed to create container: no such image
2017-01-04 17:11:59 c1 [nomad 16723]      2017/01/04 17:11:59.984749 [INFO] client: Not restarting task: minion for alloc: c5b52fc0-7b04-f24c-1dfe-e92f798daf85
2017-01-04 17:11:59 c1 [nomad 16723]  client: Not restarting task: minion for alloc: c5b52fc0-7b04-f24c-1dfe-e92f798daf85
2017-01-04 17:11:59 c1 [nomad 16723]      2017/01/04 17:11:59.984841 [INFO] client: failed to start task 'daemon' for alloc 'c5b52fc0-7b04-f24c-1dfe-e92f798daf85': Failed to create container: no such image
2017-01-04 17:11:59 c1 [nomad 16723]  client: failed to start task 'daemon' for alloc 'c5b52fc0-7b04-f24c-1dfe-e92f798daf85': Failed to create container: no such image
2017-01-04 17:12:00 c1 [nomad 16723]      2017/01/04 17:11:59.998533 [INFO] client: Not restarting task: daemon for alloc: c5b52fc0-7b04-f24c-1dfe-e92f798daf85
2017-01-04 17:12:00 c1 [nomad 16723]  client: Not restarting task: daemon for alloc: c5b52fc0-7b04-f24c-1dfe-e92f798daf85
2017-01-04 17:12:00 c1 [nomad 16723]      2017/01/04 17:12:00.068030 [INFO] driver.docker: created container cb8bb62f2bfd4044c94baf1f9eec4ea513112c06130b1d6a28131a5c356168cf
2017-01-04 17:12:00 c1 [nomad 16723]  driver.docker: created container cb8bb62f2bfd4044c94baf1f9eec4ea513112c06130b1d6a28131a5c356168cf
2017-01-04 17:12:00 c1 [nomad 16723]      2017/01/04 17:12:00.223553 [INFO] driver.docker: created container a3f022aa2838b655f7a29db382c7925b56e123bb5206ab4febefb3e09b00e40d
2017-01-04 17:12:00 c1 [nomad 16723]  driver.docker: created container a3f022aa2838b655f7a29db382c7925b56e123bb5206ab4febefb3e09b00e40d
2017-01-04 17:12:00 c1 [nomad 16723]      2017/01/04 17:12:00.290308 [INFO] driver.docker: created container 880f4fd375cf344cdf2367e37794db5d1cf1e4c03c6e4050cf9b46d228c16976
2017-01-04 17:12:00 c1 [nomad 16723]  driver.docker: created container 880f4fd375cf344cdf2367e37794db5d1cf1e4c03c6e4050cf9b46d228c16976
2017-01-04 17:12:00 c1 [nomad 16723]      2017/01/04 17:12:00.393296 [ERR] driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:00 c1 [nomad 16723]  driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:00 c1 [nomad 16723]      2017/01/04 17:12:00.393654 [INFO] client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:00 c1 [nomad 16723]      2017/01/04 17:12:00.394010 [INFO] client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 15.815480475s
2017-01-04 17:12:00 c1 [nomad 16723]  client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:00 c1 [nomad 16723]  client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 15.815480475s
2017-01-04 17:12:00 c1 [nomad 16723]      2017/01/04 17:12:00.612686 [INFO] driver.docker: started container cb8bb62f2bfd4044c94baf1f9eec4ea513112c06130b1d6a28131a5c356168cf
2017-01-04 17:12:00 c1 [nomad 16723]  driver.docker: started container cb8bb62f2bfd4044c94baf1f9eec4ea513112c06130b1d6a28131a5c356168cf
2017-01-04 17:12:00 c1 [nomad 16723]      2017/01/04 17:12:00.613811 [WARN] client: error fetching stats of task nginx: stats collection hasn't started yet
2017-01-04 17:12:00 c1 [nomad 16723]  client: error fetching stats of task nginx: stats collection hasn't started yet
2017-01-04 17:12:00 c1 [nomad 16723]      2017/01/04 17:12:00.900195 [INFO] driver.docker: started container 880f4fd375cf344cdf2367e37794db5d1cf1e4c03c6e4050cf9b46d228c16976
2017-01-04 17:12:00 c1 [nomad 16723]  driver.docker: started container 880f4fd375cf344cdf2367e37794db5d1cf1e4c03c6e4050cf9b46d228c16976
2017-01-04 17:12:00 c1 [nomad 16723]      2017/01/04 17:12:00.903715 [WARN] client: error fetching stats of task daemon: stats collection hasn't started yet
2017-01-04 17:12:00 c1 [nomad 16723]  client: error fetching stats of task daemon: stats collection hasn't started yet
2017-01-04 17:12:01 c1 [nomad 16723]      2017/01/04 17:12:01.016442 [INFO] driver.docker: started container a3f022aa2838b655f7a29db382c7925b56e123bb5206ab4febefb3e09b00e40d
2017-01-04 17:12:01 c1 [nomad 16723]  driver.docker: started container a3f022aa2838b655f7a29db382c7925b56e123bb5206ab4febefb3e09b00e40d
2017-01-04 17:12:01 c1 [nomad 16723]      2017/01/04 17:12:01.018441 [WARN] client: error fetching stats of task cron: stats collection hasn't started yet
2017-01-04 17:12:01 c1 [nomad 16723]  client: error fetching stats of task cron: stats collection hasn't started yet
2017-01-04 17:12:16 c1 [nomad 16723]      2017/01/04 17:12:16.515952 [ERR] driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:16 c1 [nomad 16723]  driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:16 c1 [nomad 16723]      2017/01/04 17:12:16.516613 [INFO] client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:16 c1 [nomad 16723]  client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:16 c1 [nomad 16723]      2017/01/04 17:12:16.517813 [INFO] client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 15.913880875s
2017-01-04 17:12:16 c1 [nomad 16723]  client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 15.913880875s
2017-01-04 17:12:32 c1 [nomad 16723]      2017/01/04 17:12:32.720394 [ERR] driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:32 c1 [nomad 16723]      2017/01/04 17:12:32.720491 [INFO] client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:32 c1 [nomad 16723]      2017/01/04 17:12:32.720853 [INFO] client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 18.20959877s
2017-01-04 17:12:32 c1 [nomad 16723]  driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:32 c1 [nomad 16723]  client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:32 c1 [nomad 16723]  client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 18.20959877s
2017-01-04 17:12:51 c1 [nomad 16723]      2017/01/04 17:12:51.153861 [ERR] driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:51 c1 [nomad 16723]      2017/01/04 17:12:51.154028 [INFO] client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:51 c1 [nomad 16723]  driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:51 c1 [nomad 16723]  client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:12:51 c1 [nomad 16723]      2017/01/04 17:12:51.154454 [INFO] client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 15.314454209s
2017-01-04 17:12:51 c1 [nomad 16723]  client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 15.314454209s
2017-01-04 17:13:06 c1 [nomad 16723]      2017/01/04 17:13:06.715419 [ERR] driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:13:06 c1 [nomad 16723]      2017/01/04 17:13:06.715536 [INFO] client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:13:06 c1 [nomad 16723]  driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:13:06 c1 [nomad 16723]  client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:13:06 c1 [nomad 16723]  client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 15.304520627s
2017-01-04 17:13:06 c1 [nomad 16723]      2017/01/04 17:13:06.715888 [INFO] client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 15.304520627s
2017-01-04 17:13:22 c1 [nomad 16723]      2017/01/04 17:13:22.459677 [ERR] driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:13:22 c1 [nomad 16723]  driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:13:22 c1 [nomad 16723]      2017/01/04 17:13:22.459929 [INFO] client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:13:22 c1 [nomad 16723]  client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:13:22 c1 [nomad 16723]      2017/01/04 17:13:22.460674 [INFO] client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 10.260181801s
2017-01-04 17:13:22 c1 [nomad 16723]  client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 10.260181801s
2017-01-04 17:13:33 c1 [nomad 16723]      2017/01/04 17:13:33.023366 [ERR] driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:13:33 c1 [nomad 16723]      2017/01/04 17:13:33.023449 [INFO] client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:13:33 c1 [nomad 16723]      2017/01/04 17:13:33.023794 [INFO] client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 15.174743289s
2017-01-04 17:13:33 c1 [nomad 16723]  driver.docker: failed pulling container registry.weave.local/main/medveduka:0.1: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:13:33 c1 [nomad 16723]  client: failed to start task 'medveduka' for alloc '0336368d-94ad-bbbc-8625-5e45d65b401d': Failed to pull `registry.weave.local/main/medveduka:0.1`: Error: Status 502 trying to pull repository main/medveduka: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
2017-01-04 17:13:33 c1 [nomad 16723]  client: Restarting task "medveduka" for alloc "0336368d-94ad-bbbc-8625-5e45d65b401d" in 15.174743289s
2017-01-04 17:14:27 c1 [nomad 16723]      2017/01/04 17:14:27.095930 [INFO] driver.docker: created container e6e32635fc1d54213a4365689a171e14359f14482507602bc72fd673fbf3ba7c
2017-01-04 17:14:27 c1 [nomad 16723]  driver.docker: created container e6e32635fc1d54213a4365689a171e14359f14482507602bc72fd673fbf3ba7c
2017-01-04 17:14:27 c1 [nomad 16723]      2017/01/04 17:14:27.569743 [INFO] driver.docker: started container e6e32635fc1d54213a4365689a171e14359f14482507602bc72fd673fbf3ba7c
2017-01-04 17:14:27 c1 [nomad 16723]  driver.docker: started container e6e32635fc1d54213a4365689a171e14359f14482507602bc72fd673fbf3ba7c
2017-01-04 17:14:27 c1 [nomad 16723]      2017/01/04 17:14:27.573546 [WARN] client: error fetching stats of task medveduka: stats collection hasn't started yet
2017-01-04 17:14:27 c1 [nomad 16723]  client: error fetching stats of task medveduka: stats collection hasn't started yet

docker

2017-01-04 17:11:27 c1 [docker 25062]  INFO: 2017/01/04 14:11:27.394793 [nameserver 86:2b:76:bd:1d:81] container 55d2ba767d44a673245102313a969b749214cb06c59266412afe08c94a29db79 died; tombstoning entry redis.weave.local. -> 10.41.64.7
2017-01-04 17:11:27 c1 [docker 25062]  INFO: 2017/01/04 14:11:27.503182 [nameserver 86:2b:76:bd:1d:81] container 76d5716471de9276243efa6789a2ac7447fadc2c438d0c071c93173097176b2c died; tombstoning entry medveduka.weave.local. -> 10.41.64.20
2017-01-04 17:11:27 c1 [docker 25062]  INFO: 2017/01/04 14:11:27.934038 [nameserver 86:2b:76:bd:1d:81] container 4a3e3ba91583362bb6d5e5982e1aeb4aeb5a8c5ad266b81b547d1d9a3e1d2242 died; tombstoning entry zabbix-frontend.weave.local. -> 10.41.64.8
2017-01-04 17:11:28 c1 [docker 27877]  time="2017-01-04T17:11:28.066406755+03:00" level=error msg="Handler for POST /containers/55d2ba767d44a673245102313a969b749214cb06c59266412afe08c94a29db79/stop returned error: Container 55d2ba767d44a673245102313a969b749214cb06c59266412afe08c94a29db79 is already stopped"
2017-01-04 17:11:28 c1 [docker 27877]  time="2017-01-04T17:11:28.070246448+03:00" level=error msg="Handler for POST /containers/76d5716471de9276243efa6789a2ac7447fadc2c438d0c071c93173097176b2c/stop returned error: Container 76d5716471de9276243efa6789a2ac7447fadc2c438d0c071c93173097176b2c is already stopped"
2017-01-04 17:11:28 c1 [docker 27877]  time="2017-01-04T17:11:28.246562622+03:00" level=error msg="Handler for POST /containers/4a3e3ba91583362bb6d5e5982e1aeb4aeb5a8c5ad266b81b547d1d9a3e1d2242/stop returned error: Container 4a3e3ba91583362bb6d5e5982e1aeb4aeb5a8c5ad266b81b547d1d9a3e1d2242 is already stopped"
2017-01-04 17:11:32 c1 [docker 27877]  time="2017-01-04T17:11:32.264594143+03:00" level=info msg="Container 2a0f9205d093dc3185c12d4749641b5a11d31988a2793d19c97572fe3b8c9b4e failed to exit within 5 seconds of signal 15 - using the force"
2017-01-04 17:11:32 c1 [docker 27877]  time="2017-01-04T17:11:32.264783651+03:00" level=info msg="Container 11a6ab590236b02f006ed400719a924fef0d106cd6a0253e195a35ba72e7fc53 failed to exit within 5 seconds of signal 15 - using the force"
2017-01-04 17:11:32 c1 [docker 27877]  time="2017-01-04T17:11:32.264836240+03:00" level=info msg="Container 42a34404113bd702c9c166df6a549c038828d6969217b52225b21d8fc4b2bc7b failed to exit within 5 seconds of signal 15 - using the force"
2017-01-04 17:11:32 c1 [docker 27877]  time="2017-01-04T17:11:32.264919522+03:00" level=info msg="Container 9d40c30a72b70816f02638f8c36d017bc6cde0f323473a727d3de4ba0777ac5a failed to exit within 5 seconds of signal 15 - using the force"
2017-01-04 17:11:32 c1 [docker 27877]  time="2017-01-04T17:11:32.264989372+03:00" level=info msg="Container 9cc5bbae2b44ee43ac6d457567513c11d605004aea911d7ca1f0e3bfa2ba010d failed to exit within 5 seconds of signal 15 - using the force"
2017-01-04 17:11:32 c1 [docker 27877]  time="2017-01-04T17:11:32.265078895+03:00" level=info msg="Container 49386194222ad4579372f5326a6fbb0ad0f9c443c4845ebe335812bd877e4a2b failed to exit within 5 seconds of signal 15 - using the force"
2017-01-04 17:11:32 c1 [docker 27877]  time="2017-01-04T17:11:32.265138692+03:00" level=info msg="Container 6d87d7190be8184bafcc29d1b912163e6f5fe37bf3314b7161733efe02ffc640 failed to exit within 5 seconds of signal 15 - using the force"
2017-01-04 17:11:32 c1 [docker 25062]  INFO: 2017/01/04 14:11:32.340362 [nameserver 86:2b:76:bd:1d:81] container 2a0f9205d093dc3185c12d4749641b5a11d31988a2793d19c97572fe3b8c9b4e died; tombstoning entry syncthing.weave.local. -> 10.41.64.11
2017-01-04 17:11:32 c1 [docker 25062]  INFO: 2017/01/04 14:11:32.385636 [nameserver 86:2b:76:bd:1d:81] container 9cc5bbae2b44ee43ac6d457567513c11d605004aea911d7ca1f0e3bfa2ba010d died; tombstoning entry mta.weave.local. -> 10.41.64.12
2017-01-04 17:11:32 c1 [docker 25062]  INFO: 2017/01/04 14:11:32.461685 [nameserver 86:2b:76:bd:1d:81] container 42a34404113bd702c9c166df6a549c038828d6969217b52225b21d8fc4b2bc7b died; tombstoning entry prepodam.weave.local. -> 10.41.64.16
2017-01-04 17:11:32 c1 [docker 25062]  INFO: 2017/01/04 14:11:32.501014 [nameserver 86:2b:76:bd:1d:81] container 6d87d7190be8184bafcc29d1b912163e6f5fe37bf3314b7161733efe02ffc640 died; tombstoning entry nginx-prepodam.weave.local. -> 10.41.64.5
2017-01-04 17:11:32 c1 [docker 27877]  time="2017-01-04T17:11:32.646826937+03:00" level=error msg="Handler for GET /images/redis:3-alpine/json returned error: No such image: redis:3-alpine"
2017-01-04 17:11:32 c1 [docker 27877]  time="2017-01-04T17:11:32.647887993+03:00" level=error msg="Handler for GET /images/registry.weave.local/main/medveduka:0.1/json returned error: No such image: registry.weave.local/main/medveduka:0.1"
2017-01-04 17:11:32 c1 [docker 27877]  time="2017-01-04T17:11:32.648016527+03:00" level=error msg="Handler for GET /images/zabbix/zabbix-web-nginx-pgsql:alpine-3.2.1/json returned error: No such image: zabbix/zabbix-web-nginx-pgsql:alpine-3.2.1"
2017-01-04 17:11:32 c1 [docker 27877]  time="2017-01-04T17:11:32.816051996+03:00" level=error msg="Handler for POST /containers/49386194222ad4579372f5326a6fbb0ad0f9c443c4845ebe335812bd877e4a2b/stop returned error: Container 49386194222ad4579372f5326a6fbb0ad0f9c443c4845ebe335812bd877e4a2b is already stopped"
2017-01-04 17:11:32 c1 [docker 27877]  time="2017-01-04T17:11:32.926126902+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:33 c1 [docker 27877]  time="2017-01-04T17:11:33.149302045+03:00" level=error msg="Handler for POST /containers/2a0f9205d093dc3185c12d4749641b5a11d31988a2793d19c97572fe3b8c9b4e/stop returned error: Container 2a0f9205d093dc3185c12d4749641b5a11d31988a2793d19c97572fe3b8c9b4e is already stopped"
2017-01-04 17:11:33 c1 [docker 27877]  time="2017-01-04T17:11:33.224825593+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:33 c1 [docker 27877]  time="2017-01-04T17:11:33.316532660+03:00" level=error msg="Handler for POST /containers/9cc5bbae2b44ee43ac6d457567513c11d605004aea911d7ca1f0e3bfa2ba010d/stop returned error: Container 9cc5bbae2b44ee43ac6d457567513c11d605004aea911d7ca1f0e3bfa2ba010d is already stopped"
2017-01-04 17:11:33 c1 [docker 27877]  time="2017-01-04T17:11:33.412821109+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:33 c1 [docker 27877]  time="2017-01-04T17:11:33.459895309+03:00" level=error msg="Handler for POST /containers/9d40c30a72b70816f02638f8c36d017bc6cde0f323473a727d3de4ba0777ac5a/stop returned error: Container 9d40c30a72b70816f02638f8c36d017bc6cde0f323473a727d3de4ba0777ac5a is already stopped"
2017-01-04 17:11:33 c1 [docker 27877]  time="2017-01-04T17:11:33.537281160+03:00" level=error msg="Handler for POST /containers/11a6ab590236b02f006ed400719a924fef0d106cd6a0253e195a35ba72e7fc53/stop returned error: Container 11a6ab590236b02f006ed400719a924fef0d106cd6a0253e195a35ba72e7fc53 is already stopped"
2017-01-04 17:11:33 c1 [docker 27877]  time="2017-01-04T17:11:33.598900638+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:33 c1 [docker 27877]  time="2017-01-04T17:11:33.612009843+03:00" level=error msg="Handler for POST /containers/42a34404113bd702c9c166df6a549c038828d6969217b52225b21d8fc4b2bc7b/stop returned error: Container 42a34404113bd702c9c166df6a549c038828d6969217b52225b21d8fc4b2bc7b is already stopped"
2017-01-04 17:11:33 c1 [docker 27877]  time="2017-01-04T17:11:33.631071436+03:00" level=error msg="Handler for DELETE /images/sha256:9fce7d109adb2543f3d45e24e9c5e55a6550ae6b3fb06444178bd3a03895b6a9 returned error: conflict: unable to delete 9fce7d109adb (must be forced) - image is being used by stopped container cba3fd369ea1"
2017-01-04 17:11:33 c1 [docker 27877]  time="2017-01-04T17:11:33.668116633+03:00" level=error msg="Handler for POST /containers/6d87d7190be8184bafcc29d1b912163e6f5fe37bf3314b7161733efe02ffc640/stop returned error: Container 6d87d7190be8184bafcc29d1b912163e6f5fe37bf3314b7161733efe02ffc640 is already stopped"
2017-01-04 17:11:33 c1 [docker 27877]  time="2017-01-04T17:11:33.786468074+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:33 c1 [docker 27877]  time="2017-01-04T17:11:33.802621101+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:33 c1 [docker 27877]  time="2017-01-04T17:11:33.833653743+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:34 c1 [docker 27877]  time="2017-01-04T17:11:34+03:00" level=info msg="Firewalld running: false"
2017-01-04 17:11:34 c1 [docker 27877]  message repeated 3 times: [ time="2017-01-04T17:11:34+03:00" level=info msg="Firewalld running: false"]
2017-01-04 17:11:34 c1 [docker 27877]  time="2017-01-04T17:11:34+03:00" level=info msg="Firewalld running: false"
2017-01-04 17:11:34 c1 [docker 27877]  time="2017-01-04T17:11:34+03:00" level=info msg="Firewalld running: false"
2017-01-04 17:11:34 c1 [docker 25062]  INFO: 2017/01/04 14:11:34.986969 [nameserver 86:2b:76:bd:1d:81] Ignoring registration 8bc6beee5c2e. 10.41.64.8 8bc6beee5c2eef034df8c7847d44c55f62419956baff0c617fa71cbd4d8b7835 (not a subdomain of weave.local.)
2017-01-04 17:11:35 c1 [docker 25062]  INFO: 2017/01/04 14:11:35.080802 [nameserver 86:2b:76:bd:1d:81] adding entry for 0629bdbaf4ba0b5255ec7e1afb4458592fbf9a409604372ae2b318c433064f66: nginx-prepodam.weave.local. -> 10.41.64.11
2017-01-04 17:11:35 c1 [docker 27877]  time="2017-01-04T17:11:35.928342151+03:00" level=error msg="Handler for DELETE /images/sha256:d845268b2966c25c094a19077119d3d160d640764d696ad918f17c82bd3398e8 returned error: conflict: unable to delete d845268b2966 (cannot be forced) - image is being used by running container 782183b6d454"
2017-01-04 17:11:35 c1 [docker 25062]  INFO: 2017/01/04 14:11:35.949045 [nameserver 86:2b:76:bd:1d:81] adding entry for a75e9422d89bbc690a46060ada9c5a2868f087f63a342b2ace9d38d61d744a49: syncthing.weave.local. -> 10.41.64.7
2017-01-04 17:11:36 c1 [docker 27877]  time="2017-01-04T17:11:36.055092070+03:00" level=error msg="Handler for DELETE /images/sha256:e63febc686e3d0c8ac9f1da9593e884552fcd6055e7a1d1b925c6d04558f3685 returned error: conflict: unable to delete e63febc686e3 (cannot be forced) - image is being used by running container a75e9422d89b"
2017-01-04 17:11:36 c1 [docker 25062]  INFO: 2017/01/04 14:11:36.069571 [nameserver 86:2b:76:bd:1d:81] adding entry for 8eae6fd84e9c9279cd8177997fbb16494373a3656f865d02b9e9e6c0b4fd264f: mta.weave.local. -> 10.41.64.5
2017-01-04 17:11:36 c1 [docker 25062]  INFO: 2017/01/04 14:11:36.274998 [nameserver 86:2b:76:bd:1d:81] Ignoring registration 65069ed3401b. 10.41.64.10 65069ed3401bb431246e6ba671c16e257fcc004450daf05daadecebead32dc52 (not a subdomain of weave.local.)
2017-01-04 17:11:36 c1 [docker 27877]  time="2017-01-04T17:11:36.376927614+03:00" level=error msg="Handler for DELETE /images/sha256:388cbe6380428263ecaf7fdaa908ea596e1d147a6193dcf655451362f3017d60 returned error: reference does not exist"
2017-01-04 17:11:36 c1 [docker 25062]  INFO: 2017/01/04 14:11:36.390530 [nameserver 86:2b:76:bd:1d:81] adding entry for 35c583a7027eec590a82ca6b7e4854583dd5a7a1c13edc364970f5b65feaf61f: prepodam.weave.local. -> 10.41.64.12
2017-01-04 17:11:36 c1 [docker 27877]  time="2017-01-04T17:11:36.409856820+03:00" level=error msg="Handler for DELETE /images/sha256:a78b0507aa9aa1f2baa7d730deb069f2326db79620883a43b2380272e1f27a99 returned error: conflict: unable to delete a78b0507aa9a (cannot be forced) - image is being used by running container 8bc6beee5c2e"
2017-01-04 17:11:36 c1 [docker 27877]  time="2017-01-04T17:11:36.419124387+03:00" level=error msg="Handler for DELETE /images/sha256:388cbe6380428263ecaf7fdaa908ea596e1d147a6193dcf655451362f3017d60 returned error: conflict: unable to delete 388cbe638042 (cannot be forced) - image is being used by running container 35c583a7027e"
2017-01-04 17:11:36 c1 [docker 27877]  time="2017-01-04T17:11:36.421293327+03:00" level=error msg="Handler for DELETE /images/sha256:7b9cb1fff9917e2ac967f05f96c7bb5de0ff3e07024b7dbcece03680f6bb00d2 returned error: conflict: unable to delete 7b9cb1fff991 (cannot be forced) - image is being used by running container 8eae6fd84e9c"
2017-01-04 17:11:37 c1 [docker 27877]  time="2017-01-04T17:11:37.350660282+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:38 c1 [docker 27877]  time="2017-01-04T17:11:38.899271137+03:00" level=error msg="Error trying v2 registry: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:11:38 c1 [docker 27877]  time="2017-01-04T17:11:38.900159659+03:00" level=error msg="Attempting next endpoint for pull after error: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:11:38 c1 [docker 27877]  time="2017-01-04T17:11:38+03:00" level=info msg="Firewalld running: false"
2017-01-04 17:11:39 c1 [docker 25062]  INFO: 2017/01/04 14:11:39.467834 [nameserver 86:2b:76:bd:1d:81] adding entry for e3c434aff1dd624bab47272a53a77f390472f61c690a57588c368b9a2df30f70: redis.weave.local. -> 10.41.64.16
2017-01-04 17:11:42 c1 [docker 27877]  time="2017-01-04T17:11:42.013730329+03:00" level=error msg="Not continuing with pull after error: Error: Status 502 trying to pull repository main/medveduka: \"<html>\\r\\n<head><title>502 Bad Gateway</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>502 Bad Gateway</h1></center>\\r\\n<hr><center>nginx/1.10.1</center>\\r\\n</body>\\r\\n</html>\\r\\n\""
2017-01-04 17:11:46 c1 [docker 27877]  time="2017-01-04T17:11:46.278902595+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:46 c1 [docker 27877]  time="2017-01-04T17:11:46+03:00" level=info msg="Firewalld running: false"
2017-01-04 17:11:46 c1 [docker 25062]  INFO: 2017/01/04 14:11:46.809478 [nameserver 86:2b:76:bd:1d:81] adding entry for ba4fbaae4ad3c50b229ed3097361de5e91a8f2bf9f0bdc3c1187b403207c8008: zabbix-frontend.weave.local. -> 10.41.64.17
2017-01-04 17:11:57 c1 [docker 27877]  time="2017-01-04T17:11:57.202963919+03:00" level=info msg="Container ffc48fe2f9924d7ae93e4a4a2e7e44181fd272357a83c243e7bbb127410aef70 failed to exit within 30 seconds of signal 15 - using the force"
2017-01-04 17:11:57 c1 [docker 27877]  time="2017-01-04T17:11:57.207601895+03:00" level=info msg="Container 3c86a0db96bf247eeaad3e8d32caf727b7434a7e1da0a46f8bec6a5a5024fb12 failed to exit within 30 seconds of signal 15 - using the force"
2017-01-04 17:11:57 c1 [docker 27877]  time="2017-01-04T17:11:57.208547269+03:00" level=info msg="Container 06805b46389a8f61e7430e6ffcf1741ec3bef04630c2ae03aaf27b5296973152 failed to exit within 30 seconds of signal 15 - using the force"
2017-01-04 17:11:57 c1 [docker 27877]  time="2017-01-04T17:11:57.214924518+03:00" level=info msg="Container e3e67774e9cb087c8085a2cbc4d32d41f676e385fb275372217ab0d6cf898c7b failed to exit within 30 seconds of signal 15 - using the force"
2017-01-04 17:11:57 c1 [docker 27877]  time="2017-01-04T17:11:57.221555757+03:00" level=info msg="Container 511e9603b1b3ae1f0be6166e4d52f10899d605bd7e13262e2896b4c31e01b119 failed to exit within 30 seconds of signal 15 - using the force"
2017-01-04 17:11:57 c1 [docker 27877]  time="2017-01-04T17:11:57.224821125+03:00" level=info msg="Container 782183b6d4546d029afca992738f1201fe7addafb33cde4713c5d1bb78ad9e3e failed to exit within 30 seconds of signal 15 - using the force"
2017-01-04 17:11:57 c1 [docker 25062]  INFO: 2017/01/04 14:11:57.283784 [nameserver 86:2b:76:bd:1d:81] container 3c86a0db96bf247eeaad3e8d32caf727b7434a7e1da0a46f8bec6a5a5024fb12 died; tombstoning entry ticker.weave.local. -> 10.41.64.13
2017-01-04 17:11:57 c1 [docker 25062]  INFO: 2017/01/04 14:11:57.345255 [nameserver 86:2b:76:bd:1d:81] container 06805b46389a8f61e7430e6ffcf1741ec3bef04630c2ae03aaf27b5296973152 died; tombstoning entry ticker-minion.weave.local. -> 10.41.64.14
2017-01-04 17:11:57 c1 [docker 25062]  INFO: 2017/01/04 14:11:57.346635 [nameserver 86:2b:76:bd:1d:81] container e3e67774e9cb087c8085a2cbc4d32d41f676e385fb275372217ab0d6cf898c7b died; tombstoning entry xvostovnet.weave.local. -> 10.41.64.19
2017-01-04 17:11:57 c1 [docker 25062]  INFO: 2017/01/04 14:11:57.359609 [nameserver 86:2b:76:bd:1d:81] container 782183b6d4546d029afca992738f1201fe7addafb33cde4713c5d1bb78ad9e3e died; tombstoning entry nginx-xvostovnet.weave.local. -> 10.41.64.6
2017-01-04 17:11:58 c1 [docker 27877]  time="2017-01-04T17:11:58.812972164+03:00" level=error msg="Handler for POST /containers/511e9603b1b3ae1f0be6166e4d52f10899d605bd7e13262e2896b4c31e01b119/stop returned error: Container 511e9603b1b3ae1f0be6166e4d52f10899d605bd7e13262e2896b4c31e01b119 is already stopped"
2017-01-04 17:11:58 c1 [docker 27877]  time="2017-01-04T17:11:58.817972355+03:00" level=error msg="Handler for POST /containers/ffc48fe2f9924d7ae93e4a4a2e7e44181fd272357a83c243e7bbb127410aef70/stop returned error: Container ffc48fe2f9924d7ae93e4a4a2e7e44181fd272357a83c243e7bbb127410aef70 is already stopped"
2017-01-04 17:11:58 c1 [docker 27877]  time="2017-01-04T17:11:58.841747292+03:00" level=error msg="Handler for POST /containers/3c86a0db96bf247eeaad3e8d32caf727b7434a7e1da0a46f8bec6a5a5024fb12/stop returned error: Container 3c86a0db96bf247eeaad3e8d32caf727b7434a7e1da0a46f8bec6a5a5024fb12 is already stopped"
2017-01-04 17:11:58 c1 [docker 27877]  time="2017-01-04T17:11:58.937392678+03:00" level=error msg="Handler for POST /containers/782183b6d4546d029afca992738f1201fe7addafb33cde4713c5d1bb78ad9e3e/stop returned error: Container 782183b6d4546d029afca992738f1201fe7addafb33cde4713c5d1bb78ad9e3e is already stopped"
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.060920593+03:00" level=error msg="Handler for DELETE /images/sha256:d845268b2966c25c094a19077119d3d160d640764d696ad918f17c82bd3398e8 returned error: conflict: unable to delete d845268b2966 (cannot be forced) - image is being used by running container 0629bdbaf4ba"
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.067809753+03:00" level=error msg="Handler for POST /containers/06805b46389a8f61e7430e6ffcf1741ec3bef04630c2ae03aaf27b5296973152/stop returned error: Container 06805b46389a8f61e7430e6ffcf1741ec3bef04630c2ae03aaf27b5296973152 is already stopped"
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.115111984+03:00" level=error msg="Handler for DELETE /images/sha256:34fe050343c5fd3d0d40e8c6f377c910ce7bc5713abd245ae130b4f60205df1b returned error: conflict: unable to delete 34fe050343c5 (must be forced) - image is being used by stopped container 06805b46389a"
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.115830230+03:00" level=error msg="Handler for DELETE /images/sha256:08026ced55d81e011857a4903b06c357dcb9ff8ae02985ebd43c08f7daeaf991 returned error: conflict: unable to delete 08026ced55d8 (must be forced) - image is being used by stopped container e3e67774e9cb"
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.116228411+03:00" level=error msg="Handler for DELETE /images/sha256:34fe050343c5fd3d0d40e8c6f377c910ce7bc5713abd245ae130b4f60205df1b returned error: conflict: unable to delete 34fe050343c5 (must be forced) - image is being used by stopped container 06805b46389a"
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.166689194+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.168693852+03:00" level=error msg="Handler for POST /containers/e3e67774e9cb087c8085a2cbc4d32d41f676e385fb275372217ab0d6cf898c7b/stop returned error: Container e3e67774e9cb087c8085a2cbc4d32d41f676e385fb275372217ab0d6cf898c7b is already stopped"
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.174391045+03:00" level=error msg="Handler for POST /containers/create returned error: No such image: registry.weave.local/main/ticker:0.23"
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.222185720+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.234140765+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.359586167+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.375889092+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.377964555+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.979607404+03:00" level=error msg="Handler for POST /containers/create returned error: No such image: registry.weave.local/main/ticker:0.23"
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.979816694+03:00" level=error msg="Handler for POST /containers/create returned error: No such image: registry.weave.local/main/ticker:0.23"
2017-01-04 17:11:59 c1 [docker 27877]  time="2017-01-04T17:11:59.979880360+03:00" level=error msg="Handler for GET /images/registry.weave.local/main/medveduka:0.1/json returned error: No such image: registry.weave.local/main/medveduka:0.1"
2017-01-04 17:12:00 c1 [docker 27877]  time="2017-01-04T17:12:00.253451939+03:00" level=error msg="Error trying v2 registry: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:12:00 c1 [docker 27877]  time="2017-01-04T17:12:00.253487148+03:00" level=error msg="Attempting next endpoint for pull after error: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:12:00 c1 [docker 27877]  time="2017-01-04T17:12:00.392566047+03:00" level=error msg="Not continuing with pull after error: Error: Status 502 trying to pull repository main/medveduka: \"<html>\\r\\n<head><title>502 Bad Gateway</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>502 Bad Gateway</h1></center>\\r\\n<hr><center>nginx/1.10.1</center>\\r\\n</body>\\r\\n</html>\\r\\n\""
2017-01-04 17:12:00 c1 [docker 27877]  time="2017-01-04T17:12:00+03:00" level=info msg="Firewalld running: false"
2017-01-04 17:12:00 c1 [docker 25062]  INFO: 2017/01/04 14:12:00.603076 [nameserver 86:2b:76:bd:1d:81] adding entry for cb8bb62f2bfd4044c94baf1f9eec4ea513112c06130b1d6a28131a5c356168cf: nginx-xvostovnet.weave.local. -> 10.41.64.6
2017-01-04 17:12:00 c1 [docker 27877]  time="2017-01-04T17:12:00+03:00" level=info msg="Firewalld running: false"
2017-01-04 17:12:00 c1 [docker 27877]  time="2017-01-04T17:12:00+03:00" level=info msg="Firewalld running: false"
2017-01-04 17:12:00 c1 [docker 25062]  INFO: 2017/01/04 14:12:00.887947 [nameserver 86:2b:76:bd:1d:81] adding entry for 880f4fd375cf344cdf2367e37794db5d1cf1e4c03c6e4050cf9b46d228c16976: xvostovnet.weave.local. -> 10.41.64.14
2017-01-04 17:12:01 c1 [docker 25062]  INFO: 2017/01/04 14:12:01.011241 [nameserver 86:2b:76:bd:1d:81] Ignoring registration a3f022aa2838. 10.41.64.13 a3f022aa2838b655f7a29db382c7925b56e123bb5206ab4febefb3e09b00e40d (not a subdomain of weave.local.)
2017-01-04 17:12:16 c1 [docker 27877]  time="2017-01-04T17:12:16.231585602+03:00" level=error msg="Handler for GET /images/registry.weave.local/main/medveduka:0.1/json returned error: No such image: registry.weave.local/main/medveduka:0.1"
2017-01-04 17:12:16 c1 [docker 27877]  time="2017-01-04T17:12:16.383407877+03:00" level=error msg="Error trying v2 registry: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:12:16 c1 [docker 27877]  time="2017-01-04T17:12:16.383465846+03:00" level=error msg="Attempting next endpoint for pull after error: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:12:16 c1 [docker 27877]  time="2017-01-04T17:12:16.514662698+03:00" level=error msg="Not continuing with pull after error: Error: Status 502 trying to pull repository main/medveduka: \"<html>\\r\\n<head><title>502 Bad Gateway</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>502 Bad Gateway</h1></center>\\r\\n<hr><center>nginx/1.10.1</center>\\r\\n</body>\\r\\n</html>\\r\\n\""
2017-01-04 17:12:32 c1 [docker 27877]  time="2017-01-04T17:12:32.450801848+03:00" level=error msg="Handler for GET /images/registry.weave.local/main/medveduka:0.1/json returned error: No such image: registry.weave.local/main/medveduka:0.1"
2017-01-04 17:12:32 c1 [docker 27877]  time="2017-01-04T17:12:32.597095272+03:00" level=error msg="Error trying v2 registry: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:12:32 c1 [docker 27877]  time="2017-01-04T17:12:32.597907515+03:00" level=error msg="Attempting next endpoint for pull after error: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:12:32 c1 [docker 27877]  time="2017-01-04T17:12:32.719638054+03:00" level=error msg="Not continuing with pull after error: Error: Status 502 trying to pull repository main/medveduka: \"<html>\\r\\n<head><title>502 Bad Gateway</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>502 Bad Gateway</h1></center>\\r\\n<hr><center>nginx/1.10.1</center>\\r\\n</body>\\r\\n</html>\\r\\n\""
2017-01-04 17:12:50 c1 [docker 27877]  time="2017-01-04T17:12:50.933491195+03:00" level=error msg="Handler for GET /images/registry.weave.local/main/medveduka:0.1/json returned error: No such image: registry.weave.local/main/medveduka:0.1"
2017-01-04 17:12:51 c1 [docker 27877]  time="2017-01-04T17:12:51.059132480+03:00" level=error msg="Error trying v2 registry: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:12:51 c1 [docker 27877]  time="2017-01-04T17:12:51.059216632+03:00" level=error msg="Attempting next endpoint for pull after error: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:12:51 c1 [docker 27877]  time="2017-01-04T17:12:51.153067215+03:00" level=error msg="Not continuing with pull after error: Error: Status 502 trying to pull repository main/medveduka: \"<html>\\r\\n<head><title>502 Bad Gateway</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>502 Bad Gateway</h1></center>\\r\\n<hr><center>nginx/1.10.1</center>\\r\\n</body>\\r\\n</html>\\r\\n\""
2017-01-04 17:13:06 c1 [docker 27877]  time="2017-01-04T17:13:06.471130583+03:00" level=error msg="Handler for GET /images/registry.weave.local/main/medveduka:0.1/json returned error: No such image: registry.weave.local/main/medveduka:0.1"
2017-01-04 17:13:06 c1 [docker 27877]  time="2017-01-04T17:13:06.613192620+03:00" level=error msg="Error trying v2 registry: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:13:06 c1 [docker 27877]  time="2017-01-04T17:13:06.613494338+03:00" level=error msg="Attempting next endpoint for pull after error: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:13:06 c1 [docker 27877]  time="2017-01-04T17:13:06.712960787+03:00" level=error msg="Not continuing with pull after error: Error: Status 502 trying to pull repository main/medveduka: \"<html>\\r\\n<head><title>502 Bad Gateway</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>502 Bad Gateway</h1></center>\\r\\n<hr><center>nginx/1.10.1</center>\\r\\n</body>\\r\\n</html>\\r\\n\""
2017-01-04 17:13:22 c1 [docker 27877]  time="2017-01-04T17:13:22.023238086+03:00" level=error msg="Handler for GET /images/registry.weave.local/main/medveduka:0.1/json returned error: No such image: registry.weave.local/main/medveduka:0.1"
2017-01-04 17:13:22 c1 [docker 27877]  time="2017-01-04T17:13:22.311037038+03:00" level=error msg="Error trying v2 registry: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:13:22 c1 [docker 27877]  time="2017-01-04T17:13:22.311081006+03:00" level=error msg="Attempting next endpoint for pull after error: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:13:22 c1 [docker 27877]  time="2017-01-04T17:13:22.459124063+03:00" level=error msg="Not continuing with pull after error: Error: Status 502 trying to pull repository main/medveduka: \"<html>\\r\\n<head><title>502 Bad Gateway</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>502 Bad Gateway</h1></center>\\r\\n<hr><center>nginx/1.10.1</center>\\r\\n</body>\\r\\n</html>\\r\\n\""
2017-01-04 17:13:32 c1 [docker 27877]  time="2017-01-04T17:13:32.724812234+03:00" level=error msg="Handler for GET /images/registry.weave.local/main/medveduka:0.1/json returned error: No such image: registry.weave.local/main/medveduka:0.1"
2017-01-04 17:13:32 c1 [docker 27877]  time="2017-01-04T17:13:32.884897185+03:00" level=error msg="Error trying v2 registry: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:13:32 c1 [docker 27877]  time="2017-01-04T17:13:32.885235035+03:00" level=error msg="Attempting next endpoint for pull after error: received unexpected HTTP status: 502 Bad Gateway"
2017-01-04 17:13:33 c1 [docker 27877]  time="2017-01-04T17:13:33.022589450+03:00" level=error msg="Not continuing with pull after error: Error: Status 502 trying to pull repository main/medveduka: \"<html>\\r\\n<head><title>502 Bad Gateway</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>502 Bad Gateway</h1></center>\\r\\n<hr><center>nginx/1.10.1</center>\\r\\n</body>\\r\\n</html>\\r\\n\""
2017-01-04 17:13:48 c1 [docker 27877]  time="2017-01-04T17:13:48.202056870+03:00" level=error msg="Handler for GET /images/registry.weave.local/main/medveduka:0.1/json returned error: No such image: registry.weave.local/main/medveduka:0.1"
2017-01-04 17:14:27 c1 [docker 27877]  time="2017-01-04T17:14:27.021633785+03:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
2017-01-04 17:14:27 c1 [docker 27877]  time="2017-01-04T17:14:27+03:00" level=info msg="Firewalld running: false"
2017-01-04 17:14:27 c1 [docker 25062]  INFO: 2017/01/04 14:14:27.566702 [nameserver 86:2b:76:bd:1d:81] adding entry for e6e32635fc1d54213a4365689a171e14359f14482507602bc72fd673fbf3ba7c: medveduka.weave.local. -> 10.41.64.15
dadgar commented 7 years ago

Thanks for the logs! There is a race between one driver deleting an image and another driver trying to use it. Will get it fixed!

camerondavison commented 7 years ago

While the race condition fix would be great, seems like making some of these ephemeral docker failures recoverable would help. for instance I got the following Failed to create container: API error (500): device or resource busy would be nice to have that not make the task as dead where the only way to get it to start again is to resubmit the nomad job. Maybe #1461 would help?

cross referencing #2014

kak-tus commented 7 years ago

As I understand - #1461 was fixed in 0.5.1.

dbresson commented 7 years ago

Error message: failed to start task 'engine' for alloc '95f00ef3-8f3f-456a-32fb-b34f302a2ef1': Failed to determine image id for compellon/engine:Build709: no such image

docker log:

time="2017-01-09T21:49:23.354481285Z" level=error msg="Handler for POST /containers/f2c693ca8119c6181e9bda9f795e9d38923008eaf2ac5c7194eef9d75fbc6e3f/stop returned error: Container f2c693ca8119c6181e9bda9f795e9d38923008eaf2ac5c7194eef9d75fbc6e3f is already stopped" time="2017-01-09T21:49:23.459124370Z" level=error msg="Handler for GET /images/compellon/engine:Build709/json returned error: No such image: compellon/engine:Build709" time="2017-01-09T21:49:23.492944746Z" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap." time="2017-01-09T21:49:45.120859491Z" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap." time="2017-01-09T21:57:30.159503607Z" level=error msg="Handler for POST /containers/9a9872ca6dd8a14a9774440af7815810f5ee5beb99293f6b10c9c9eda516426e/stop returned error: Container 9a9872ca6dd8a14a9774440af7815810f5ee5beb99293f6b10c9c9eda516426e is already stopped" time="2017-01-09T21:57:30.167115370Z" level=error msg="Handler for DELETE /images/sha256:34540456adadbd2b7008e564b45d9d452ce316904971d8d5de29e88438f5db5c returned error: conflict: unable to delete 34540456adad (cannot be forced) - image is being used by running container ebac1e25bbdb" time="2017-01-09T21:57:30.648067127Z" level=error msg="Handler for POST /containers/9502e9484a53105d4f92756ab8318b66e9598324c3abd64387785e410a77db07/stop returned error: Container 9502e9484a53105d4f92756ab8318b66e9598324c3abd64387785e410a77db07 is already stopped" time="2017-01-09T21:57:30.669077023Z" level=error msg="Handler for GET /images/compellon/engine:Build709/json returned error: No such image: compellon/engine:Build709" time="2017-01-09T21:57:30.701393715Z" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap." time="2017-01-09T21:57:31.346531349Z" level=error msg="Handler for POST /containers/f6d32c8723aed06fea0aa684ac822633ef07b30476d1e03860d4a16a9def6768/stop returned error: Container f6d32c8723aed06fea0aa684ac822633ef07b30476d1e03860d4a16a9def6768 is already stopped" time="2017-01-09T21:57:31.354035601Z" level=error msg="Handler for DELETE /images/sha256:34540456adadbd2b7008e564b45d9d452ce316904971d8d5de29e88438f5db5c returned error: conflict: unable to delete 34540456adad (cannot be forced) - image is being used by running container 66fe283769e8" time="2017-01-09T22:00:53.891600404Z" level=error msg="Handler for GET /images/compellon/engine:Build709/json returned error: No such image: compellon/engine:Build709" time="2017-01-09T22:00:53.925956457Z" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap." time="2017-01-09T22:01:15.085761420Z" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap." time="2017-01-09T22:16:55.413340844Z" level=error msg="Handler for POST /containers/86520c2d5561321d2dcfc55df7815039bdebe6ceaddd6016e31bcb605ac0625b/stop returned error: Container 86520c2d5561321d2dcfc55df7815039bdebe6ceaddd6016e31bcb605ac0625b is already stopped" time="2017-01-09T22:16:55.423750391Z" level=error msg="Handler for POST /containers/e97196de6de0a751acfc14e0d17e012f3eb6dde97e568c466c8353a62156730d/stop returned error: Container e97196de6de0a751acfc14e0d17e012f3eb6dde97e568c466c8353a62156730d is already stopped" time="2017-01-09T22:16:55.482381499Z" level=error msg="Handler for DELETE /images/sha256:8ed685874fa58e01e6fc18bcfd5154d2ca79fcdac60c58624ad58d20e4807e20 returned error: conflict: unable to delete 8ed685874fa5 (cannot be forced) - image is being used by running container d0fe3968a3f6"

Nomad log: 2017/01/09 21:57:30.152077 [INFO] driver.docker: stopped container 9a9872ca6dd8a14a9774440af7815810f5ee5beb99293f6b10c9c9eda516426e 2017/01/09 21:57:30.170994 [ERR] client: dropping update to alloc 'c86076a8-0a31-e5f3-998f-f55ee0062a3e' 2017/01/09 21:57:30.387966 [ERR] client: dropping update to alloc 'c86076a8-0a31-e5f3-998f-f55ee0062a3e' 2017/01/09 21:57:30.622178 [INFO] driver.docker: stopped container 9502e9484a53105d4f92756ab8318b66e9598324c3abd64387785e410a77db07 2017/01/09 21:57:30.669255 [ERR] driver.docker: failed getting image id for compellon/engine:Build709: no such image 2017/01/09 21:57:30.669311 [INFO] client: failed to start task 'engine' for alloc '95f00ef3-8f3f-456a-32fb-b34f302a2ef1': Failed to determine image id forcompellon/engine:Build709: no such image 2017/01/09 21:57:30.672859 [INFO] client: Not restarting task: engine for alloc: 95f00ef3-8f3f-456a-32fb-b34f302a2ef1

kak-tus commented 7 years ago

@dadgar I think, this issue may be closed.

dadgar commented 7 years ago

Sounds good! I think 0.5.5 will also greatly mitigate this as we have image reference counting now!

blalor commented 7 years ago

I've got this problem with an instance running 0.5.5-rc2.

2017/03/14 15:52:14.640542 [DEBUG] client: starting task runners for alloc '468019cd-0a8f-9a42-5363-854f3864c876'
2017/03/14 15:52:14.641146 [DEBUG] client: starting task context for 'datadogcontainer' (alloc '468019cd-0a8f-9a42-5363-854f3864c876')
2017/03/14 15:52:14.645844 [DEBUG] client: driver event for alloc "468019cd-0a8f-9a42-5363-854f3864c876": Downloading image datadog/docker-dd-agent:latest
2017/03/14 15:52:14.726082 [DEBUG] driver.docker: docker pull datadog/docker-dd-agent:latest succeeded
2017/03/14 15:52:14.727479 [DEBUG] driver.docker: cancelling removal of image "datadog/docker-dd-agent:latest"
2017/03/14 15:52:14.727511 [DEBUG] driver.docker: image "datadog/docker-dd-agent:latest" (sha256:bbe5934017784344e0de3bc8ab71abb8203ff4405e35f29dc392093a773fba8b) reference count incremented: 1
2017/03/14 15:52:14 [DEBUG] plugin: starting plugin: /usr/local/bin/nomad []string{"/usr/local/bin/nomad", "executor", "{\"LogFile\":\"/var/lib/nomad/alloc/468019cd-0a8f-9a42-5363-854f3864c876/datadogcontainer/executor.out\",\"LogLevel\":\"DEBUG\"}"}
2017/03/14 15:52:14 [DEBUG] plugin: waiting for RPC address for: /usr/local/bin/nomad
2017/03/14 15:52:14 [DEBUG] plugin: nomad: 2017/03/14 15:52:14 [DEBUG] plugin: plugin address: unix /tmp/plugin978036711
2017/03/14 15:52:14.757653 [DEBUG] driver.docker: Setting default logging options to syslog and unix:///tmp/plugin540439578
2017/03/14 15:52:14.757696 [DEBUG] driver.docker: Using config for logging: {Type:syslog ConfigRaw:[] Config:map[syslog-address:unix:///tmp/plugin540439578]}
2017/03/14 15:52:14.757715 [DEBUG] driver.docker: using 536870912 bytes memory for datadogcontainer
2017/03/14 15:52:14.757725 [DEBUG] driver.docker: using 500 cpu shares for datadogcontainer
2017/03/14 15:52:14.757765 [DEBUG] driver.docker: binding directories []string{"/var/lib/nomad/alloc/468019cd-0a8f-9a42-5363-854f3864c876/alloc:/alloc", "/var/lib/nomad/alloc/468019cd-0a8f-9a42-5363-854f3864c876/datadogcontainer/local:/local", "/var/lib/nomad/alloc/468019cd-0a8f-9a42-5363-854f3864c876/datadogcontainer/secrets:/secrets", "/var/run/docker.sock:/var/run/docker.sock:ro", "/proc/:/host/proc/:ro", "/sys/fs/cgroup/:/host/sys/fs/cgroup:ro"} for datadogcontainer
2017/03/14 15:52:14.757782 [DEBUG] driver.docker: networking mode not specified; defaulting to bridge
2017/03/14 15:52:14.757897 [DEBUG] driver.docker: applied labels on the container: map[group:datadog-monitoring]
2017/03/14 15:52:14.757926 [DEBUG] driver.docker: setting container name to: datadogcontainer-468019cd-0a8f-9a42-5363-854f3864c876
2017/03/14 15:52:14.808062 [DEBUG] driver.docker: failed to create container "datadogcontainer-468019cd-0a8f-9a42-5363-854f3864c876" from image "datadog/docker-dd-agent:latest" (ID: "sha256:bbe5934017784344e0de3bc8ab71abb8203ff4405e35f29dc392093a773fba8b") (attempt 1): no such image
2017/03/14 15:52:14.808097 [ERR] driver.docker: failed to create container: no such image
2017/03/14 15:52:14 [DEBUG] plugin: /usr/local/bin/nomad: plugin process exited
2017/03/14 15:52:14.811433 [WARN] client: failed to start task "datadogcontainer" for alloc "468019cd-0a8f-9a42-5363-854f3864c876": Failed to create container: no such image
2017/03/14 15:52:14.812123 [INFO] client: Not restarting task: datadogcontainer for alloc: 468019cd-0a8f-9a42-5363-854f3864c876
2017/03/14 15:52:14.812914 [DEBUG] driver.docker: image id "sha256:bbe5934017784344e0de3bc8ab71abb8203ff4405e35f29dc392093a773fba8b" reference count decremented: 0
2017/03/14 15:52:14.813327 [INFO] client: marking allocation 468019cd-0a8f-9a42-5363-854f3864c876 for GC
2017/03/14 15:52:14.813925 [INFO] client: marking allocation 468019cd-0a8f-9a42-5363-854f3864c876 for GC
2017/03/14 15:52:14.814020 [DEBUG] client: couldn't add alloc 468019cd-0a8f-9a42-5363-854f3864c876 for GC: alloc 468019cd-0a8f-9a42-5363-854f3864c876 already being tracked for GC
github-actions[bot] commented 1 year ago

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.