Stopped jobs are not properly stopped on all clients

bsphere commented 7 years ago

Sometimes, after stopping a job, it is not properly stopped on all clients and even when the alloc status is "complete", the process is still running.

completely stopping the job without restarting doesn't solve this and it keeps being run by the client.

didn't experience this before 0.5.0

here's an example, alloc 0484fb00 is stopped, but still running on the client:

Allocations
ID        Eval ID   Node ID   Task Group        Desired  Status    Created At
5a1a78ea  476928b5  841ceb2e  feature-switcher  run      running   11/21/16 06:37:54 UTC
f395530d  476928b5  24758041  feature-switcher  run      running   11/21/16 06:37:54 UTC
0484fb00  4d972cf4  24758041  feature-switcher  stop     complete  11/21/16 06:30:32 UTC
6f7772e4  26ebeea7  841ceb2e  feature-switcher  stop     complete  11/21/16 06:29:32 UTC
6c8efdff  315f0b6f  841ceb2e  feature-switcher  stop     complete  11/21/16 06:06:31 UTC
9c995bf9  315f0b6f  24758041  feature-switcher  stop     complete  11/21/16 06:06:31 UTC

ps ax | grep feature
 9103 ?        Ssl    0:00 /opt/bin/nomad executor /var/lib/nomad/alloc/f395530d-b34d-575c-1ebd-ae127491ccb0/feature-switcher/feature-switcher-executor.out
 9201 ?        Ssl    0:00 /go/bin/microservice-feature-switcher
 9735 ?        Sl     0:00 /local/remote_syslog/remote_syslog -D --hostname=f395530d -d localhost -p 5140 feature-switcher=/alloc/logs/feature-switcher*
11290 ?        Ssl    0:49 /go/bin/microservice-feature-switcher
12733 pts/0    S+     0:00 grep --colour=auto feature
15302 ?        Ssl    0:07 /go/bin/microservice-feature-switcher
24207 ?        Ssl    0:09 /go/bin/microservice-feature-switcher
24593 ?        Ssl    0:09 /go/bin/microservice-feature-switcher
30347 ?        Ssl    0:00 /opt/bin/nomad executor /var/lib/nomad/alloc/0484fb00-a496-34a2-b863-c7cd85b9a250/feature-switcher/feature-switcher-executor.out
30500 ?        Ssl    0:00 /go/bin/microservice-feature-switcher
31258 ?        Sl     0:00 /local/remote_syslog/remote_syslog -D --hostname=0484fb00 -d localhost -p 5140 feature-switcher=/alloc/logs/feature-switcher*

Nomad version

0.5.0

Operating system and Environment details

Coreos-stable

jippi commented 7 years ago

I've observed this as well.. can you please submit your alloc executor log and the nomad client logs for the allocation? :)

bsphere commented 7 years ago

couldn't find anything interesting in the forwarded logs that I have...

jippi commented 7 years ago

do you run the client in DEBUG mode? there don't need to be any errors per se, but will greatly help the team try and reverse the cause of it :)

bsphere commented 7 years ago

its not in debug mode, the forwarded logs that I have shows almost nothing about that alloc id.. i suspect not everything was forwarded properly.

rebooting the client's VM made that "stale" task disappear. (maybe restart nomad client and docker kill would have helped as well)

On 21 November 2016 at 11:12, Christian Winther notifications@github.com wrote:

do you run the client in DEBUG mode? there don't need to be any errors per se, but will greatly help the team try and reverse the cause of it :)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hashicorp/nomad/issues/2016#issuecomment-261883346, or mute the thread https://github.com/notifications/unsubscribe-auth/AB2LTDt8BSodD9UuN1r5hasj7o5AZgyOks5rAWBigaJpZM4K33G8 .

dadgar commented 7 years ago

Could you grab the output of nomad alloc-status <bad-alloc-id> and also the logs on the client if you still have them or if this happens again?

bsphere commented 7 years ago

that alloc is gone... (i rebooted the VM). I actually did that when it happened and it looked like a normal running allocation.. (nomad status showed it as "completed")

dadgar commented 7 years ago

@bsphere If this happens again would you mind capturing that data? Do you have any steps to reproduce this behavior?

kak-tus commented 7 years ago

Same for me. I'll try add logs for some time. It seems, that this incorrect behavior after killing docker container of some task.

wuub commented 7 years ago

I think we've seen similiar behaviour on 0.5.0 (docker 1.12.3) when we accidentally updated a bit too many machines (don't ask ;)).

It caused a rush of allocations and task reshuffling. What we think we saw before rolling back to 0.4.1.

1) Docker allocations stuck in pending status unable to create containers because a container with the same name was already present on the host from previous run.

2) Docker allocations unable to start because of some lost, running container was occupying a port.

3) nomad executors keeping an eye on and restarting a running containers/allocations that were stopped according to API.

4) silliness in consul. 10 instances for nomad groups with count 5 (containers from previous run). Running tasks w/o consul registration. Etc.

Unfortunately we were unable to gather logs methodically due to accidentally large rollout, but we will attempt again this week and let you know

EDIT: we've seen exactly the same thing @bsphere is showing in the next comment ▾▾▾▾

bsphere commented 7 years ago

something wrong just happened again, I saw in the logs that a certain task is constantly being restarted and there's an error that a container cannot be created because it already exists.

this is the alloc-status:

nomad alloc-status -stats 4bd0f3ef
ID                 = 4bd0f3ef
Eval ID            = 737fc7b5
Name               = ci05-prom-exporters.prom-exporters[0]
Node ID            = 84cb4860
Job ID             = ci05-prom-exporters
Client Status      = running
Client Description = <none>
Created At         = 11/22/16 07:06:56 UTC

Task "augury-exporter" is "running"
Task Resources
CPU       Memory          Disk  IOPS  Addresses
3/50 MHz  11 MiB/200 MiB  0 B   0     http: 172.31.63.227:34644

Memory Stats
Cache    Max Usage  RSS     Swap
7.9 MiB  20 MiB     11 MiB  32 KiB

CPU Stats
Percent  Throttled Periods  Throttled Time
0.13%    0                  0

Recent Events:
Time                   Type      Description
11/22/16 07:07:01 UTC  Started   Task started by client
11/22/16 07:06:59 UTC  Received  Task received by client

Task "mongo-exporter" is "pending"
Task Resources
CPU     Memory  Disk  IOPS  Addresses
20 MHz  20 MiB  0 B   0     http: 172.31.63.227:45004

Recent Events:
Time                   Type            Description
11/22/16 09:44:11 UTC  Restarting      Task restarting in 1.135635306s
11/22/16 09:44:11 UTC  Driver Failure  failed to start task 'mongo-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists
11/22/16 09:44:09 UTC  Restarting      Task restarting in 1.245224377s
11/22/16 09:44:09 UTC  Driver Failure  failed to start task 'mongo-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists
11/22/16 09:44:08 UTC  Restarting      Task restarting in 1.235035471s
11/22/16 09:44:08 UTC  Driver Failure  failed to start task 'mongo-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists
11/22/16 09:44:07 UTC  Restarting      Task restarting in 1.13212092s
11/22/16 09:44:07 UTC  Driver Failure  failed to start task 'mongo-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists
11/22/16 09:44:05 UTC  Restarting      Task restarting in 1.062686352s
11/22/16 09:44:05 UTC  Driver Failure  failed to start task 'mongo-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists

Task "redis-exporter" is "pending"
Task Resources
CPU     Memory  Disk  IOPS  Addresses
20 MHz  20 MiB  0 B   0     http: 172.31.63.227:37780

Recent Events:
Time                   Type            Description
11/22/16 09:44:10 UTC  Restarting      Task restarting in 1.23742022s
11/22/16 09:44:10 UTC  Driver Failure  failed to start task 'redis-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists
11/22/16 09:44:09 UTC  Restarting      Task restarting in 1.008212702s
11/22/16 09:44:09 UTC  Driver Failure  failed to start task 'redis-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists
11/22/16 09:44:07 UTC  Restarting      Task restarting in 1.209586414s
11/22/16 09:44:07 UTC  Driver Failure  failed to start task 'redis-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists
11/22/16 09:44:06 UTC  Restarting      Task restarting in 1.18354629s
11/22/16 09:44:06 UTC  Driver Failure  failed to start task 'redis-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists
11/22/16 09:44:05 UTC  Restarting      Task restarting in 1.136458048s
11/22/16 09:44:05 UTC  Driver Failure  failed to start task 'redis-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists

I stopped the job, and that alloc is still running... here's the alloc-status again after job is stopped:

nomad alloc-status 4bd0f3ef
ID                 = 4bd0f3ef
Eval ID            = 737fc7b5
Name               = ci05-prom-exporters.prom-exporters[0]
Node ID            = 84cb4860
Job ID             = ci05-prom-exporters
Client Status      = running
Client Description = <none>
Created At         = 11/22/16 07:06:56 UTC

Task "augury-exporter" is "running"
Task Resources
CPU     Memory   Disk  IOPS  Addresses
50 MHz  200 MiB  0 B   0     http: 172.31.63.227:34644

Recent Events:
Time                   Type            Description
11/22/16 09:45:10 UTC  Started         Task started by client
11/22/16 09:45:09 UTC  Restarting      Task restarting in 1.114369247s
11/22/16 09:45:09 UTC  Driver Failure  failed to start task 'augury-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists
11/22/16 09:45:08 UTC  Restarting      Task restarting in 1.014606971s
11/22/16 09:45:08 UTC  Driver Failure  failed to start task 'augury-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists
11/22/16 09:45:06 UTC  Restarting      Task restarting in 1.189789042s
11/22/16 09:45:06 UTC  Driver Failure  failed to start task 'augury-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists
11/22/16 09:44:50 UTC  Restarting      Exceeded allowed attempts, applying a delay - Task restarting in 15.659892805s
11/22/16 09:44:50 UTC  Driver Failure  failed to start task 'augury-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists
11/22/16 09:44:49 UTC  Restarting      Task restarting in 1.245895267s

Task "mongo-exporter" is "running"
Task Resources
CPU     Memory  Disk  IOPS  Addresses
20 MHz  20 MiB  0 B   0     http: 172.31.63.227:45004

Recent Events:
Time                   Type      Description
11/22/16 07:07:00 UTC  Started   Task started by client
11/22/16 07:06:59 UTC  Received  Task received by client

Task "redis-exporter" is "running"
Task Resources
CPU     Memory  Disk  IOPS  Addresses
20 MHz  20 MiB  0 B   0     http: 172.31.63.227:37780

Recent Events:
Time                   Type      Description
11/22/16 07:07:01 UTC  Started   Task started by client
11/22/16 07:06:59 UTC  Received  Task received by client

client logs (they're repeating so I put some of it) -

Nov 22 09:08:05 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:08:05.150587 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:08:05 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:05.153286 [INFO] client: failed to start task 'redis-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:08:05 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:05.161287 [INFO] client: Restarting task "redis-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.00332746s Nov 22 07:08:05 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:05.248023 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:08:05 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:05.248452 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:08:05 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:05.248931 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:08:05 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:05.249164 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:08:05 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:05.249174 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:08:05 ip-172-31-63-227.ec2.internal consul[31724]:     2016/11/22 07:08:05 [INFO] agent: Synced service '_nomad-executor-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93-mongo-exporter-ci05-prom-mongo-exporter' Nov 22 07:08:05 ip-172-31-63-227.ec2.internal consul[31724]:     2016/11/22 07:08:05 [INFO] agent: Synced check 'fe79a1fcbc931c16ceef65f26a8e08be20068abe' Nov 22 07:08:05 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:05.401791989Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." 
Nov 22 09:08:32 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:08:32.400725 [INFO] client: failed to start task 'augury-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:08:32 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:32.401052 [INFO] client: Restarting task "augury-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.198284611s Nov 22 07:08:32 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:32.571010839Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:32 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:32.576092383Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:32 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:32.581837032Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:32 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:32.587642017Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:32 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:32.592281854Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:32 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:32.597120133Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:32 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:32.600204 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:08:32 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:32.601408 [INFO] client: failed to start task 'redis-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:08:32 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:32.601677 [INFO] client: Restarting task "redis-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.198284611s Nov 22 07:08:32 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:32.657658 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' 
Nov 22 09:08:34 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:08:34.297053 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:08:34 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:34.297072 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:08:34 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:34.297078 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:08:34 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:34.297082 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:08:34 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:34.297088 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:08:34 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:34.352899872Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:34 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:34.358506200Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:34 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:34.363623511Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:34 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:34.369495384Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:34 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:34.374395633Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:34 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:34.379675030Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:34 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:34.383190 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:08:34 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:34.384126 [INFO] client: failed to start task 'mongo-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:08:34 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:34.384389 [INFO] client: Restarting task "mongo-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.150986222s Nov 22 07:08:34 ip-172-31-63-227.ec2.internal sshd[30957]: pam_tally2(sshd:auth): Tally overflowed for user root 
Nov 22 09:08:39 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:08:39.443827 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:08:39 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:39.444118 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:08:39 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:39.444127 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:08:39 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:39.444132 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:08:39 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:39.444139 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:08:39 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:39.598901667Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:39 ip-172-31-63-227.ec2.internal sshd[31279]: pam_tally2(sshd:auth): Tally overflowed for user root 
Nov 22 09:08:40 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:08:40.641201 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:08:40 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:08:40.641206 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:08:40 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:40.857375351Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:40 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:40.867414848Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:40 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:40.874750386Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:40 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:40.881041819Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:40 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:40.887787592Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:08:40 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:08:40.894935275Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." 
Nov 22 09:09:03 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:09:02.858157 [INFO] client: failed to start task 'augury-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:09:02 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:02.858708 [INFO] client: Restarting task "augury-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.127809077s Nov 22 07:09:02 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:02.896942 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:09:02 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:02.896956 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:09:02 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:02.896963 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:09:02 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:02.896968 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:09:02 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:02.896973 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:09:02 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:02.976273660Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:02 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:02.985192245Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:02 ip-172-31-63-227.ec2.internal consul[31724]:     2016/11/22 07:09:02 [INFO] agent: Synced service '_nomad-executor-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93-redis-exporter-ci05-prom-redis-exporter' Nov 22 07:09:02 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:02.994118838Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:03 ip-172-31-63-227.ec2.internal consul[31724]:     2016/11/22 07:09:03 [INFO] agent: Synced check '86f16ffff9c0fc4621f6f3d304d8209840d21bc5' Nov 22 07:09:03 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:03.005139020Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:03 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:03.011140523Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:03 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:03.015912595Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:03 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:03.021313 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:09:03 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:03.032371 [INFO] client: failed to start task 'redis-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:09:03 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:03.032670 [INFO] client: Restarting task "redis-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.127809077s Nov 22 07:09:03 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:03.131058 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' 
Nov 22 09:09:13 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:09:13.526807 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.526825 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.526830 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.526835 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.526839 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:09:13 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:13.734730460Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:13 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:13.740221048Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:13 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:13.745085308Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:13 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:13.750052824Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:13 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:13.754917228Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:13 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:13.774506431Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.777448 [ERR] driver.docker: failed to create container: container already exists 
Nov 22 09:09:14 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:09:13.779040 [INFO] client: failed to start task 'augury-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.779663 [INFO] client: Restarting task "augury-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.164460463s Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.845750 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.845768 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.845774 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.845780 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.845785 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:09:13 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:13.863908009Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:13 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:13.872595401Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:13 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:13.881414396Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:13 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:13.888450330Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:13 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:13.894855071Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:13 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:13.902471879Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.905931 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.907547 [INFO] client: failed to start task 'mongo-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:09:13 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:13.908020 [INFO] client: Restarting task "mongo-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.164460463s Nov 22 07:09:14 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:14.131184 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' 
Nov 22 09:09:33 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:09:32.910578 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:09:32 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:32.910595 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:09:32 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:32.910600 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:09:32 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:32.910606 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:09:32 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:32.910610 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:09:32 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:32.938283310Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." 
Nov 22 09:09:35 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:09:35.516144 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:09:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:35.516149 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:09:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:35.516155 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:09:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:35.516160 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:09:35 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:35.547985137Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:35 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:35.552906860Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:35 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:35.569769448Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:35 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:35.588356086Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:35 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:35.602145337Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:35 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:35.608005999Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:35.611197 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:09:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:35.612495 [INFO] client: failed to start task 'redis-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:09:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:35.612795 [INFO] client: Restarting task "redis-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.032958608s Nov 22 07:09:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:35.731839 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:09:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:35.731855 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:09:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:35.731861 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:09:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:35.731866 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:09:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:35.731872 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' 
Nov 22 09:09:42 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:09:41.928513 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:09:41 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:41.928517 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:09:41 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:41.928522 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:09:41 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:41.994615660Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:42 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:42.000703993Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:42 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:42.005866073Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:42 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:42.010976786Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:42 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:42.016039769Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:42 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:09:42.020791274Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:09:42 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:09:42.023925 [ERR] driver.docker: failed to create container: container already exists 
Nov 22 09:10:00 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:10:00.485314 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:10:00 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:00.486331 [INFO] client: failed to start task 'redis-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:10:00 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:00.486617 [INFO] client: Restarting task "redis-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.184490263s Nov 22 07:10:00 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:00.492685014Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:00 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:00.497643992Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:00 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:00.502583485Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." 
Nov 22 09:10:02 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:10:01.922809 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:10:01 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:01.922824 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:10:01 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:01.922830 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:10:01 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:01.922836 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:10:01 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:01.922841 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:10:01 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:01.970979446Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:01 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:01.987783036Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:01 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:01.994914053Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:02 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:02.013381247Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:02 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:02.017663079Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:02 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:02.031761992Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:02 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:02.033652523Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:02 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:02.040995076Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:02 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:02.042343329Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:02 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:02.049161311Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:02 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:02.050513884Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:02 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:02.053878 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:10:02 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:02.055674470Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:02 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:02.057083 [INFO] client: failed to start task 'redis-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:10:02 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:02.057386 [INFO] client: Restarting task "redis-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.202732013s Nov 22 07:10:02 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:02.060404 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:10:02 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:02.061405 [INFO] client: failed to start task 'augury-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:10:02 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:02.061619 [INFO] client: Restarting task "augury-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.202732013s Nov 22 07:10:02 ip-172-31-63-227.ec2.internal consul[31724]:     2016/11/22 07:10:02 [INFO] agent: Synced service '_nomad-executor-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93-mongo-exporter-ci05-prom-mongo-exporter' 
Nov 22 09:10:06 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:10:06.306669 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:10:06 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:06.307282 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:10:06 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:06.307559 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:10:06 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:06.307795 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:10:06 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:06.307812 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:10:06 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:06.377879249Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:06 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:06.383111317Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:06 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:06.388185405Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:06 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:06.393784301Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:06 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:06.398580826Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:06 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:06.403518609Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:06 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:06.406649 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:10:06 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:06.407598 [INFO] client: failed to start task 'augury-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:10:06 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:06.407846 [INFO] client: Restarting task "augury-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.117452208s Nov 22 07:10:06 ip-172-31-63-227.ec2.internal consul[31724]:     2016/11/22 07:10:06 [INFO] agent: Deregistered service '_nomad-executor-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93-redis-exporter-ci05-prom-redis-exporter' 
Nov 22 09:10:12 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:10:12.342568 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:10:12 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:12.342591 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:10:12 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:12.342599 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:10:12 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:12.342604 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:10:12 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:12.342609 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:10:12 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:12.565440156Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:12 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:12.570243590Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:12 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:12.575207428Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:12 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:12.579947068Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:12 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:12.584762474Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:12 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:12.589330673Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:12 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:12.592450 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:10:12 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:12.593269 [INFO] client: failed to start task 'redis-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists 
Nov 22 09:10:13 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:10:12.918823 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:10:12 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:12.918838 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:10:12 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:12.918846 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:10:12 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:12.918852 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:10:12 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:12.918858 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:10:13 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:13.266289455Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." 
Nov 22 09:10:32 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:10:31.835087 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:10:31 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:31.836837 [INFO] client: failed to start task 'mongo-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:10:31 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:31.837133 [INFO] client: Restarting task "mongo-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.199709558s Nov 22 07:10:31 ip-172-31-63-227.ec2.internal consul[31724]:     2016/11/22 07:10:31 [INFO] agent: Synced service '_nomad-executor-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93-redis-exporter-ci05-prom-redis-exporter' Nov 22 07:10:31 ip-172-31-63-227.ec2.internal consul[31724]:     2016/11/22 07:10:31 [INFO] agent: Synced check '86f16ffff9c0fc4621f6f3d304d8209840d21bc5' Nov 22 07:10:32 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:32.002459386Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:32 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:32.009377508Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:32 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:32.014889922Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:32 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:32.019550421Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:32 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:32.024385112Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:32 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:32.046979533Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/redis-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container a4c373011fb1eb1b33ab31127e5f123109a9b5dff4215d133d8be5f148fa5845. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:32 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:32.064124 [ERR] driver.docker: failed to create container: container already exists 
Nov 22 09:10:35 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:10:35.294800 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:10:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:35.294820 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:10:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:35.294826 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:10:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:35.294831 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:10:35 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:35.294836 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:10:35 ip-172-31-63-227.ec2.internal systemd[1]: Started OpenSSH per-connection server daemon (218.65.30.60:1263). Nov 22 07:10:35 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:35.531799893Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:35 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:35.536854177Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:35 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:35.541727927Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." 
Nov 22 09:10:36 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:10:36.446786 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:10:36 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:36.446791 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:10:36 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:36.446796 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:10:36 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:36.446803 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:10:36 ip-172-31-63-227.ec2.internal consul[31724]:     2016/11/22 07:10:36 [INFO] agent: Deregistered service '_nomad-executor-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93-redis-exporter-ci05-prom-redis-exporter' Nov 22 07:10:36 ip-172-31-63-227.ec2.internal consul[31724]:     2016/11/22 07:10:36 [INFO] agent: Deregistered check '86f16ffff9c0fc4621f6f3d304d8209840d21bc5' Nov 22 07:10:36 ip-172-31-63-227.ec2.internal consul[31724]:     2016/11/22 07:10:36 [INFO] agent: Deregistered service '_nomad-executor-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93-mongo-exporter-ci05-prom-mongo-exporter' Nov 22 07:10:36 ip-172-31-63-227.ec2.internal consul[31724]:     2016/11/22 07:10:36 [INFO] agent: Deregistered check 'fe79a1fcbc931c16ceef65f26a8e08be20068abe' Nov 22 07:10:36 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:36.675897179Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:36 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:36.681012618Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:36 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:36.685881218Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:36 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:36.690502303Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:36 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:36.695140330Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:36 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:36.699966585Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." 
Nov 22 09:10:40 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:10:40.511075 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:10:40 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:40.511476 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:10:40 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:40.511483 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:10:40 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:40.511488 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:10:40 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:40.511493 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:10:40 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:40.588667365Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:40 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:40.593775917Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:40 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:40.598485683Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:40 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:40.603114158Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:40 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:40.607988525Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:40 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:40.612918411Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/mongo-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 747f23738d1cc131cc8b34cd28e16b76df2dd4a02142f054afc361da89a4521a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:40 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:40.616112 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:10:40 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:40.616868 [INFO] client: failed to start task 'mongo-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:10:40 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:40.617088 [INFO] client: Restarting task "mongo-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.014061321s 
Nov 22 09:10:42 ip-172-31-63-227.ec2.internal nomad:      2016/11/22 07:10:41.914541 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:10:41 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:41.914555 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:10:41 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:41.914561 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:10:41 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:41.914566 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:10:41 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:41.914571 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:10:42 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:42.076301782Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:42 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:42.081469717Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:42 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:42.086668040Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:42 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:42.091566953Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:42 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:42.096506147Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:42 ip-172-31-63-227.ec2.internal dockerd[1186]: time="2016-11-22T07:10:42.101404867Z" level=error msg="Handler for POST /containers/create returned error: Conflict. The name \"/augury-exporter-4bd0f3ef-d9fc-ef18-79be-6b62f127ea93\" is already in use by container 50a8f731fe194b1aa94e5597f016765d05ddeb4a2c31992bbcf90b6ff4c5021a. You have to remove (or rename) that container to be able to reuse that name." Nov 22 07:10:42 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:42.102279 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612' Nov 22 07:10:42 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:42.102296 [ERR] client: dropping update to alloc '9b5ceb92-a41a-b447-d62a-829c27ab48b1' Nov 22 07:10:42 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:42.102302 [ERR] client: dropping update to alloc '3e92286a-d239-5cab-259a-b352acb959ca' Nov 22 07:10:42 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:42.102308 [ERR] client: dropping update to alloc 'ddef22e9-96eb-936f-2e14-c3813968eda5' Nov 22 07:10:42 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:42.102314 [ERR] client: dropping update to alloc 'd94f154d-138c-72b6-c843-4aea72e00138' Nov 22 07:10:42 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:42.107148 [ERR] driver.docker: failed to create container: container already exists Nov 22 07:10:42 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:42.112237 [INFO] client: failed to start task 'augury-exporter' for alloc '4bd0f3ef-d9fc-ef18-79be-6b62f127ea93': Failed to create container: container already exists Nov 22 07:10:42 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:42.112700 [INFO] client: Restarting task "augury-exporter" for alloc "4bd0f3ef-d9fc-ef18-79be-6b62f127ea93" in 1.010937862s Nov 22 07:10:42 ip-172-31-63-227.ec2.internal nomad[3507]:     2016/11/22 07:10:42.241490 [ERR] client: dropping update to alloc 'aa4b6e1e-757a-1bdc-ecc6-6476dfebf612'

dadgar commented 7 years ago

@bsphere This will help a lot thanks! Have you found a way to reproduce this?

tantra35 commented 7 years ago

Seems that we have similar problem in our test enviroment. After lost of connectivity(little downtime of network switch) was lost of cluster leader, and on one of the server we saw job with allocation in pending state:

 alloc-status 44c12420
ID                 = 44c12420
Eval ID            = 43431e7a
Name               = fishdomDDDynamoNode.fishdomDDDynamoNode[0]
Node ID            = 9d6dcc47
Job ID             = fishdomDDDynamoNode
Client Status      = running
Client Description = <none>
Created At         = 11/26/16 13:22:41 MSK

Task "fishdomDDDynamoNode" is "running"
Task Resources
CPU        Memory           Disk  IOPS  Addresses
8/150 MHz  128 MiB/800 MiB  0 B   0     appport: 172.16.7.5:43128

Recent Events:
Time                   Type            Description
11/27/16 02:08:15 MSK  Started         Task started by client
11/27/16 02:07:46 MSK  Restarting      Task restarting in 26.84597984s
11/27/16 02:07:46 MSK  Driver Failure  failed to start task 'fishdomDDDynamoNode' for alloc '44c12420-e21d-d887-39ab-359e2be3e82c': Failed to create container: container already exists
11/27/16 02:07:15 MSK  Restarting      Task restarting in 30.488927226s
11/27/16 02:07:15 MSK  Driver Failure  failed to start task 'fishdomDDDynamoNode' for alloc '44c12420-e21d-d887-39ab-359e2be3e82c': Failed to create container: container already exists
11/27/16 02:06:48 MSK  Restarting      Task restarting in 27.48680681s
11/27/16 02:06:48 MSK  Driver Failure  failed to start task 'fishdomDDDynamoNode' for alloc '44c12420-e21d-d887-39ab-359e2be3e82c': Failed to create container: container already exists
11/27/16 02:06:20 MSK  Restarting      Task restarting in 27.476876604s
11/27/16 02:06:20 MSK  Driver Failure  failed to start task 'fishdomDDDynamoNode' for alloc '44c12420-e21d-d887-39ab-359e2be3e82c': Failed to create container: container already exists
11/27/16 02:05:38 MSK  Restarting      Exceeded allowed attempts, applying a delay - Task restarting in 42.523840297s

Task "fluend" is "running"
Task Resources
CPU        Memory          Disk  IOPS  Addresses
19/50 MHz  56 MiB/300 MiB  0 B   0

Recent Events:
Time                   Type        Description
11/26/16 13:24:17 MSK  Started     Task started by client
11/26/16 13:23:46 MSK  Restarting  Task restarting in 30.677634693s
11/26/16 13:23:46 MSK  Terminated  Exit Code: 0
11/26/16 13:23:45 MSK  Started     Task started by client
11/26/16 13:23:18 MSK  Restarting  Task restarting in 26.355124214s
11/26/16 13:23:18 MSK  Terminated  Exit Code: 0
11/26/16 13:23:16 MSK  Started     Task started by client
11/26/16 13:22:50 MSK  Restarting  Task restarting in 26.409686848s
11/26/16 13:22:50 MSK  Terminated  Exit Code: 0
11/26/16 13:22:44 MSK  Started     Task started by client

Our logs on moment of lost nomad leadership:

    2016/11/26 13:21:48 [WARN] raft: Heartbeat timeout from "192.168.30.1:4647" reached, starting election
    2016/11/26 13:21:50 [WARN] raft: Election timeout reached, restarting election
    2016/11/26 13:21:52 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2016/11/26 13:21:53.475301 [ERR] client: heartbeating failed. Retrying in 22.583879072s: failed to update status: No cluster leader
    2016/11/26 13:21:54.068771 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/26 13:21:54.093740 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/26 13:21:54.114396 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/26 13:21:54.179840 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/26 13:21:55.086868 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
    2016/11/26 13:21:55.086870 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
    2016/11/26 13:21:55.086887 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
    2016/11/26 13:21:55.086901 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
    2016/11/26 13:21:55.167779 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
    2016/11/26 13:21:55.167968 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
    2016/11/26 13:21:55.167874 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
    2016/11/26 13:21:55.167837 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
    2016/11/26 13:22:52 [WARN] raft: Rejecting vote request from 192.168.30.1:4647 since we have a leader: 192.168.30.6:4647
    2016/11/26 13:23:01 [ERR] memberlist: Failed TCP fallback ping: read tcp 192.168.30.5:59438->192.168.30.1:4648: i/o timeout
    2016/11/26 13:23:13.847350 [WARN] client: error fetching stats of task redis: stats collection hasn't started yet
    2016/11/26 13:24:05.095446 [WARN] client: error fetching stats of task gardenscapesFBDynamoNode: stats collection hasn't started yet
    2016/11/26 13:24:05.096097 [WARN] client: error fetching stats of task gardenscapesMacDynamo: stats collection hasn't started yet
    2016/11/26 13:24:05.294723 [WARN] client: error fetching stats of task fishdomDDDynamoNode: stats collection hasn't started yet
    2016/11/26 13:24:06.364104 [WARN] client: error fetching stats of task townshipMacDynamoNode: stats collection hasn't started yet
    2016/11/26 13:24:06.366004 [WARN] client: error fetching stats of task townshipWPDynamoNode: stats collection hasn't started yet
    2016/11/26 13:24:06.381523 [ERR] driver.docker: failed to create container: container already exists
    2016/11/26 13:24:32.638629 [ERR] driver.docker: failed to create container: container already exists
    2016/11/26 13:24:58.580372 [ERR] driver.docker: failed to create container: container already exists
    2016/11/26 13:25:28.287943 [ERR] driver.docker: failed to create container: container already exists
    2016/11/26 13:25:58.916660 [ERR] driver.docker: failed to create container: container already exists

and time to time with follow messages:

    2016/11/26 15:48:49.725650 [ERR] driver.docker: failed to create container: container already exists
    2016/11/26 15:49:16.615787 [ERR] driver.docker: failed to create container: container already exists
    2016/11/26 15:49:47.796130 [ERR] driver.docker: failed to create container: container already exists
    2016/11/26 15:50:02.290829 [ERR] driver.docker: failed to create container: container already exists
    2016/11/26 15:50:29.674524 [ERR] driver.docker: failed to create container: container already exists
    2016/11/26 15:50:55.686682 [ERR] driver.docker: failed to create container: container already exists
2016/11/26 15:50:55 [ERR] yamux: Failed to write header: write unix @->/tmp/plugin062805702: write: broken pipe
==> Caught signal: broken pipe
    2016/11/26 15:51:25.816153 [ERR] driver.docker: failed to create container: container already exists
    2016/11/26 15:51:56.124662 [ERR] driver.docker: failed to create container: container already exists
    2016/11/26 15:52:22.643962 [ERR] driver.docker: failed to create container: container already exists
    2016/11/26 15:52:51.276695 [ERR] driver.docker: failed to create container: container already exists

We tried 2 times on node with problem, remove existing container(docker stop <container id>), but nomad again launching them, and error continued. At third time (we again stop container on which nomad complain that it already exists) and nomad begins work normally, and allocation transform from pending state to running state

And at last for test we make nomad node-drain -enable -self -yes on server with problem container. And found that nomad lost some of it exicutors:

After drain we see foollow status:

root@server5:/home/ruslan# /opt/nomad/nomad node-status -self
ID     = 9d6dcc47
Name   = server5
Class  = ceph
DC     = test
Drain  = true
Status = ready
Uptime = 168h47m10s

Allocated Resources
CPU         Memory       Disk         IOPS
0/6956 MHz  0 B/5.8 GiB  0 B/824 GiB  0/0

Allocation Resource Utilization
CPU         Memory
0/6956 MHz  0 B/5.8 GiB

Host Resource Utilization
CPU            Memory           Disk
5005/7980 MHz  4.2 GiB/9.8 GiB  17 GiB/886 GiB

Allocations
ID        Eval ID   Job ID                    Task Group                Desired Status  Client Status
aced5b2c  8fb85127  zooMMMDynamo              zooMMMDynamo              stop            complete
c34297f6  4f2ac982  townshipMacDynamoNode     townshipMacDynamoNode     stop            complete
b0b7f7a8  9f844212  townshipDynamoTeamServer  townshipDynamoTeamServer  stop            complete
9c48bd86  3b6269f1  skycharmsDynamo           skycharmsDynamo           stop            complete
72132314  c2b38fff  gardenscapesMacDynamo     gardenscapesMacDynamo     stop            complete
c7f09f33  b5af3e70  redis                     redis                     stop            complete

and at processlist we see follow orphaned nomad executor:

root      9749  0.0  0.0  17040  1860 tty1     Ss+  нояб.22   0:00 /sbin/getty -8 38400 tty1
root     29888  0.6  0.1 618212 14704 ?        Ssl  нояб.26  27:26 /opt/nomad/nomad_0.5.0-rc2/nomad executor /var/lib/nomad/alloc/44c12420-e21d-d887-39ab-359e2be3e8
root     29916  0.0  0.3 387560 32628 ?        Sl   нояб.26   3:32  \_ /opt/td-agent/embedded/bin/ruby /usr/sbin/td-agent --no-supervisor -c /etc/td-agent/td-agent.
root     32631  0.2  0.0 517524  8152 ?        Ssl  нояб.27   7:28 /opt/nomad/nomad_0.5.0-rc2/nomad executor /var/lib/nomad/alloc/44c12420-e21d-d887-39ab-359e2be3e8
root     24426  1.7  0.3 1974996 37084 ?       Ssl  нояб.27  62:51 /opt/nomad/nomad agent -config=/etc/nomad/nomad.json

akaspin commented 7 years ago

Bump

orthecreedence commented 6 years ago

I believe this is happening to me as well. This was after a cluster-wide upgrade. Here was my process:

Had 4 nomad servers (1 vote-only), 8 nomad clients, all running (0.7.1)
Upgraded each server, one-by-one (leader server last) to 0.8.3
Installed the new 0.8.3 binary on all clients at the same time
Accidentally restarted nomad agent on all clients at the same time
Many of our jobs stopped working, services stopped registering with consul
Removing /var/lib/nomad and restarting the nomad agent on the clients seemed to fix, this brought the cluster back up
Now, seeing several [ERR] client: dropping update to alloc 'abc123...' messages looping in some of the clients

    2018/06/13 07:36:02.203108 [ERR] client: dropping update to alloc 'fdffa6f5-c3d2-2a01-ce9f-3b1c8a885404'
    2018/06/13 07:36:02.203113 [ERR] client: dropping update to alloc '2e04fa2d-a066-d8ad-5bef-3c741a56fdd8'
    2018/06/13 07:36:02.203119 [ERR] client: dropping update to alloc '57582246-94de-cc54-2960-d22d6561889b'
    2018/06/13 07:36:02.203124 [ERR] client: dropping update to alloc 'cc91c5c8-bdd0-563f-ed2e-b51472bb01dd'
    2018/06/13 07:36:02.203129 [ERR] client: dropping update to alloc '55832972-0040-e0d9-0a49-77527c77c577'
    2018/06/13 07:36:02.203134 [ERR] client: dropping update to alloc '384a241c-1b59-a6f6-95d7-b84fad61716f'
    2018/06/13 07:36:02.203140 [ERR] client: dropping update to alloc '48b730f0-3099-6ecc-7ff1-7a6e1e82409f'
    2018/06/13 07:36:02.203145 [ERR] client: dropping update to alloc '457a8d9b-ceb5-f11f-0343-e582606b4e34'
    2018/06/13 07:36:02.203150 [ERR] client: dropping update to alloc 'a6b2c9f9-3fba-406c-089c-9ec13db55e27'
    2018/06/13 07:36:02.203155 [ERR] client: dropping update to alloc 'bcf1de33-5fcf-0bc3-c942-aa99a8177d14'
    2018/06/13 07:36:02.203162 [ERR] client: dropping update to alloc '9c32f6fe-b076-c0a1-ac5b-30744d5ccf47'
    2018/06/13 07:36:02.203173 [ERR] client: dropping update to alloc '5e8eaa25-b8b5-4d49-14c9-0b897b8aea5f'
    2018/06/13 07:36:02.203179 [ERR] client: dropping update to alloc '1b285a3f-5429-cb84-fc95-d4654b618738'
    2018/06/13 07:36:02.203185 [ERR] client: dropping update to alloc 'e5732cff-bde1-88ca-04ff-c314bbf81082'
    2018/06/13 07:36:02.203190 [ERR] client: dropping update to alloc '05a32a38-244e-0215-1546-90b6d5c4491d'
    2018/06/13 07:36:02.203195 [ERR] client: dropping update to alloc '6bdeeb87-6982-a082-93aa-262bc83df0ea'
    2018/06/13 07:36:02.203200 [ERR] client: dropping update to alloc 'f2e2ffa7-b7c0-9ada-598d-616d359fb0f1'
    2018/06/13 07:36:02.203210 [ERR] client: dropping update to alloc '8305a110-fc63-b69e-dfd2-0719692e7149'
    2018/06/13 07:36:02.203215 [ERR] client: dropping update to alloc 'cc955851-5abb-1e4c-3813-e897cdbb353d'

root@hive01:/home/server# nomad alloc-status 8305a110
ID                  = 8305a110
Eval ID             = fc2072ca
Name                = worker.api.worker[0]
Node ID             = a396d9d7
Job ID              = worker.api
Job Version         = 1143
Client Status       = complete
Client Description  = <none>
Desired Status      = stop
Desired Description = alloc not needed due to job update
Created             = 17h42m ago
Modified            = 30s ago

Task "worker-instance" is "dead"
Task Resources
CPU       Memory   Disk     IOPS  Addresses
1200 MHz  1.0 GiB  300 MiB  0

Task Events:
Started At     = 2018-06-12T23:00:08Z
Finished At    = 2018-06-12T23:08:10Z
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type        Description
2018-06-12T16:08:10-07:00  Killed      Task successfully killed
2018-06-12T16:08:05-07:00  Killing     Sent interrupt. Waiting 5s before force killing
2018-06-12T16:00:08-07:00  Started     Task started by client
2018-06-12T16:00:07-07:00  Task Setup  Building Task Directory
2018-06-12T16:00:07-07:00  Received    Task received by client

If I can provide more info, please let me know.

schmichael commented 6 years ago

Replies to @orthecreedence:

Had 4 nomad servers (1 vote-only), 8 nomad clients, all running (0.7.1)

Hm, I'm confused by this: Nomad does not have vote only servers. Nomad 0.8 Enterprise supports non-voting servers: https://www.nomadproject.io/docs/agent/configuration/server.html#non_voting_server

Nomad always suggests an odd number of voting servers to ensure one side of partitions is able to form a quorum.

Many of our jobs stopped working, services stopped registering with consul

Restarting all clients should not have broken anything. While a rolling restart is safer, clients should have restarted and restored running allocations without issue.

Can you provide more information on why they stopped working? Similar error messages to the other posters? Nomad 0.8 should fix some or all of the issues from above. Ideally we could open new issues for specific restart bugs.

Removing /var/lib/nomad and restarting the nomad agent on the clients seemed to fix, this brought the cluster back up

This will cause all jobs to be rescheduled -- which is a fine way to recover except that the actual task processes/containers are likely still running along with their Nomad executors (sidecar processes for logging and containerization).

Depending on the design of your services this might be fine. However, you will want to clean up your cluster at some point.

Now, seeing several [ERR] client: dropping update to alloc 'abc123...' messages looping in some of the clients

This is unfortunate but likely just a useless log line. From the alloc status you posted the allocation has been stopped which should be expected in this circumstance as all jobs are being rescheduled which will create new allocations. Allocations can never transition from stopped to running again, so dropping updates is fine.

We're investigating why this occurred and how to improve the logging.

shantanugadgil commented 6 years ago

jfyi ...

https://github.com/hashicorp/nomad/issues/4401#issuecomment-396556309

Updating running Nomad agents causes issues for me too. More pronounced when I update amd reboot than when I only update.

orthecreedence commented 6 years ago

Hm, I'm confused by this: Nomad does not have vote only servers. Nomad 0.8 Enterprise supports non-voting servers: https://www.nomadproject.io/docs/agent/configuration/server.html#non_voting_server

Nomad always suggests an odd number of voting servers to ensure one side of partitions is able to form a quorum.

Whoops, my bad. I thought I had configured non-voting, maybe I am losing my mind. Also, good tip on the odd servers. I'll make sure to change to 3 servers.

Can you provide more information on why they stopped working? Similar error messages to the other posters? Nomad 0.8 should fix some or all of the issues from above. Ideally we could open new issues for specific restart bugs.

Probably not. I wasn't running the logs in debug mode and was pretty frantic trying to get everything running once the cluster emptied out. This isn't really my issue though, since everything is running fine now.

This is unfortunate but likely just a useless log line.

That's fine, but the log line keeps looping. I have dozens of allocations that, every minute, report as [ERR] client: dropping update to alloc ...

So really my issue is more along the lines of...how do I get nomad to stop trying to update these allocs? Some of them are for jobs that have been purged, yet they keep showing up. Everything seems to be working normally, but I'm wondering if there's a way to stop looping these log messages, or if this is a symptom of a larger problem.

Thanks!

orthecreedence commented 6 years ago

In my nomad servers, I'm seeing this (minutely):

2018/06/13 18:39:02.728189 [WARN] nomad: Attempting to apply large raft entry (type 13) (1099408 bytes)

I wonder if this has something to do with the minutely dropping update to alloc messages?

schmichael commented 6 years ago

@orthecreedence Likely although completely unrelated to the original issue. The original issue should have been fixed a long time ago, and we forgot to close this issue!

I'm going to close this issue. Please open a new one and follow the template providing as much logging as you can (you can attach zips and such to Github issues).

A sample job status, alloc status, job file for one of the allocs in the error messages you mentioned above would be useful as well.

To anyway affected by the original issue. Please open a new issue if something like this is still occurring. The bug should be fixed by now or at the very least displaying different task events and log messages.

github-actions[bot] commented 1 year ago

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / nomad

Stopped jobs are not properly stopped on all clients #2016

Nomad version

Operating system and Environment details