Nodes won't garbage collect

Nomad version

Nomad v0.7.0

Operating system and Environment details

Ubuntu 16.04 consul 0.9.2/0.9.3

Issue

Nodes won't GC old jobs:

root@ip-10-30-1-43:/etc/nomad.d# nomad job status | wc -l
220

I see submit's back as far as September. I've enabled debug logging, and my client settings are:

client {
    enabled = true
    gc_max_allocs = 1000

    node_class = "uat"
    no_host_uuid = false

    max_kill_timeout = "30s"

    network_speed = 0
    cpu_total_compute = 0

    gc_interval = "1m"
    gc_disk_usage_threshold = 80
    gc_inode_usage_threshold = 70
    gc_parallel_destroys = 2

    reserved {
        cpu = 0
        memory = 0
        disk = 0
        reserved_ports = "22"
    }

    options = {
        "driver.blacklist" = "raw_exec,qemu"
        "docker.auth.config" = "/etc/nomad.d/.dockercfg"
    }

    meta = {
    }
    }

I see zero log lines matching 'garbage', which is what I expect to see based on gc.go.

Reproduction steps

This one is hard, I don't know why the gc gets skipped/ignored so I'm not sure what to say about reproduction.

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

I deleted all the lines matching 'secret', and snipped out some company name stuff. Hopefully this shows the lack of garbage/gc tho.

``` Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "381e8b66-8190-e2e1-7594-6e6c16c5d804" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "38d77692-9c1a-6b9c-185b-a99902b580b2" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "399c836c-9e5f-6eb5-5e57-346d995eb61c" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "3b983e9d-f6d1-7f0c-8d0e-e1209abd0bea" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "3c1a5d50-9cf1-5070-c616-faa5c878307a" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "3c4c8178-c8a5-a4fc-505e-12439e498d2e" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "3d74e161-f4ad-fb47-6bf4-24308c042ea8" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "3ebb8609-0a0f-22ec-da55-091301002b03" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/local/packetbeat.ctmpl" => "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/new/packetbeat.yml" Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) checking template bdd2cd5c1f0936027fea314561f2e73c Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/local/symbiosis.uat.ctmpl" => "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/new/symbiosis.conf" Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "403baf3c-00a2-6e5d-5f14-88fcd4cc688d" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "41c3c6a5-e52d-308c-3dcc-0b7f45fe8a1d" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "44aae9b7-d17e-79d1-33e0-ed31e5bc7cd0" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "44ae7315-243e-cfdb-3f34-d95b277a9565" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "454db4b9-733a-1211-bb5c-370cb8a4bcc1" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "457d3482-09fc-142a-3e15-6b450bad3583" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: waiting for Vault token for task api in alloc "34d1328e-8da7-25e8-eee0-989aaf1ec8ca" Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: error fetching stats of task api: stats collection hasn't started yet Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: driver.docker: re-attaching to docker process: e72ce139be202e49712c1cc61eef410f7268e645ca9b0a770d45982c4e70bc11 Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5c82b5f7-574a-ad91-8517-c9ac7a3a46e8" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 35dd56b3-109c-606b-2ef7-2a5e87720286 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 381e8b66-8190-e2e1-7594-6e6c16c5d804 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 38d77692-9c1a-6b9c-185b-a99902b580b2 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "46b50b1e-042d-f0dc-f4ca-c616d94a3b0e" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 399c836c-9e5f-6eb5-5e57-346d995eb61c for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "478c768c-aa3b-a869-daa8-da41132590b8" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 3b983e9d-f6d1-7f0c-8d0e-e1209abd0bea for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 3c1a5d50-9cf1-5070-c616-faa5c878307a for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 3c4c8178-c8a5-a4fc-505e-12439e498d2e for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "48ae0f5e-53e1-7a09-0ced-9970a2e1c86f" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "4a594eca-8a63-acbe-99e9-11c2673c2519" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "4b8e3fe5-0ffe-8bf9-44b5-679f14a560ae" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "4c48c911-2fee-7f13-175f-3f706efa6ddb" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "4cc8f47d-3f32-61f5-df8a-33ddf439eba3" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "4cd9956c-80be-23d0-1cb5-2ace59bcfd3c" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "4f7cbfbd-f4b1-6644-a561-973d9aae0128" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5050f1e7-c45b-0136-af40-4b9c15f3d41c" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "50d7116e-808b-4ae0-f6f2-5be1a06ad746" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5149620f-f211-cf40-a102-ac51c8c23caa" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "515ac6ae-0076-d6ec-ffc3-c53ef2262725" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5162b0ed-a687-ac1d-4f30-33406acaac63" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "51b7fca8-5941-2b38-7681-75b3bc495cec" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "51bb7b8e-2f7d-b478-aca7-6083fa94fb9b" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "54dd04ba-52fb-aa87-33b9-7825853888e0" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "556ad3d2-0612-5229-261b-e0000e09d67b" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "55d24bd1-93ab-b99b-ffe3-2075b4a7c27d" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "562cbd87-3d30-5d7c-b714-5d5ed92c3dea" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "570da840-3b69-3bba-b3c1-a94af4aa1f63" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "58927b32-e1f9-6628-2bb6-b8192366c032" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5977c5d3-5541-caed-df66-ce0b57a91889" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5a2dc874-8c15-d3a0-12d1-4ec3c3a591c4" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5b3533fa-1a2e-18e8-a54f-4689f41056e9" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5ba17d69-11b6-056b-adb9-e7cca5151640" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 3d74e161-f4ad-fb47-6bf4-24308c042ea8 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5c6aff2d-f11f-f18e-fef6-b8e92a13c167" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 3ebb8609-0a0f-22ec-da55-091301002b03 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 403baf3c-00a2-6e5d-5f14-88fcd4cc688d for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 41c3c6a5-e52d-308c-3dcc-0b7f45fe8a1d for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 44aae9b7-d17e-79d1-33e0-ed31e5bc7cd0 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 44ae7315-243e-cfdb-3f34-d95b277a9565 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 454db4b9-733a-1211-bb5c-370cb8a4bcc1 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 457d3482-09fc-142a-3e15-6b450bad3583 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: driver.docker: re-attached to handle: DOCKER:{"Version":"0.7.0","Image":"tricorder:1.7.0-SNAPSHOT","ImageID":"sha256:1908e504484857f3396bff0997c32a810835b0ceb98b1478e403322c7617e98c","ContainerID":"e72ce139be202e49712c1cc61eef410f7268e645ca9b0a770d45982c4e70bc11","KillTimeout":5000000000,"MaxKillTimeout":30000000000,"PluginConfig":{"Pid":31748,"AddrNet":"unix","AddrName":"/tmp/plugin834450568"}} Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5c82b5f7-574a-ad91-8517-c9ac7a3a46e8 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 46b50b1e-042d-f0dc-f4ca-c616d94a3b0e for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 478c768c-aa3b-a869-daa8-da41132590b8 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 48ae0f5e-53e1-7a09-0ced-9970a2e1c86f for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 4a594eca-8a63-acbe-99e9-11c2673c2519 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 4b8e3fe5-0ffe-8bf9-44b5-679f14a560ae for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 4c48c911-2fee-7f13-175f-3f706efa6ddb for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 4cc8f47d-3f32-61f5-df8a-33ddf439eba3 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 4cd9956c-80be-23d0-1cb5-2ace59bcfd3c for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5050f1e7-c45b-0136-af40-4b9c15f3d41c for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 50d7116e-808b-4ae0-f6f2-5be1a06ad746 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 515ac6ae-0076-d6ec-ffc3-c53ef2262725 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5162b0ed-a687-ac1d-4f30-33406acaac63 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 51b7fca8-5941-2b38-7681-75b3bc495cec for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 51bb7b8e-2f7d-b478-aca7-6083fa94fb9b for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 54dd04ba-52fb-aa87-33b9-7825853888e0 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 556ad3d2-0612-5229-261b-e0000e09d67b for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 55d24bd1-93ab-b99b-ffe3-2075b4a7c27d for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 562cbd87-3d30-5d7c-b714-5d5ed92c3dea for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 570da840-3b69-3bba-b3c1-a94af4aa1f63 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 58927b32-e1f9-6628-2bb6-b8192366c032 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5977c5d3-5541-caed-df66-ce0b57a91889 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5a2dc874-8c15-d3a0-12d1-4ec3c3a591c4 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5b3533fa-1a2e-18e8-a54f-4689f41056e9 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5ba17d69-11b6-056b-adb9-e7cca5151640 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5c6aff2d-f11f-f18e-fef6-b8e92a13c167 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) rendered "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/local/symbiosis.uat.ctmpl" => "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/new/symbiosis.conf" Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) checking template 0d345435336ca507d95eac9d44730182 Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/local/secrets.ctmpl" => "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/new/secrets.conf" Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) watching 24 dependencies Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) all templates rendered Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: unblocking task api for alloc "06540f5a-3a84-7ba7-e2f7-cf403a0fd43e": Template Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 4f7cbfbd-f4b1-6644-a561-973d9aae0128 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5149620f-f211-cf40-a102-ac51c8c23caa for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: driver.docker: version of executor: 1.1.0 Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: driver.docker: image "tricorder:1.7.0-SNAPSHOT" (sha256:1908e504484857f3396bff0997c32a810835b0ceb98b1478e403322c7617e98c) reference count incremented: 1 Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: starting task context for 'api' (alloc '5d3fbe30-29a0-8245-b179-6f317fd6b35e') Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: waiting for Vault token for task api in alloc "5d3fbe30-29a0-8245-b179-6f317fd6b35e" Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: consul.sync: successfully updated services in Consul Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "89a880f4-2f7a-3b44-c1e4-2f8c22b89179" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 89a880f4-2f7a-3b44-c1e4-2f8c22b89179 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "8be4fa7a-ad4f-ae00-47cd-9d5ac0c663cd" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 8be4fa7a-ad4f-ae00-47cd-9d5ac0c663cd for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "8d60c99e-b719-6abb-0a3d-b004b08f069c" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 8d60c99e-b719-6abb-0a3d-b004b08f069c for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: starting task runners for alloc '5d3fbe30-29a0-8245-b179-6f317fd6b35e' Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5dc5b022-30b2-a5ae-875e-d9ee382db679" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "8e895f9b-d99b-c4ee-a214-9e2f4db5c8d7" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 8e895f9b-d99b-c4ee-a214-9e2f4db5c8d7 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "8ec810f6-e606-9e14-51a6-2de2b1dbf28f" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 8ec810f6-e606-9e14-51a6-2de2b1dbf28f for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "9326e3ee-e253-776f-2340-92cd84021861" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 9326e3ee-e253-776f-2340-92cd84021861 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "94936d21-87ec-68da-ac40-31fc7d8f74fd" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 94936d21-87ec-68da-ac40-31fc7d8f74fd for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "8fb00559-13f0-c137-786a-25c91ddb806e" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 8fb00559-13f0-c137-786a-25c91ddb806e for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "95effff9-5d45-c61a-0f80-58a946c8f17b" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "971b712b-52da-486d-1ad5-878259681530" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 971b712b-52da-486d-1ad5-878259681530 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 95effff9-5d45-c61a-0f80-58a946c8f17b for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "9550675b-f357-9159-065c-9df3c43df388" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 9550675b-f357-9159-065c-9df3c43df388 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "c6128070-e458-e8a5-29d8-5ceaca237ca2" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation c6128070-e458-e8a5-29d8-5ceaca237ca2 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "c624f808-ce02-85b1-e0fd-6baaa0d44660" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation c624f808-ce02-85b1-e0fd-6baaa0d44660 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "c6d0970f-3e26-14ed-0484-67bfd26f2f63" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation c6d0970f-3e26-14ed-0484-67bfd26f2f63 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "c737bb4f-acfa-7702-3066-3d4da26ea150" in terminal status, waiting for destroy Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation c737bb4f-acfa-7702-3066-3d4da26ea150 for GC Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "c9fd26c2-e11f-621f-8fdb-8c07b816a4a7" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation c9fd26c2-e11f-621f-8fdb-8c07b816a4a7 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ce236fc1-94d8-f1d2-2516-73ca47a43a57" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation ce236fc1-94d8-f1d2-2516-73ca47a43a57 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ce69ca97-1b9b-d1df-52ce-57b0c1a6d0c0" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation ce69ca97-1b9b-d1df-52ce-57b0c1a6d0c0 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "cf95d8b3-5847-e53d-9908-cef313e624c4" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation cf95d8b3-5847-e53d-9908-cef313e624c4 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d04bca0a-de7a-227b-b8b5-571dc701d68f" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d04bca0a-de7a-227b-b8b5-571dc701d68f for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d16d4240-ee9a-9bea-4eac-23a032b75336" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d16d4240-ee9a-9bea-4eac-23a032b75336 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d173f415-f515-5965-fda3-c591b4a0c136" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d173f415-f515-5965-fda3-c591b4a0c136 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d2b14329-7241-ff9e-6d84-d7c8965a38cc" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d2b14329-7241-ff9e-6d84-d7c8965a38cc for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "f4eb781e-738b-70f3-17e3-797ca0148864" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation f4eb781e-738b-70f3-17e3-797ca0148864 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9e7ef90a-a863-b0aa-a3e7-9a17e1830700" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 9e7ef90a-a863-b0aa-a3e7-9a17e1830700 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c2e3d045-bcaf-69a4-aff9-9e4e2f9b7073" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation c2e3d045-bcaf-69a4-aff9-9e4e2f9b7073 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: retrieved Vault token for task api in alloc "34d1328e-8da7-25e8-eee0-989aaf1ec8ca" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) creating new runner (dry: false, once: false) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) final config: {"Consul":{"Address":"10.30.1.43:8500","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"4dd5818e-8c54-477c-5d93-59e21cb3f989","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000},"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"WARN","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/scribe.conf","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/scribe.uat.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/filebeat.yml","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/filebeat.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/packetbeat.yml","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/packetbeat.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"}],"Vault":{"Address":"http://vault.service.consul:8200","Enabled":true,"Grace":0,"RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0}} Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) creating watcher Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) starting Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) running initial templates Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 682fda72530db76619ac43b9c401faa0 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) was not watching 6 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/scribe/host) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/scribe/port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/auth0/realm) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/auth0/domain) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/auth0/port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/scribe/auth0/client_id) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 915a99f10d848e1e2574aa93400c1e4e Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) was not watching 2 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/elasticsearch/cluster_name) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/uat/logger) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template ad21b17e45fc4983f2cbb42e63caad8a Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 2 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 8 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: driver.docker: re-attaching to docker process: fb33dae2482d130cc42ad298ccda03656277fe3785e3d929046bc73afc628852 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: driver.docker: re-attached to handle: DOCKER:{"Version":"0.7.0","Image":"ingestion:1.7.0-SNAPSHOT","ImageID":"sha256:5bf29961b34956d131dc0707009b66e3b07165d49d92511d0dd8e833125713a3","ContainerID":"fb33dae2482d130cc42ad298ccda03656277fe3785e3d929046bc73afc628852","KillTimeout":5000000000,"MaxKillTimeout":30000000000,"PluginConfig":{"Pid":26992,"AddrNet":"unix","AddrName":"/tmp/plugin936240069"}} Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d62bec89-0c99-4a20-a47d-c4cd275eb355" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d62bec89-0c99-4a20-a47d-c4cd275eb355 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 5dc5b022-30b2-a5ae-875e-d9ee382db679 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "f143817c-b23d-563c-4f60-b84e34cb7ae3" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation f143817c-b23d-563c-4f60-b84e34cb7ae3 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "60f13b2f-17d3-34ff-7ce0-8cb6a0aec7a8" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 60f13b2f-17d3-34ff-7ce0-8cb6a0aec7a8 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "61621de8-e028-5d72-c0b8-298398bc471a" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 61621de8-e028-5d72-c0b8-298398bc471a for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "62360a88-840e-3c45-0648-d63177699216" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 62360a88-840e-3c45-0648-d63177699216 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "6276e411-655d-1738-ff54-5c16472ee7e2" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/elasticsearch/cluster_name) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/scribe/auth0/client_id) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/auth0/domain) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 682fda72530db76619ac43b9c401faa0 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 4 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 915a99f10d848e1e2574aa93400c1e4e Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template ad21b17e45fc4983f2cbb42e63caad8a Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 8 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/auth0/realm) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/scribe/port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/scribe/host) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 682fda72530db76619ac43b9c401faa0 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 915a99f10d848e1e2574aa93400c1e4e Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template ad21b17e45fc4983f2cbb42e63caad8a Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 8 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/auth0/port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 682fda72530db76619ac43b9c401faa0 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/scribe.uat.ctmpl" => "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/scribe.conf" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 915a99f10d848e1e2574aa93400c1e4e Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 6276e411-655d-1738-ff54-5c16472ee7e2 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "63a1179d-0ae4-620c-2f27-e117c513f135" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 63a1179d-0ae4-620c-2f27-e117c513f135 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "65f06722-772a-8c48-d2e1-183d68307535" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 65f06722-772a-8c48-d2e1-183d68307535 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template ad21b17e45fc4983f2cbb42e63caad8a Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 8 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency vault.read(secret/uat/logger) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 682fda72530db76619ac43b9c401faa0 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/scribe.uat.ctmpl" => "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/scribe.conf" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 915a99f10d848e1e2574aa93400c1e4e Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/filebeat.ctmpl" => "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/filebeat.yml" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template ad21b17e45fc4983f2cbb42e63caad8a Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/packetbeat.ctmpl" => "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/packetbeat.yml" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 8 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) all templates rendered Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: retrieved Vault token for task api in alloc "5d3fbe30-29a0-8245-b179-6f317fd6b35e" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) creating new runner (dry: false, once: false) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "6b8c58f9-d5a2-900e-ef6d-ec258fb819d8" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) final config: {"Consul":{"Address":"10.30.1.43:8500","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"4dd5818e-8c54-477c-5d93-59e21cb3f989","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000},"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"WARN","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/tricorder.conf","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/tricorder.uat.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/secrets.conf","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/secrets.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/filebeat.yml","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/filebeat.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/packetbeat.yml","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/packetbeat.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"}],"Vault":{"Address":"http://vault.service.consul:8200","Enabled":true,"Grace":0,"RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0}} Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) creating watcher Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) starting Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) running initial templates Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0af01921e526f7ed9227f49001c30d2e Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "6c9ff7d7-827e-3e91-3ccc-efe1b62590b8" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "6d00d422-5bcc-0015-571a-5b1bff4e2efe" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "6d88842f-13b6-fc9f-667c-1da28934de23" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "6f3568d9-8b8f-fbe1-27f8-759983b61c00" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) was not watching 19 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/tricorder/host) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/tricorder/port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/tricorder/external_host) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/tricorder/external_port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/schema_audit/bucket) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/auth0/domain) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/auth0/port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/tricorder/auth0/client_id) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/influx/postgres/database) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.get(uat/postgres/consul/address) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/influx/postgres/port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(database/creds/tricorder-uat) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/influx/postgres/test) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/elasticsearch/cluster_name) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/uat/tricorder) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.get(uat/elasticsearch/consul/address) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/elasticsearch/transport/port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.get(uat/tricorder/deploy/git_sha) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/mrn) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0d345435336ca507d95eac9d44730182 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "710df41f-94f0-b3a9-ae4b-90e0082f3f69" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "717e8e23-478b-5a89-23db-4991edf8fe37" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7180fee2-c804-7484-ad90-5bb6f8727039" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7214744e-86e6-0e98-d7c7-46564eec83d8" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7405bfcb-6679-661c-28d6-1e74ff100c58" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "756baf79-f842-b57b-e541-56b5010c0972" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "768581b0-7a97-c2dc-4c7a-597e097cab56" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "76c8e63c-15d4-945a-c073-1d290bffce92" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "79f5ed48-6e4e-f5b9-878a-70aa206770a8" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7bfa7846-32ed-c73e-096b-7ec0c6de4b4a" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "89514bb8-2395-3e2d-446c-51dec5eeb9d4" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7d7fd2a3-932d-f644-5478-7e0ba59e16e7" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7d9d1c40-4e30-13ad-505e-1e703fe8ae15" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7dea9772-5e2f-0551-2184-ccf0eb4f6003" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7e88bb46-d31f-88aa-86d1-6c71bbd8d407" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7ec8e849-5a76-0c5a-570d-963d3233727d" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7f8bab5e-e0e7-6d07-26a3-26f0f821e1ab" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "80ff8676-2a91-791b-d217-ea2119647c16" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "83b19aa3-9ec4-72f3-be49-ebd09efe756b" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "84592f6a-a5ea-e44d-e4b9-ea3d95b0e24b" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "861e6f18-d5d4-189e-2974-2dd4308944e9" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "87491d75-e296-15ee-d265-fc2d766a8bc0" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) was not watching 9 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/s3/encryption/production-integration-messages-es) [vault.read snips] Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/s3/encryption/staging-integration-messages-es) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/s3/encryption/uat-integration-messages-es) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template d74433fe4620cb7647d82eaf4f8e7835 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) was not watching 1 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/uat/logger) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 362d96b61c7729f34665b804cc36fefe Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 2 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 29 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "8856ec7d-f8ac-a58f-9a56-b961e3720bc1" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "92df5c45-bbac-1175-d526-eca9f0641fad" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "8dfc4075-ed7b-d3df-7f11-e30f1c189d10" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "8e5c4942-f826-661b-9461-506b1cc48b89" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "95929532-3909-e568-aca4-362079c7b7b5" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9618af56-658e-1e10-a45f-13bf23b81a10" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "60bcb176-7853-af47-0c3b-d382600eb35f" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9bacbac5-f1e9-f19b-6288-269ae56c5b64" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d6f24bdc-b3ad-5aaa-241c-d74bb3191245" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: error fetching stats of task api: stats collection hasn't started yet Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9bf6f578-3e6f-e10f-3589-8515450f3c83" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d751bf37-e051-42fb-61b4-3c279db8f983" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d779c585-addd-a9e2-df7a-2e253f6a9c2d" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9c0f45eb-8196-4b4d-6158-e53cd6935ed0" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a2d72cd6-9876-0d38-8ef6-728f01220902" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a3580421-357c-8b67-fc1a-63f15a508a3a" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a3d8d31c-9104-972d-2dde-05bd2fa27549" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a40bf408-d692-b116-dc5e-979b5511deed" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a46d87cc-0b6a-ed11-7332-ad9cf83e1746" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a4e7aaa8-47e3-2233-acc7-3ecd187d1fa0" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.get(uat/tricorder/deploy/git_sha) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0af01921e526f7ed9227f49001c30d2e Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 18 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0d345435336ca507d95eac9d44730182 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a5777b51-9280-128c-1c72-102e5a5737c9" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a5bc53b9-aa97-f73c-a425-79484e819223" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a65e9103-3651-7a69-d8ee-a94f83c52079" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a8a89a65-1c62-35ea-d5d9-2cf2ec3dc646" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a947549d-2684-b846-3f61-d1df70c710be" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a9f8f4cc-96dd-6bca-cc7d-d819049b0d98" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "aad858f5-32d7-f4eb-1041-434d5218561a" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ab158044-69a9-9467-d8c9-54b154b194e1" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 9 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template d74433fe4620cb7647d82eaf4f8e7835 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 2 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 362d96b61c7729f34665b804cc36fefe Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 2 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies [snip still needed stuff] Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 29 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/elasticsearch/transport/port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/tricorder/external_host) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/tricorder/host) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/tricorder/port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/tricorder/auth0/client_id) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/auth0/domain) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.get(uat/postgres/consul/address) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/auth0/port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/schema_audit/bucket) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/influx/postgres/test) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/influx/postgres/port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/elasticsearch/cluster_name) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.get(uat/elasticsearch/consul/address) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/influx/postgres/database) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency vault.read(secret/s3/encryption/production-integration-messages-es) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency vault.read(secret/mrn) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency vault.read(secret/uat/tricorder) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/tricorder/external_port) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency vault.read(secret/s3/encryption/staging-docbot-out) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0af01921e526f7ed9227f49001c30d2e Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ac834e5b-025c-a947-eac0-93323bdf25ea" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "aea79f81-4f59-0a83-7319-6edae590f61f" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0d345435336ca507d95eac9d44730182 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "af195838-dd14-6612-e6c3-4c9ad03ab455" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "af54da3b-6a54-514b-9cad-94d907301aed" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "b1deefd6-79e4-013b-a863-3a3bcb39b5f2" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "b5651fa2-fa12-f74d-5c5e-be231a5731ce" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "b73bba18-671e-5f9d-4217-1c9b2b22f723" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "b97cd23b-354a-c4f2-2753-9252eea00c61" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ba5f1ce0-2202-245f-5ea3-438651c8ca97" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ba63d222-db04-e7f7-02b3-2479d10ea7c5" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "bad0bc14-7e5e-0728-9ee9-ab9bb49739dd" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "bce49302-0578-ceaf-4288-645552124ed1" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "bf28e0af-7f35-9258-e946-c48bf94819ce" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "bf5ae2d1-60fd-d040-5321-714198eea793" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "bf966954-eb9b-090c-2708-5c8709f3cb50" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c03b42b7-ff48-189b-0e2b-f57e3888bb0d" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c07b875d-d7d5-9b46-6ca2-4687682532eb" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d91ddb19-42f6-ea2b-3d68-c41d094d6efa" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "db913530-1ece-1d26-0e62-893e742ff758" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "dc4add48-2db4-2c53-1287-e845372d7445" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "dd088b39-d286-5e48-9b5b-0abde1631951" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "dd0cfd72-3a94-a9b5-1afc-320b3388499f" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "de08ffbd-1bbf-615b-6a69-56d3c56cc170" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "de2d724a-1ef3-7414-be4e-c9075158995b" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "de72049a-f30d-cd1f-2341-c56dc1afffcf" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "de9cabd7-4b85-152a-da52-13706bb01fd6" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c131ee79-b8d3-31e1-f3b6-a21a1ce4c8d8" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c17c6e42-1445-27b2-aa83-d38a9f0b788b" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c1f47321-c997-e448-dc70-c24a14a16019" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c26ffb7b-2449-32ea-d7e8-5251e6a59a85" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c2871eff-20c1-15dd-03f8-1f75a7f49aca" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "e1c7fe90-38d4-a1c8-f63b-90144c3400c8" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "e2963bd6-2b29-4783-c570-867bd81f4504" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c2de33c2-31b0-ad88-883e-d8c75a681723" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "e5fe5d98-e783-50f3-2b14-dcc91e04e0bb" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "e71b3486-559e-e304-b6f5-17e74a50b34f" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "e71b4a8e-a7dd-b54a-8789-544c314668e2" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "e8b6ff71-fdad-216e-9e45-03e5c5e5250c" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "921e7989-36c1-3248-e7dc-4f4256295b54" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9aa58df0-9bd1-2015-61c9-7089bf6d3132" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ed22dad2-8a86-2771-b756-86b0d5713ee6" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "989066bb-e16d-54da-1fd6-a9ee4ab6b35f" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "97d494a3-c113-ca17-8be2-2f1fb4438f06" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "5e0496dd-f515-86fc-fd64-5d19807061a0" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9d9e09b9-ce50-b7c7-1777-7dce63b8557c" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d9149162-0f62-5187-e3d2-eb7361037ce5" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 7 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template d74433fe4620cb7647d82eaf4f8e7835 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9b1f7023-1b14-0604-b03b-e01ad934930e" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "edd4d0f9-9eb8-f740-3495-64feac1104c5" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ee504eb8-c2a6-e9e0-966b-104adfdefc78" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "f068f3b8-1386-b330-fe8e-1ba9b19a740c" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "f39327f9-cc03-c013-8d40-5d57d2b609db" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: unblocking task api for alloc "34d1328e-8da7-25e8-eee0-989aaf1ec8ca": Template Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: driver.docker: version of executor: 1.1.0 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 6b8c58f9-d5a2-900e-ef6d-ec258fb819d8 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 6c9ff7d7-827e-3e91-3ccc-efe1b62590b8 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 6d00d422-5bcc-0015-571a-5b1bff4e2efe for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 6d88842f-13b6-fc9f-667c-1da28934de23 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 6f3568d9-8b8f-fbe1-27f8-759983b61c00 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 710df41f-94f0-b3a9-ae4b-90e0082f3f69 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 717e8e23-478b-5a89-23db-4991edf8fe37 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7180fee2-c804-7484-ad90-5bb6f8727039 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7214744e-86e6-0e98-d7c7-46564eec83d8 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7405bfcb-6679-661c-28d6-1e74ff100c58 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 756baf79-f842-b57b-e541-56b5010c0972 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 768581b0-7a97-c2dc-4c7a-597e097cab56 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 76c8e63c-15d4-945a-c073-1d290bffce92 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 79f5ed48-6e4e-f5b9-878a-70aa206770a8 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7bfa7846-32ed-c73e-096b-7ec0c6de4b4a for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 89514bb8-2395-3e2d-446c-51dec5eeb9d4 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7d7fd2a3-932d-f644-5478-7e0ba59e16e7 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7d9d1c40-4e30-13ad-505e-1e703fe8ae15 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7dea9772-5e2f-0551-2184-ccf0eb4f6003 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7e88bb46-d31f-88aa-86d1-6c71bbd8d407 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7ec8e849-5a76-0c5a-570d-963d3233727d for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7f8bab5e-e0e7-6d07-26a3-26f0f821e1ab for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 80ff8676-2a91-791b-d217-ea2119647c16 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 83b19aa3-9ec4-72f3-be49-ebd09efe756b for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 84592f6a-a5ea-e44d-e4b9-ea3d95b0e24b for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 861e6f18-d5d4-189e-2974-2dd4308944e9 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 87491d75-e296-15ee-d265-fc2d766a8bc0 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 8856ec7d-f8ac-a58f-9a56-b961e3720bc1 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 92df5c45-bbac-1175-d526-eca9f0641fad for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 8dfc4075-ed7b-d3df-7f11-e30f1c189d10 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 8e5c4942-f826-661b-9461-506b1cc48b89 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 95929532-3909-e568-aca4-362079c7b7b5 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 9618af56-658e-1e10-a45f-13bf23b81a10 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 60bcb176-7853-af47-0c3b-d382600eb35f for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 9bacbac5-f1e9-f19b-6288-269ae56c5b64 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d6f24bdc-b3ad-5aaa-241c-d74bb3191245 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 9bf6f578-3e6f-e10f-3589-8515450f3c83 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d751bf37-e051-42fb-61b4-3c279db8f983 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d779c585-addd-a9e2-df7a-2e253f6a9c2d for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 9c0f45eb-8196-4b4d-6158-e53cd6935ed0 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a0ff8bf3-3d82-c513-002b-189a82f87f98 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a2d72cd6-9876-0d38-8ef6-728f01220902 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a3580421-357c-8b67-fc1a-63f15a508a3a for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a3d8d31c-9104-972d-2dde-05bd2fa27549 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a40bf408-d692-b116-dc5e-979b5511deed for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a46d87cc-0b6a-ed11-7332-ad9cf83e1746 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a4e7aaa8-47e3-2233-acc7-3ecd187d1fa0 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a5777b51-9280-128c-1c72-102e5a5737c9 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a5bc53b9-aa97-f73c-a425-79484e819223 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a65e9103-3651-7a69-d8ee-a94f83c52079 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a8a89a65-1c62-35ea-d5d9-2cf2ec3dc646 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a947549d-2684-b846-3f61-d1df70c710be for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a9f8f4cc-96dd-6bca-cc7d-d819049b0d98 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation aad858f5-32d7-f4eb-1041-434d5218561a for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation ab158044-69a9-9467-d8c9-54b154b194e1 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation aea79f81-4f59-0a83-7319-6edae590f61f for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation af54da3b-6a54-514b-9cad-94d907301aed for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation b1deefd6-79e4-013b-a863-3a3bcb39b5f2 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation b73bba18-671e-5f9d-4217-1c9b2b22f723 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation b5651fa2-fa12-f74d-5c5e-be231a5731ce for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation b97cd23b-354a-c4f2-2753-9252eea00c61 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation ba5f1ce0-2202-245f-5ea3-438651c8ca97 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation ba63d222-db04-e7f7-02b3-2479d10ea7c5 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation bad0bc14-7e5e-0728-9ee9-ab9bb49739dd for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation bce49302-0578-ceaf-4288-645552124ed1 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation bf28e0af-7f35-9258-e946-c48bf94819ce for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation bf5ae2d1-60fd-d040-5321-714198eea793 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: ==> Nomad agent configuration: Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: Client: true Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: Log Level: DEBUG Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: Region: us-east-1 (DC: us-east-1) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: Server: false Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: Version: 0.7.0 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: ==> Nomad agent started! Log data will stream in below: Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.625309 [INFO] client: using state directory /var/nomad/client Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.625584 [INFO] client: using alloc directory /var/nomad/alloc Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.627017 [DEBUG] client: built-in fingerprints: [arch cgroup consul cpu host memory network nomad signal storage vault env_aws env_gce] Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.627770 [INFO] fingerprint.cgroups: cgroups are available Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.628533 [DEBUG] client: fingerprinting cgroup every 15s Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.630171 [INFO] fingerprint.consul: consul agent is available Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.630591 [DEBUG] fingerprint.cpu: frequency: 2400 MHz Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.630809 [DEBUG] fingerprint.cpu: core count: 2 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.632534 [DEBUG] client: fingerprinting consul every 15s Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.633659 [DEBUG] fingerprint.network: link speed for ens3 set to 10000 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.633943 [DEBUG] fingerprint.network: Detected interface ens3 with IP: 10.30.1.43 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.644098 [INFO] fingerprint.vault: Vault is available Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.644543 [DEBUG] client: fingerprinting vault every 15s Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.647351 [WARN]: fingerprint.env_aws: Could not read value for attribute "public-ipv4" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.650979 [WARN]: fingerprint.env_aws: Could not read value for attribute "public-hostname" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.653953 [DEBUG] fingerprint.env_gce: Could not read value for attribute "machine-type" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.654092 [DEBUG] client: applied fingerprints [arch cgroup consul cpu host memory network nomad signal storage vault env_aws] Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.654139 [DEBUG] driver.docker: using client connection initialized from environment Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation bf966954-eb9b-090c-2708-5c8709f3cb50 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.656375 [DEBUG] driver.exec: exec driver is enabled Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.656475 [DEBUG] client: available drivers [docker exec] Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.656499 [DEBUG] client: drivers skipped due to white/blacklist: [raw_exec qemu] Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.656856 [DEBUG] client: fingerprinting docker every 15s Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.656892 [DEBUG] client: fingerprinting exec every 15s Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.656929 [DEBUG] client: fingerprinting rkt every 15s Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.658521 [DEBUG] client.consul: bootstrap contacting following Consul DCs: ["us-east-1"] Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.659825 [INFO] driver.docker: re-attaching to docker process: 6e9a311e11251f04cf92c9bde5dc460ba22fffaeca0b782e284fcda73a55b818 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.660421 [DEBUG] driver.docker: re-attached to handle: DOCKER:{"Version":"0.7.0","Image":"symbiosis:1.7.0-SNAPSHOT","ImageID":"sha256:d4b632ec32d3b55198fe95af7fc2347b7ef7e7cabfc36f73a778b93d0e619824","ContainerID":"6e9a311e11251f04cf92c9bde5dc460ba22fffaeca0b782e284fcda73a55b818","KillTimeout":5000000000,"MaxKillTimeout":30000000000,"PluginConfig":{"Pid":21963,"AddrNet":"unix","AddrName":"/tmp/plugin612029439"}} Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.661211 [DEBUG] client: alloc "001d90fa-f585-e05a-c9a9-e692dd8be47a" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.661402 [INFO] client: marking allocation 001d90fa-f585-e05a-c9a9-e692dd8be47a for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.661784 [DEBUG] client: alloc "015b6bd1-21a8-3b19-c9ac-8fe6ef444c6d" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.661930 [INFO] client: marking allocation 015b6bd1-21a8-3b19-c9ac-8fe6ef444c6d for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.662617 [DEBUG] client: alloc "0360f079-e7dc-3c0c-e57d-9e9a481929ea" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.662774 [INFO] client: marking allocation 0360f079-e7dc-3c0c-e57d-9e9a481929ea for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation c03b42b7-ff48-189b-0e2b-f57e3888bb0d for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.663308 [INFO] client.consul: discovered following Servers: 10.50.1.95:4647,10.50.1.29:4647,10.50.1.101:4647 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.663340 [DEBUG] client: alloc "062e4938-ddad-0fe6-8a34-36c89639bc4d" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.665772 [INFO] client: marking allocation 062e4938-ddad-0fe6-8a34-36c89639bc4d for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.663532 [DEBUG] client: alloc "0440a31d-14d9-3b0d-73e9-1a4f2f274023" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.663785 [DEBUG] client: alloc "05412a3d-6609-646a-6bc2-89947a18f919" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.667155 [INFO] client: marking allocation 05412a3d-6609-646a-6bc2-89947a18f919 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.666176 [DEBUG] client: alloc "03f5f08f-1f8d-66f7-e28a-22f22465694d" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.667739 [INFO] client: marking allocation 03f5f08f-1f8d-66f7-e28a-22f22465694d for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.666728 [INFO] client: marking allocation 0440a31d-14d9-3b0d-73e9-1a4f2f274023 for GC Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.668817 [DEBUG] driver.docker: version of executor: 1.1.0 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.668858 [DEBUG] driver.docker: image "symbiosis:1.7.0-SNAPSHOT" (sha256:d4b632ec32d3b55198fe95af7fc2347b7ef7e7cabfc36f73a778b93d0e619824) reference count incremented: 1 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.670010 [DEBUG] client: starting task context for 'api' (alloc '06540f5a-3a84-7ba7-e2f7-cf403a0fd43e') Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.671949 [DEBUG] client: waiting for Vault token for task api in alloc "06540f5a-3a84-7ba7-e2f7-cf403a0fd43e" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.672792 [WARN] client: error fetching stats of task api: stats collection hasn't started yet Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:07.675821 [DEBUG] client: alloc "093d27b4-3a9e-3307-b1f3-f9e5eae4d878" in terminal status, waiting for destroy Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: error fetching stats of task api: stats collection hasn't started yet Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [INFO] (runner) rendered "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/tricorder.uat.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/tricorder.conf" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) checking template 0d345435336ca507d95eac9d44730182 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendered "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/tricorder.uat.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/tricorder.conf" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0d345435336ca507d95eac9d44730182 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [INFO] (runner) rendered "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/ingestion.uat.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/ingestion.conf" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) checking template 3f899786697a252f914e48c385b3fac5 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendered "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/ingestion.uat.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/ingestion.conf" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 3f899786697a252f914e48c385b3fac5 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) rendering "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/secrets.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/secrets.conf" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/secrets.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/secrets.conf" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) checking template d74433fe4620cb7647d82eaf4f8e7835 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template d74433fe4620cb7647d82eaf4f8e7835 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) rendering "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/filebeat.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/filebeat.yml" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/filebeat.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/filebeat.yml" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) checking template 362d96b61c7729f34665b804cc36fefe Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 362d96b61c7729f34665b804cc36fefe Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) rendering "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/packetbeat.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/packetbeat.yml" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/packetbeat.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/packetbeat.yml" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) diffing and updating dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies [snip] [snip] Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) watching 29 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) all templates rendered Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08.589761 [DEBUG] client: unblocking task api for alloc "5d3fbe30-29a0-8245-b179-6f317fd6b35e": Template Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) rendering "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/secrets.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/secrets.conf" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) checking template de2638ff63506a18feeccea5c5b770a0 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) rendering "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/filebeat.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/filebeat.yml" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) checking template 20d663d8d756b7418c40e5e6c095cba2 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) rendering "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/packetbeat.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/packetbeat.yml" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) diffing and updating dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) watching 36 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08 [DEBUG] (runner) all templates rendered Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08.603922 [DEBUG] client: unblocking task api for alloc "fa37f5c3-e340-0a1c-81ab-ed90d142de75": Template Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08.605626 [DEBUG] consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 29 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) all templates rendered Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: unblocking task api for alloc "5d3fbe30-29a0-8245-b179-6f317fd6b35e": Template Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/secrets.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/secrets.conf" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08.613266 [DEBUG] http: Request /v1/status/leader (29.942089ms) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template de2638ff63506a18feeccea5c5b770a0 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/filebeat.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/filebeat.yml" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 20d663d8d756b7418c40e5e6c095cba2 Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/packetbeat.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/packetbeat.yml" Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 36 dependencies Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) all templates rendered Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: unblocking task api for alloc "fa37f5c3-e340-0a1c-81ab-ed90d142de75": Template Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: http: Request /v1/status/leader (29.942089ms) Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08.631187 [DEBUG] consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08.640709 [DEBUG] client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039' Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08.640805 [DEBUG] client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039' Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039' Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039' Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08.649242 [DEBUG] consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08.665178 [DEBUG] consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:08.820070 [DEBUG] consul.sync: registered 0 services, 0 checks; deregistered 0 services, 0 checks Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: consul.sync: registered 0 services, 0 checks; deregistered 0 services, 0 checks Dec 13 16:59:16 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:16.744385 [DEBUG] http: Request /v1/agent/health?type=client (458.142µs) Dec 13 16:59:16 ip-10-30-1-43 nomad[29064]: http: Request /v1/agent/health?type=client (458.142µs) Dec 13 16:59:16 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:16.978960 [DEBUG] client: 34 evaluations triggered by node update Dec 13 16:59:16 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:16.979022 [DEBUG] client: state updated to ready Dec 13 16:59:16 ip-10-30-1-43 nomad[29064]: client: 34 evaluations triggered by node update Dec 13 16:59:16 ip-10-30-1-43 nomad[29064]: client: state updated to ready Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:17.129672 [DEBUG] client: updated allocations at index 605006 (total 253) (pulled 3) (filtered 250) Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:17.129982 [DEBUG] client: allocs: (added 0) (removed 0) (updated 3) (ignore 250) Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:17.130055 [DEBUG] client: dropping update to terminal alloc '0b1e29f2-976e-1c85-2ab5-0689915a6244' Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:17.130081 [DEBUG] client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039' Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:17.130102 [DEBUG] client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565' Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: updated allocations at index 605006 (total 253) (pulled 3) (filtered 250) Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: allocs: (added 0) (removed 0) (updated 3) (ignore 250) Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '0b1e29f2-976e-1c85-2ab5-0689915a6244' Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039' Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565' Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:17.285999 [DEBUG] client: updated allocations at index 605038 (total 253) (pulled 3) (filtered 250) Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:17.286431 [DEBUG] client: allocs: (added 0) (removed 0) (updated 3) (ignore 250) Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: updated allocations at index 605038 (total 253) (pulled 3) (filtered 250) Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:17.286569 [DEBUG] client: dropping update to terminal alloc '0b1e29f2-976e-1c85-2ab5-0689915a6244' Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:17.286602 [DEBUG] client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039' Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:17.286629 [DEBUG] client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565' Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: allocs: (added 0) (removed 0) (updated 3) (ignore 250) Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '0b1e29f2-976e-1c85-2ab5-0689915a6244' Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039' Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565' Dec 13 16:59:23 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:23.615107 [DEBUG] http: Request /v1/status/leader (914.296µs) Dec 13 16:59:23 ip-10-30-1-43 nomad[29064]: http: Request /v1/status/leader (914.296µs) Dec 13 16:59:26 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:26.745767 [DEBUG] http: Request /v1/agent/health?type=client (257.378µs) Dec 13 16:59:26 ip-10-30-1-43 nomad[29064]: http: Request /v1/agent/health?type=client (257.378µs) Dec 13 16:59:36 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:36.747088 [DEBUG] http: Request /v1/agent/health?type=client (242.798µs) Dec 13 16:59:36 ip-10-30-1-43 nomad[29064]: http: Request /v1/agent/health?type=client (242.798µs) Dec 13 16:59:38 ip-10-30-1-43 nomad[29064]: 2017/12/13 16:59:38.616545 [DEBUG] http: Request /v1/status/leader (508.628µs) Dec 13 16:59:38 ip-10-30-1-43 nomad[29064]: http: Request /v1/status/leader (508.628µs) ```

Job file (if appropriate)

Would you be able to test Nomad 0.7.1-rc1 that was just released Monday? Link to Linux amd64 binary: https://releases.hashicorp.com/nomad/0.7.1-rc1/nomad_0.7.1-rc1_linux_amd64.zip Release notes: https://groups.google.com/d/msg/nomad-tool/-aUbGM2_ou0/FMawwLq1AgAJ

I believe #3445 fixes your issue.

@apenney

Of those jobs how many are dead?
Can you do a curl -XPUT http://127.0.0.1:4646/v1/system/gc and collect the logs from all the servers.
For a job that is dead and you believe should be garbage collected can you collect: curl http://127.0.0.1:4646/v1/job/<job>/allocations

0.7.1 doesn't seem to have helped @schmichael, I'm still in the same situation.

1: Most of them are dead:

root@ip-10-30-1-43:/var/nomad/alloc# nomad job status | grep dead | wc -l
167

Here's the logs from the 0.7.1-rc1 box:

``` Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.423133 [DEBUG] client: alloc "8d60c99e-b719-6abb-0a3d-b004b08f069c" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "8d60c99e-b719-6abb-0a3d-b004b08f069c" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.436503 [DEBUG] client: alloc "97d494a3-c113-ca17-8be2-2f1fb4438f06" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "97d494a3-c113-ca17-8be2-2f1fb4438f06" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 8d60c99e-b719-6abb-0a3d-b004b08f069c for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.436824 [INFO] client.gc: marking allocation 8d60c99e-b719-6abb-0a3d-b004b08f069c for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.449583 [INFO] client.gc: marking allocation 97d494a3-c113-ca17-8be2-2f1fb4438f06 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 97d494a3-c113-ca17-8be2-2f1fb4438f06 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.450270 [DEBUG] client: alloc "4f7cbfbd-f4b1-6644-a561-973d9aae0128" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "4f7cbfbd-f4b1-6644-a561-973d9aae0128" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.473093 [INFO] client.gc: marking allocation 4f7cbfbd-f4b1-6644-a561-973d9aae0128 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 4f7cbfbd-f4b1-6644-a561-973d9aae0128 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.474111 [DEBUG] client: alloc "aad858f5-32d7-f4eb-1041-434d5218561a" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "aad858f5-32d7-f4eb-1041-434d5218561a" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.486222 [INFO] client.gc: marking allocation aad858f5-32d7-f4eb-1041-434d5218561a for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation aad858f5-32d7-f4eb-1041-434d5218561a for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.487075 [DEBUG] client: alloc "454db4b9-733a-1211-bb5c-370cb8a4bcc1" waiting for remote previous alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" to terminate Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "454db4b9-733a-1211-bb5c-370cb8a4bcc1" waiting for remote previous alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" to terminate Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.488876 [DEBUG] client: alloc "454db4b9-733a-1211-bb5c-370cb8a4bcc1" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "454db4b9-733a-1211-bb5c-370cb8a4bcc1" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.523354 [INFO] client.gc: marking allocation 454db4b9-733a-1211-bb5c-370cb8a4bcc1 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 454db4b9-733a-1211-bb5c-370cb8a4bcc1 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.524139 [DEBUG] client: alloc "60bcb176-7853-af47-0c3b-d382600eb35f" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "60bcb176-7853-af47-0c3b-d382600eb35f" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.537929 [DEBUG] client: alloc "76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca" waiting for remote previous alloc "403baf3c-00a2-6e5d-5f14-88fcd4cc688d" to terminate Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca" waiting for remote previous alloc "403baf3c-00a2-6e5d-5f14-88fcd4cc688d" to terminate Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.538940 [DEBUG] client: alloc "76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.543500 [DEBUG] client: updated allocations at index 607150 (total 260) (pulled 149) (filtered 111) Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: updated allocations at index 607150 (total 260) (pulled 149) (filtered 111) Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.549697 [INFO] client.gc: marking allocation 76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.549881 [INFO] client.gc: marking allocation 60bcb176-7853-af47-0c3b-d382600eb35f for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 60bcb176-7853-af47-0c3b-d382600eb35f for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.550959 [DEBUG] client: alloc "c03b42b7-ff48-189b-0e2b-f57e3888bb0d" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "c03b42b7-ff48-189b-0e2b-f57e3888bb0d" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.563213 [INFO] client.gc: marking allocation c03b42b7-ff48-189b-0e2b-f57e3888bb0d for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation c03b42b7-ff48-189b-0e2b-f57e3888bb0d for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.563757 [DEBUG] client: alloc "65f06722-772a-8c48-d2e1-183d68307535" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "65f06722-772a-8c48-d2e1-183d68307535" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.576996 [INFO] client.gc: marking allocation 65f06722-772a-8c48-d2e1-183d68307535 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 65f06722-772a-8c48-d2e1-183d68307535 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.577865 [DEBUG] client: alloc "7bfa7846-32ed-c73e-096b-7ec0c6de4b4a" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "7bfa7846-32ed-c73e-096b-7ec0c6de4b4a" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.591752 [INFO] client.gc: marking allocation 7bfa7846-32ed-c73e-096b-7ec0c6de4b4a for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 7bfa7846-32ed-c73e-096b-7ec0c6de4b4a for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.592901 [DEBUG] client: alloc "c17c6e42-1445-27b2-aa83-d38a9f0b788b" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "c17c6e42-1445-27b2-aa83-d38a9f0b788b" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.605541 [INFO] client.gc: marking allocation c17c6e42-1445-27b2-aa83-d38a9f0b788b for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation c17c6e42-1445-27b2-aa83-d38a9f0b788b for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.605839 [DEBUG] client: alloc "cf95d8b3-5847-e53d-9908-cef313e624c4" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "cf95d8b3-5847-e53d-9908-cef313e624c4" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.618239 [INFO] client.gc: marking allocation cf95d8b3-5847-e53d-9908-cef313e624c4 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation cf95d8b3-5847-e53d-9908-cef313e624c4 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.619272 [DEBUG] client: alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" waiting for remote previous alloc "756baf79-f842-b57b-e541-56b5010c0972" to terminate Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" waiting for remote previous alloc "756baf79-f842-b57b-e541-56b5010c0972" to terminate Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.620149 [DEBUG] client: alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.633089 [INFO] client.gc: marking allocation a0ff8bf3-3d82-c513-002b-189a82f87f98 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation a0ff8bf3-3d82-c513-002b-189a82f87f98 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.633913 [DEBUG] client: alloc "89a880f4-2f7a-3b44-c1e4-2f8c22b89179" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "89a880f4-2f7a-3b44-c1e4-2f8c22b89179" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.647838 [INFO] client.gc: marking allocation 89a880f4-2f7a-3b44-c1e4-2f8c22b89179 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.648249 [DEBUG] client: alloc "74b37c99-5f20-e968-b05a-3642ade44ae9" waiting for remote previous alloc "fa37f5c3-e340-0a1c-81ab-ed90d142de75" to terminate Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 89a880f4-2f7a-3b44-c1e4-2f8c22b89179 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "74b37c99-5f20-e968-b05a-3642ade44ae9" waiting for remote previous alloc "fa37f5c3-e340-0a1c-81ab-ed90d142de75" to terminate Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.649543 [DEBUG] client: alloc "74b37c99-5f20-e968-b05a-3642ade44ae9" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "74b37c99-5f20-e968-b05a-3642ade44ae9" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.661565 [INFO] client.gc: marking allocation 74b37c99-5f20-e968-b05a-3642ade44ae9 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 74b37c99-5f20-e968-b05a-3642ade44ae9 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.662386 [DEBUG] client: alloc "5162b0ed-a687-ac1d-4f30-33406acaac63" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "5162b0ed-a687-ac1d-4f30-33406acaac63" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.674501 [INFO] client.gc: marking allocation 5162b0ed-a687-ac1d-4f30-33406acaac63 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 5162b0ed-a687-ac1d-4f30-33406acaac63 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.674834 [DEBUG] client: alloc "7dea9772-5e2f-0551-2184-ccf0eb4f6003" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "7dea9772-5e2f-0551-2184-ccf0eb4f6003" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.686990 [INFO] client.gc: marking allocation 7dea9772-5e2f-0551-2184-ccf0eb4f6003 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 7dea9772-5e2f-0551-2184-ccf0eb4f6003 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.687898 [DEBUG] client: alloc "8ec810f6-e606-9e14-51a6-2de2b1dbf28f" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "8ec810f6-e606-9e14-51a6-2de2b1dbf28f" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.699946 [INFO] client.gc: marking allocation 8ec810f6-e606-9e14-51a6-2de2b1dbf28f for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 8ec810f6-e606-9e14-51a6-2de2b1dbf28f for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.700405 [DEBUG] client: alloc "2f531a29-431e-4d31-451b-d08c84cc2176" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "2f531a29-431e-4d31-451b-d08c84cc2176" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.713843 [INFO] client.gc: marking allocation 2f531a29-431e-4d31-451b-d08c84cc2176 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 2f531a29-431e-4d31-451b-d08c84cc2176 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.714922 [DEBUG] client: alloc "d62bec89-0c99-4a20-a47d-c4cd275eb355" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "d62bec89-0c99-4a20-a47d-c4cd275eb355" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.727722 [INFO] client.gc: marking allocation d62bec89-0c99-4a20-a47d-c4cd275eb355 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation d62bec89-0c99-4a20-a47d-c4cd275eb355 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.728291 [DEBUG] client: alloc "5c6aff2d-f11f-f18e-fef6-b8e92a13c167" waiting for remote previous alloc "dd0cfd72-3a94-a9b5-1afc-320b3388499f" to terminate Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "5c6aff2d-f11f-f18e-fef6-b8e92a13c167" waiting for remote previous alloc "dd0cfd72-3a94-a9b5-1afc-320b3388499f" to terminate Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.729906 [DEBUG] client: alloc "5c6aff2d-f11f-f18e-fef6-b8e92a13c167" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "5c6aff2d-f11f-f18e-fef6-b8e92a13c167" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.747433 [INFO] client.gc: marking allocation 5c6aff2d-f11f-f18e-fef6-b8e92a13c167 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 5c6aff2d-f11f-f18e-fef6-b8e92a13c167 for GC Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.747842 [DEBUG] client: alloc "8e895f9b-d99b-c4ee-a214-9e2f4db5c8d7" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "8e895f9b-d99b-c4ee-a214-9e2f4db5c8d7" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.759880 [DEBUG] client: updated allocations at index 607151 (total 260) (pulled 133) (filtered 127) Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: updated allocations at index 607151 (total 260) (pulled 133) (filtered 127) Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:19.761176 [DEBUG] client: alloc "58927b32-e1f9-6628-2bb6-b8192366c032" in terminal status, waiting for destroy Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "58927b32-e1f9-6628-2bb6-b8192366c032" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.456301 [DEBUG] client: 34 evaluations triggered by node update Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.456421 [DEBUG] client: state updated to ready Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: 34 evaluations triggered by node update Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: state updated to ready Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.733452 [INFO] client.gc: marking allocation 8e895f9b-d99b-c4ee-a214-9e2f4db5c8d7 for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 8e895f9b-d99b-c4ee-a214-9e2f4db5c8d7 for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.734360 [INFO] client.gc: marking allocation 58927b32-e1f9-6628-2bb6-b8192366c032 for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 58927b32-e1f9-6628-2bb6-b8192366c032 for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.735073 [DEBUG] client: alloc "3ebb8609-0a0f-22ec-da55-091301002b03" waiting for remote previous alloc "03f5f08f-1f8d-66f7-e28a-22f22465694d" to terminate Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "3ebb8609-0a0f-22ec-da55-091301002b03" waiting for remote previous alloc "03f5f08f-1f8d-66f7-e28a-22f22465694d" to terminate Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.736350 [DEBUG] client: alloc "3ebb8609-0a0f-22ec-da55-091301002b03" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "3ebb8609-0a0f-22ec-da55-091301002b03" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.751440 [INFO] client.gc: marking allocation 3ebb8609-0a0f-22ec-da55-091301002b03 for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 3ebb8609-0a0f-22ec-da55-091301002b03 for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.752284 [DEBUG] client: alloc "c2871eff-20c1-15dd-03f8-1f75a7f49aca" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "c2871eff-20c1-15dd-03f8-1f75a7f49aca" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.794053 [INFO] client.gc: marking allocation c2871eff-20c1-15dd-03f8-1f75a7f49aca for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation c2871eff-20c1-15dd-03f8-1f75a7f49aca for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.794479 [DEBUG] client: alloc "3b983e9d-f6d1-7f0c-8d0e-e1209abd0bea" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "3b983e9d-f6d1-7f0c-8d0e-e1209abd0bea" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.795495 [DEBUG] client: updated allocations at index 607152 (total 260) (pulled 130) (filtered 130) Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: updated allocations at index 607152 (total 260) (pulled 130) (filtered 130) Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.830706 [DEBUG] client: alloc "af54da3b-6a54-514b-9cad-94d907301aed" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "af54da3b-6a54-514b-9cad-94d907301aed" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.843986 [INFO] client.gc: marking allocation af54da3b-6a54-514b-9cad-94d907301aed for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation af54da3b-6a54-514b-9cad-94d907301aed for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.845132 [DEBUG] client: alloc "23251e63-074b-59d9-80ef-310fce801d42" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "23251e63-074b-59d9-80ef-310fce801d42" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.857386 [INFO] client.gc: marking allocation 23251e63-074b-59d9-80ef-310fce801d42 for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 23251e63-074b-59d9-80ef-310fce801d42 for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.858984 [DEBUG] client: alloc "ba63d222-db04-e7f7-02b3-2479d10ea7c5" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "ba63d222-db04-e7f7-02b3-2479d10ea7c5" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.864374 [DEBUG] client: updated allocations at index 607204 (total 260) (pulled 127) (filtered 133) Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: updated allocations at index 607204 (total 260) (pulled 127) (filtered 133) Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.871166 [INFO] client.gc: marking allocation 3b983e9d-f6d1-7f0c-8d0e-e1209abd0bea for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 3b983e9d-f6d1-7f0c-8d0e-e1209abd0bea for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.871939 [INFO] client.gc: marking allocation ba63d222-db04-e7f7-02b3-2479d10ea7c5 for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.871855 [DEBUG] client: alloc "dd0cfd72-3a94-a9b5-1afc-320b3388499f" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation ba63d222-db04-e7f7-02b3-2479d10ea7c5 for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "dd0cfd72-3a94-a9b5-1afc-320b3388499f" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.885526 [INFO] client.gc: marking allocation dd0cfd72-3a94-a9b5-1afc-320b3388499f for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation dd0cfd72-3a94-a9b5-1afc-320b3388499f for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.885815 [DEBUG] client: alloc "50d7116e-808b-4ae0-f6f2-5be1a06ad746" waiting for remote previous alloc "4a594eca-8a63-acbe-99e9-11c2673c2519" to terminate Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "50d7116e-808b-4ae0-f6f2-5be1a06ad746" waiting for remote previous alloc "4a594eca-8a63-acbe-99e9-11c2673c2519" to terminate Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.887516 [DEBUG] client: alloc "50d7116e-808b-4ae0-f6f2-5be1a06ad746" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "50d7116e-808b-4ae0-f6f2-5be1a06ad746" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.898990 [INFO] client.gc: marking allocation 50d7116e-808b-4ae0-f6f2-5be1a06ad746 for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 50d7116e-808b-4ae0-f6f2-5be1a06ad746 for GC Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.900023 [DEBUG] client: alloc "de9cabd7-4b85-152a-da52-13706bb01fd6" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "de9cabd7-4b85-152a-da52-13706bb01fd6" in terminal status, waiting for destroy Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:20.905435 [DEBUG] client: updated allocations at index 607216 (total 260) (pulled 124) (filtered 136) Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: updated allocations at index 607216 (total 260) (pulled 124) (filtered 136) Dec 13 19:35:34 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:34.169026 [DEBUG] http: Request /v1/agent/health?type=client (267.633µs) Dec 13 19:35:34 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (267.633µs) Dec 13 19:35:40 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:40.257261 [DEBUG] http: Request /v1/status/leader (8.952714ms) Dec 13 19:35:40 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (8.952714ms) Dec 13 19:35:44 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:44.174416 [DEBUG] http: Request /v1/agent/health?type=client (4.448616ms) Dec 13 19:35:44 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (4.448616ms) Dec 13 19:35:47 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:47.858337 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (3.829894ms) Dec 13 19:35:47 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (3.829894ms) Dec 13 19:35:54 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:54.176002 [DEBUG] http: Request /v1/agent/health?type=client (241.173µs) Dec 13 19:35:54 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (241.173µs) Dec 13 19:35:55 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:55.259426 [DEBUG] http: Request /v1/status/leader (547.139µs) Dec 13 19:35:55 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (547.139µs) Dec 13 19:35:58 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:58.018504 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (245.288µs) Dec 13 19:35:58 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (245.288µs) Dec 13 19:35:58 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:35:58.037559 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (349.099µs) Dec 13 19:35:58 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (349.099µs) Dec 13 19:36:04 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:36:04.177455 [DEBUG] http: Request /v1/agent/health?type=client (256.83µs) Dec 13 19:36:04 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (256.83µs) Dec 13 19:36:10 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:36:10.261140 [DEBUG] http: Request /v1/status/leader (564.829µs) Dec 13 19:36:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (564.829µs) Dec 13 19:36:14 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:36:14.178673 [DEBUG] http: Request /v1/agent/health?type=client (216.296µs) Dec 13 19:36:14 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (216.296µs) Dec 13 19:36:24 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:36:24.189967 [DEBUG] http: Request /v1/agent/health?type=client (10.144846ms) Dec 13 19:36:24 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (10.144846ms) Dec 13 19:36:25 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:36:25.262840 [DEBUG] http: Request /v1/status/leader (551.843µs) Dec 13 19:36:25 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (551.843µs) Dec 13 19:36:34 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:36:34.191218 [DEBUG] http: Request /v1/agent/health?type=client (242.978µs) Dec 13 19:36:34 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (242.978µs) Dec 13 19:36:40 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:36:40.264611 [DEBUG] http: Request /v1/status/leader (474.955µs) Dec 13 19:36:40 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (474.955µs) Dec 13 19:36:44 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:36:44.192342 [DEBUG] http: Request /v1/agent/health?type=client (217.45µs) Dec 13 19:36:44 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (217.45µs) Dec 13 19:36:54 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:36:54.193885 [DEBUG] http: Request /v1/agent/health?type=client (267.08µs) Dec 13 19:36:54 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (267.08µs) Dec 13 19:36:55 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:36:55.266656 [DEBUG] http: Request /v1/status/leader (689.019µs) Dec 13 19:36:55 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (689.019µs) Dec 13 19:37:04 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:37:04.195508 [DEBUG] http: Request /v1/agent/health?type=client (283.838µs) Dec 13 19:37:04 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (283.838µs) Dec 13 19:37:10 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:37:10.268937 [DEBUG] http: Request /v1/status/leader (965.641µs) Dec 13 19:37:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (965.641µs) Dec 13 19:37:14 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:37:14.197216 [DEBUG] http: Request /v1/agent/health?type=client (316.67µs) Dec 13 19:37:14 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (316.67µs) Dec 13 19:37:18 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:37:18.463994 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (216.193µs) Dec 13 19:37:18 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (216.193µs) Dec 13 19:37:24 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:37:24.203260 [DEBUG] http: Request /v1/agent/health?type=client (4.790514ms) Dec 13 19:37:24 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (4.790514ms) Dec 13 19:37:25 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:37:25.270538 [DEBUG] http: Request /v1/status/leader (503.546µs) Dec 13 19:37:25 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (503.546µs) Dec 13 19:37:29 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:37:29.099502 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (307.012µs) Dec 13 19:37:29 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (307.012µs) Dec 13 19:37:29 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:37:29.128447 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (174.287µs) Dec 13 19:37:29 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (174.287µs) Dec 13 19:37:34 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:37:34.208357 [DEBUG] http: Request /v1/agent/health?type=client (228.085µs) Dec 13 19:37:34 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (228.085µs) Dec 13 19:37:40 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:37:40.272654 [DEBUG] http: Request /v1/status/leader (955.632µs) Dec 13 19:37:40 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (955.632µs) Dec 13 19:37:44 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:37:44.209568 [DEBUG] http: Request /v1/agent/health?type=client (234.668µs) Dec 13 19:37:44 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (234.668µs) Dec 13 19:37:54 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:37:54.210789 [DEBUG] http: Request /v1/agent/health?type=client (247.035µs) Dec 13 19:37:54 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (247.035µs) Dec 13 19:37:55 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:37:55.274748 [DEBUG] http: Request /v1/status/leader (904.08µs) Dec 13 19:37:55 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (904.08µs) Dec 13 19:38:04 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:04.212259 [DEBUG] http: Request /v1/agent/health?type=client (201.362µs) Dec 13 19:38:04 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (201.362µs) Dec 13 19:38:10 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:10.277078 [DEBUG] http: Request /v1/status/leader (975.083µs) Dec 13 19:38:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (975.083µs) Dec 13 19:38:14 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:14.213465 [DEBUG] http: Request /v1/agent/health?type=client (220.513µs) Dec 13 19:38:14 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (220.513µs) Dec 13 19:38:16 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:16.307316 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (229.282µs) Dec 13 19:38:16 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (229.282µs) Dec 13 19:38:23 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:23.631207 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (219.83µs) Dec 13 19:38:23 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (219.83µs) Dec 13 19:38:24 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:24.214647 [DEBUG] http: Request /v1/agent/health?type=client (198.392µs) Dec 13 19:38:24 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (198.392µs) Dec 13 19:38:25 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:25.278952 [DEBUG] http: Request /v1/status/leader (611.382µs) Dec 13 19:38:25 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (611.382µs) Dec 13 19:38:26 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:26.609274 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (309.496µs) Dec 13 19:38:26 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (309.496µs) Dec 13 19:38:26 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:26.626161 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (210.562µs) Dec 13 19:38:26 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (210.562µs) Dec 13 19:38:33 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:33.698901 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (247.474µs) Dec 13 19:38:33 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (247.474µs) Dec 13 19:38:33 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:33.722253 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (89.375µs) Dec 13 19:38:33 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (89.375µs) Dec 13 19:38:34 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:34.215804 [DEBUG] http: Request /v1/agent/health?type=client (157.479µs) Dec 13 19:38:34 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (157.479µs) Dec 13 19:38:40 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:40.280577 [DEBUG] http: Request /v1/status/leader (514.259µs) Dec 13 19:38:40 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (514.259µs) Dec 13 19:38:44 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:44.216929 [DEBUG] http: Request /v1/agent/health?type=client (234.622µs) Dec 13 19:38:44 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (234.622µs) Dec 13 19:38:49 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:49.561948 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (262.264µs) Dec 13 19:38:49 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (262.264µs) Dec 13 19:38:54 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:54.218095 [DEBUG] http: Request /v1/agent/health?type=client (213.49µs) Dec 13 19:38:54 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (213.49µs) Dec 13 19:38:55 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:55.282517 [DEBUG] http: Request /v1/status/leader (612.803µs) Dec 13 19:38:55 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (612.803µs) Dec 13 19:38:59 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:59.926153 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (276.397µs) Dec 13 19:38:59 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (276.397µs) Dec 13 19:38:59 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:38:59.952674 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (249.274µs) Dec 13 19:38:59 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (249.274µs) Dec 13 19:39:04 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:04.219420 [DEBUG] http: Request /v1/agent/health?type=client (230.915µs) Dec 13 19:39:04 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (230.915µs) Dec 13 19:39:10 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:10.106300 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (380.693µs) Dec 13 19:39:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (380.693µs) Dec 13 19:39:10 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:10.124885 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (106.49µs) Dec 13 19:39:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (106.49µs) Dec 13 19:39:10 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:10.289128 [DEBUG] http: Request /v1/status/leader (5.290217ms) Dec 13 19:39:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (5.290217ms) Dec 13 19:39:14 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:14.220878 [DEBUG] http: Request /v1/agent/health?type=client (319.849µs) Dec 13 19:39:14 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (319.849µs) Dec 13 19:39:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:20.581199 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (296.461µs) Dec 13 19:39:20 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (296.461µs) Dec 13 19:39:20 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:20.598649 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (132.847µs) Dec 13 19:39:20 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (132.847µs) Dec 13 19:39:24 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:24.222382 [DEBUG] http: Request /v1/agent/health?type=client (228.187µs) Dec 13 19:39:24 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (228.187µs) Dec 13 19:39:25 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:25.291681 [DEBUG] http: Request /v1/status/leader (1.511535ms) Dec 13 19:39:25 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (1.511535ms) Dec 13 19:39:25 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:25.668224 [DEBUG] http: Request /v1/system/gc (693.903µs) Dec 13 19:39:25 ip-10-30-1-43 nomad[15209]: http: Request /v1/system/gc (693.903µs) Dec 13 19:39:30 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:30.836928 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (231.916µs) Dec 13 19:39:30 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (231.916µs) Dec 13 19:39:30 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:30.858899 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (231.254µs) Dec 13 19:39:30 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (231.254µs) Dec 13 19:39:34 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:34.223638 [DEBUG] http: Request /v1/agent/health?type=client (214.667µs) Dec 13 19:39:34 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (214.667µs) Dec 13 19:39:40 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:40.296374 [DEBUG] http: Request /v1/status/leader (535.247µs) Dec 13 19:39:40 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (535.247µs) Dec 13 19:39:41 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:41.341266 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (248.522µs) Dec 13 19:39:41 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (248.522µs) Dec 13 19:39:41 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:41.356719 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (287.126µs) Dec 13 19:39:41 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (287.126µs) Dec 13 19:39:44 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:44.224862 [DEBUG] http: Request /v1/agent/health?type=client (219.14µs) Dec 13 19:39:44 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (219.14µs) Dec 13 19:39:51 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:51.568582 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (4.993536ms) Dec 13 19:39:51 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (4.993536ms) Dec 13 19:39:51 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:51.593021 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (128.688µs) Dec 13 19:39:51 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (128.688µs) Dec 13 19:39:54 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:54.226374 [DEBUG] http: Request /v1/agent/health?type=client (223.71µs) Dec 13 19:39:54 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (223.71µs) Dec 13 19:39:55 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:39:55.304475 [DEBUG] http: Request /v1/status/leader (560.34µs) Dec 13 19:39:55 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (560.34µs) Dec 13 19:40:00 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:00.790799 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (227.27µs) Dec 13 19:40:00 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (227.27µs) Dec 13 19:40:02 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:02.175335 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (229.492µs) Dec 13 19:40:02 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (229.492µs) Dec 13 19:40:02 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:02.215937 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (173.094µs) Dec 13 19:40:02 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (173.094µs) Dec 13 19:40:02 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:02.897214 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (242.247µs) Dec 13 19:40:02 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (242.247µs) Dec 13 19:40:04 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:04.227936 [DEBUG] http: Request /v1/agent/health?type=client (217.322µs) Dec 13 19:40:04 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (217.322µs) Dec 13 19:40:10 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:10.311347 [DEBUG] http: Request /v1/status/leader (5.536999ms) Dec 13 19:40:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (5.536999ms) Dec 13 19:40:11 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:11.397282 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (175.538µs) Dec 13 19:40:11 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (175.538µs) Dec 13 19:40:11 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:11.421125 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (207.155µs) Dec 13 19:40:11 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (207.155µs) Dec 13 19:40:13 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:13.535315 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (257.324µs) Dec 13 19:40:13 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (257.324µs) Dec 13 19:40:13 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:13.562052 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (227.65µs) Dec 13 19:40:13 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (227.65µs) Dec 13 19:40:14 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:14.230066 [DEBUG] http: Request /v1/agent/health?type=client (274.836µs) Dec 13 19:40:14 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (274.836µs) Dec 13 19:40:18 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:18.259910 [DEBUG] http: Request /v1/jobs (10.676546ms) Dec 13 19:40:18 ip-10-30-1-43 nomad[15209]: http: Request /v1/jobs (10.676546ms) Dec 13 19:40:19 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:19.979375 [DEBUG] http: Request /v1/jobs (10.359401ms) Dec 13 19:40:19 ip-10-30-1-43 nomad[15209]: http: Request /v1/jobs (10.359401ms) Dec 13 19:40:23 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:23.321174 [DEBUG] http: Request /v1/jobs (7.162803ms) Dec 13 19:40:23 ip-10-30-1-43 nomad[15209]: http: Request /v1/jobs (7.162803ms) Dec 13 19:40:24 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:24.231753 [DEBUG] http: Request /v1/agent/health?type=client (162.462µs) Dec 13 19:40:24 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (162.462µs) Dec 13 19:40:25 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:25.313501 [DEBUG] http: Request /v1/status/leader (652.374µs) Dec 13 19:40:25 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (652.374µs) Dec 13 19:40:34 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:34.233104 [DEBUG] http: Request /v1/agent/health?type=client (218.8µs) Dec 13 19:40:34 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (218.8µs) Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:36.173973 [DEBUG] client: updated allocations at index 607227 (total 260) (pulled 3) (filtered 257) Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:36.174315 [DEBUG] client: allocs: (added 0) (removed 0) (updated 3) (ignore 257) Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]: client: updated allocations at index 607227 (total 260) (pulled 3) (filtered 257) Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:36.174398 [DEBUG] client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039' Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:36.174435 [DEBUG] client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565' Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]: client: allocs: (added 0) (removed 0) (updated 3) (ignore 257) Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]: client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039' Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]: client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565' ^[[B^[[BDec 13 19:40:40 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:40.315631 [DEBUG] http: Request /v1/status/leader (1.02672ms) Dec 13 19:40:40 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (1.02672ms) Dec 13 19:40:44 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:44.234169 [DEBUG] http: Request /v1/agent/health?type=client (203.076µs) Dec 13 19:40:44 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (203.076µs) Dec 13 19:40:54 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:54.235522 [DEBUG] http: Request /v1/agent/health?type=client (234.772µs) Dec 13 19:40:54 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (234.772µs) Dec 13 19:40:55 ip-10-30-1-43 nomad[15209]: 2017/12/13 19:40:55.317594 [DEBUG] http: Request /v1/status/leader (996.679µs) Dec 13 19:40:55 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (996.679µs) ``` 3: ``` curl http://10.30.1.43:4646/v1/job/uniporter-uat/periodic-1510743600/allocations [{"ID":"5e0496dd-f515-86fc-fd64-5d19807061a0","EvalID":"95537cd5-6166-f41c-ef7c-c90edd8d87b6","Name":"uniporter-uat/periodic-1510743600.ibm-wfo[0]","NodeID":"7cd5a435-ae09-ab0a-1f90-c4228be55cce","JobID":"uniporter-uat/periodic-1510743600","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"failed","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":true,"Restarts":0,"LastRestart":"0001-01-01T00:00:00Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Received","Time":1511049208686408058,"Message":"","DisplayMessage":"Task received by client","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Task Setup","Time":1511049208686482136,"Message":"Building Task Directory","DisplayMessage":"Building Task Directory","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killing","Time":1511049239383850300,"Message":"","DisplayMessage":"Killing task: vault: failed to derive token: failed to unwrap the token for task \"jtcc\"","Details":null,"FailsTask":true,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"vault: failed to derive token: failed to unwrap the token for task \"jtcc\"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":425697,"ModifyIndex":607221,"CreateTime":1511049208218050041,"ModifyTime":0},{"ID":"fc6d66ba-7719-062a-dc32-889a60709eb1","EvalID":"1500803d-ef4d-6856-1b58-43e11ad812f9","Name":"uniporter-uat/periodic-1510743600.ibm-wfo[0]","NodeID":"7cd5a435-ae09-ab0a-1f90-c4228be55cce","JobID":"uniporter-uat/periodic-1510743600","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"stop","DesiredDescription":"alloc is lost since its node is down","ClientStatus":"complete","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":false,"Restarts":16,"LastRestart":"2017-11-15T11:04:13.390821067Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Driver","Time":1510743820393178908,"Message":"","DisplayMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","GenericSource":""},{"Type":"Driver Failure","Time":1510743820478967357,"Message":"","DisplayMessage":"failed to initialize task \"jtcc\" for alloc \"fc6d66ba-7719-062a-dc32-889a60709eb1\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"failed to initialize task \"jtcc\" for alloc \"fc6d66ba-7719-062a-dc32-889a60709eb1\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Restarting","Time":1510743820479004253,"Message":"","DisplayMessage":"Task restarting in 17.09760235s","Details":null,"FailsTask":false,"RestartReason":"Restart within policy","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":17097602350,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Driver","Time":1510743837577786703,"Message":"","DisplayMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","GenericSource":""},{"Type":"Driver Failure","Time":1510743837652731353,"Message":"","DisplayMessage":"failed to initialize task \"jtcc\" for alloc \"fc6d66ba-7719-062a-dc32-889a60709eb1\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"failed to initialize task \"jtcc\" for alloc \"fc6d66ba-7719-062a-dc32-889a60709eb1\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Restarting","Time":1510743837652766788,"Message":"","DisplayMessage":"Task restarting in 15.611816545s","Details":null,"FailsTask":false,"RestartReason":"Restart within policy","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":15611816545,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Driver","Time":1510743853265843002,"Message":"","DisplayMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","GenericSource":""},{"Type":"Driver Failure","Time":1510743853390778326,"Message":"","DisplayMessage":"failed to initialize task \"jtcc\" for alloc \"fc6d66ba-7719-062a-dc32-889a60709eb1\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"failed to initialize task \"jtcc\" for alloc \"fc6d66ba-7719-062a-dc32-889a60709eb1\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Restarting","Time":1510743853390821067,"Message":"","DisplayMessage":"Exceeded allowed attempts, applying a delay - Task restarting in 167h55m46.723984944s","Details":null,"FailsTask":false,"RestartReason":"Exceeded allowed attempts, applying a delay","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":604546723984944,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killed","Time":1511049197330244197,"Message":"","DisplayMessage":"Task successfully killed","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":418809,"ModifyIndex":607146,"CreateTime":1510743600083464036,"ModifyTime":0}] ```

That job:

uniporter-uat/periodic-1511175600         batch           50        dead (stopped)  11/20/17 11:00:00 UTC

I managed to run nomad stop -purge $job on all these jobs, and they no longer exist:

curl http://10.30.1.43:4646/v1/job/uniporter-uat/periodic-1511953200
job not found

root@ip-10-30-1-43:/var/nomad# curl http://10.30.1.43:4646/v1/job/uniporter-uat/periodic-1511953200/allocations
[{"ID":"3b81f235-c6c2-0a04-9dca-28e9652fa43d","EvalID":"cafede1e-575e-b099-22ce-e6cb837acdb8","Name":"uniporter-uat/periodic-1511953200.ibm-wfo[0]","NodeID":"7cd5a435-ae09-ab0a-1f90-c4228be55cce","JobID":"uniporter-uat/periodic-1511953200","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"stop","DesiredDescription":"alloc not needed due to job update","ClientStatus":"complete","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":false,"Restarts":23,"LastRestart":"2017-12-13T19:52:37.06046043Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Driver","Time":1513194722835388017,"Message":"","DisplayMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","GenericSource":""},{"Type":"Driver Failure","Time":1513194722918026138,"Message":"","DisplayMessage":"failed to initialize task \"jtcc\" for alloc \"3b81f235-c6c2-0a04-9dca-28e9652fa43d\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"failed to initialize task \"jtcc\" for alloc \"3b81f235-c6c2-0a04-9dca-28e9652fa43d\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Restarting","Time":1513194722918074121,"Message":"","DisplayMessage":"Task restarting in 15.960969333s","Details":null,"FailsTask":false,"RestartReason":"Restart within policy","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":15960969333,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Driver","Time":1513194738881431161,"Message":"","DisplayMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","GenericSource":""},{"Type":"Driver Failure","Time":1513194738962467969,"Message":"","DisplayMessage":"failed to initialize task \"jtcc\" for alloc \"3b81f235-c6c2-0a04-9dca-28e9652fa43d\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"failed to initialize task \"jtcc\" for alloc \"3b81f235-c6c2-0a04-9dca-28e9652fa43d\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Restarting","Time":1513194738962563682,"Message":"","DisplayMessage":"Task restarting in 17.911645299s","Details":null,"FailsTask":false,"RestartReason":"Restart within policy","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":17911645299,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Driver","Time":1513194756879242992,"Message":"","DisplayMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","GenericSource":""},{"Type":"Driver Failure","Time":1513194757059836499,"Message":"","DisplayMessage":"failed to initialize task \"jtcc\" for alloc \"3b81f235-c6c2-0a04-9dca-28e9652fa43d\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"failed to initialize task \"jtcc\" for alloc \"3b81f235-c6c2-0a04-9dca-28e9652fa43d\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Restarting","Time":1513194757060460430,"Message":"","DisplayMessage":"Task restarting in 15.453403262s","Details":null,"FailsTask":false,"RestartReason":"Restart within policy","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":15453403262,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killed","Time":1513194765024101601,"Message":"","DisplayMessage":"Task successfully killed","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":607390,"ModifyIndex":607806,"CreateTime":1513194279915270907,"ModifyTime":0},{"ID":"8c83f1cb-8589-7773-b88a-82f63ad1caf9","EvalID":"d2258389-43f3-8ab7-e738-0e5b57e1b2c6","Name":"uniporter-uat/periodic-1511953200.ibm-wfo[0]","NodeID":"7cdec421-166e-04da-07e1-2b37858357a3","JobID":"uniporter-uat/periodic-1511953200","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"failed","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":true,"Restarts":0,"LastRestart":"0001-01-01T00:00:00Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Received","Time":1513115825621384244,"Message":"","DisplayMessage":"Task received by client","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Task Setup","Time":1513115825621496358,"Message":"Building Task Directory","DisplayMessage":"Building Task Directory","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killing","Time":1513115825636789224,"Message":"","DisplayMessage":"Killing task: vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"8c83f1cb-8589-7773-b88a-82f63ad1caf9\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","Details":null,"FailsTask":true,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"8c83f1cb-8589-7773-b88a-82f63ad1caf9\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":595105,"ModifyIndex":595302,"CreateTime":1513115818001542460,"ModifyTime":0},{"ID":"94936d21-87ec-68da-ac40-31fc7d8f74fd","EvalID":"5039797e-5886-cef2-b7b0-3c85efec7108","Name":"uniporter-uat/periodic-1511953200.ibm-wfo[0]","NodeID":"7cd5a435-ae09-ab0a-1f90-c4228be55cce","JobID":"uniporter-uat/periodic-1511953200","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"failed","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":true,"Restarts":0,"LastRestart":"0001-01-01T00:00:00Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Received","Time":1513112912870960502,"Message":"","DisplayMessage":"Task received by client","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Task Setup","Time":1513112912871048881,"Message":"Building Task Directory","DisplayMessage":"Building Task Directory","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killing","Time":1513112912901171716,"Message":"","DisplayMessage":"Killing task: vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"94936d21-87ec-68da-ac40-31fc7d8f74fd\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","Details":null,"FailsTask":true,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"94936d21-87ec-68da-ac40-31fc7d8f74fd\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":592027,"ModifyIndex":607786,"CreateTime":1513112903271182797,"ModifyTime":0},{"ID":"b73bba18-671e-5f9d-4217-1c9b2b22f723","EvalID":"9d143393-2a24-5f75-7517-7146997074e2","Name":"uniporter-uat/periodic-1511953200.ibm-wfo[0]","NodeID":"7cd5a435-ae09-ab0a-1f90-c4228be55cce","JobID":"uniporter-uat/periodic-1511953200","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"failed","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":true,"Restarts":0,"LastRestart":"0001-01-01T00:00:00Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Received","Time":1513115803904081416,"Message":"","DisplayMessage":"Task received by client","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Task Setup","Time":1513115803904151831,"Message":"Building Task Directory","DisplayMessage":"Building Task Directory","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killing","Time":1513115803931710036,"Message":"","DisplayMessage":"Killing task: vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"b73bba18-671e-5f9d-4217-1c9b2b22f723\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","Details":null,"FailsTask":true,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"b73bba18-671e-5f9d-4217-1c9b2b22f723\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":594855,"ModifyIndex":607786,"CreateTime":1513115803876082994,"ModifyTime":0},{"ID":"c7d1093e-ca1b-f385-0502-34d67cf7b9d3","EvalID":"1b9981c7-fcbf-ea3a-1bf9-ded95897aadc","Name":"uniporter-uat/periodic-1511953200.ibm-wfo[0]","NodeID":"7cdec421-166e-04da-07e1-2b37858357a3","JobID":"uniporter-uat/periodic-1511953200","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"failed","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":true,"Restarts":0,"LastRestart":"0001-01-01T00:00:00Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Received","Time":1513115398637824126,"Message":"","DisplayMessage":"Task received by client","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Task Setup","Time":1513115398637986165,"Message":"Building Task Directory","DisplayMessage":"Building Task Directory","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killing","Time":1513115398653332452,"Message":"","DisplayMessage":"Killing task: vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"c7d1093e-ca1b-f385-0502-34d67cf7b9d3\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","Details":null,"FailsTask":true,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"c7d1093e-ca1b-f385-0502-34d67cf7b9d3\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":594082,"ModifyIndex":594373,"CreateTime":1513115383337841929,"ModifyTime":0},{"ID":"d16d4240-ee9a-9bea-4eac-23a032b75336","EvalID":"7fbe95da-90eb-f40d-cdbb-7ab7ca716a4d","Name":"uniporter-uat/periodic-1511953200.ibm-wfo[0]","NodeID":"7cd5a435-ae09-ab0a-1f90-c4228be55cce","JobID":"uniporter-uat/periodic-1511953200","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"failed","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":true,"Restarts":0,"LastRestart":"0001-01-01T00:00:00Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Received","Time":1513112735189272618,"Message":"","DisplayMessage":"Task received by client","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Task Setup","Time":1513112735189349266,"Message":"Building Task Directory","DisplayMessage":"Building Task Directory","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killing","Time":1513112735204538511,"Message":"","DisplayMessage":"Killing task: vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"d16d4240-ee9a-9bea-4eac-23a032b75336\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","Details":null,"FailsTask":true,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"d16d4240-ee9a-9bea-4eac-23a032b75336\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":591541,"ModifyIndex":607786,"CreateTime":1513112730585011087,"ModifyTime":0}]

Then the allocs still can't gc:

Dec 13 19:59:34 ip-10-30-1-43 nomad[28460]: client: allocs: (added 0) (removed 0) (updated 4) (ignore 265)
Dec 13 19:59:34 ip-10-30-1-43 nomad[28460]: client: dropping update to terminal alloc '3b81f235-c6c2-0a04-9dca-28e9652fa43d'
Dec 13 19:59:34 ip-10-30-1-43 nomad[28460]: client: dropping update to terminal alloc '0b1e29f2-976e-1c85-2ab5-0689915a6244'
Dec 13 19:59:34 ip-10-30-1-43 nomad[28460]: client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039'
Dec 13 19:59:34 ip-10-30-1-43 nomad[28460]: client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565'

We ended up bringing down all masters, wiping all raft/client.db, and starting from scratch. All gc'ing was broken across all nodes, and it looks like it's been broken since 0.5->0.6 upgrade.

@apenney Unfortunately the logs you gave are client logs not the server logs. So there really isn't enough information to try to track this down.

I am going to close this until we do get more information or you can provide repro steps. If it does happen again, following the steps I outlined in my response should provide the information we need.

Not sure if I can reopen this but here are the server logs during a gc, as this issue is now reoccurring. Server3 is the master in the below paste.

Symptoms, just to recap: Nodes fill up with allocs, and can't gc. I have over ~3000 jobs when I should have about 20, most of those are "dead" batch jobs that never go away. On the servers I can see it constantly trying to alloc and failing to get vault tokens for jobs that are dead and shouldn't be doing anything.

Server1:

``` Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:04.740810 [DEBUG] http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (8.153384ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (8.153384ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:04.741363 [DEBUG] http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (6.86882ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (6.86882ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:04.741910 [DEBUG] http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (7.240519ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (7.240519ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:04.742494 [DEBUG] http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (6.516327ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (6.516327ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:04.743035 [DEBUG] http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (7.563235ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (7.563235ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:04.743606 [DEBUG] http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (8.413474ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (8.413474ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:04.744147 [DEBUG] http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (8.832525ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (8.832525ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (8.941974ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:04.744684 [DEBUG] http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (8.941974ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:04.745211 [DEBUG] http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (9.584528ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (9.584528ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (4.438317ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:04.739275 [DEBUG] http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (4.438317ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:04.746330 [DEBUG] http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (11.370118ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (11.370118ms) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:04.747287 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (181.824µs) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (181.824µs) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:04.773791 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (116.087µs) Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (116.087µs) Jan 12 15:52:05 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:05 [DEBUG] memberlist: TCP connection from=10.50.2.47:35878 Jan 12 15:52:05 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.2.47:35878 Jan 12 15:52:06 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:06.528954 [DEBUG] http: Request /v1/agent/health?type=server (1.063618ms) Jan 12 15:52:06 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (1.063618ms) Jan 12 15:52:06 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:06.904706 [DEBUG] http: Request /v1/agent/health?type=client (279.685µs) Jan 12 15:52:06 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (279.685µs) Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:07.097761 [DEBUG] http: Request /v1/agent/health?type=client (649.065µs) Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (649.065µs) Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:07.184990 [DEBUG] http: Request /v1/agent/health?type=client (192.565µs) Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (192.565µs) Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:07.187744 [DEBUG] http: Request /v1/agent/health?type=server (507.651µs) Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (507.651µs) Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:07.201052 [DEBUG] http: Request /v1/agent/health?type=server (468.691µs) Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (468.691µs) Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:07 [DEBUG] memberlist: TCP connection from=10.50.5.8:52664 Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52664 Jan 12 15:52:08 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52666 Jan 12 15:52:08 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:08 [DEBUG] memberlist: TCP connection from=10.50.5.8:52666 Jan 12 15:52:09 ip-10-50-1-29 nomad[1436]: http: Request /v1/status/leader?region=us-east-1&wait=60000ms (540.14µs) Jan 12 15:52:09 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:09.965589 [DEBUG] http: Request /v1/status/leader?region=us-east-1&wait=60000ms (540.14µs) Jan 12 15:52:10 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:10.526439 [DEBUG] http: Request /v1/agent/health?type=client (344.193µs) Jan 12 15:52:10 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (344.193µs) Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:11.245061 [DEBUG] http: Request /v1/agent/health?type=server (1.566163ms) Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (1.566163ms) Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:11.816870 [DEBUG] http: Request /v1/agent/health?type=server (663.73µs) Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (663.73µs) Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:11 [DEBUG] memberlist: TCP connection from=10.50.4.247:35548 Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35548 Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:11 [DEBUG] memberlist: TCP connection from=10.50.4.247:35550 Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35550 Jan 12 15:52:12 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (217.803µs) Jan 12 15:52:12 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:12.597037 [DEBUG] http: Request /v1/agent/health?type=client (217.803µs) Jan 12 15:52:13 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:13.553745 [DEBUG] http: Request /v1/allocations?index=696204®ion=us-east-1&stale=&wait=120000ms (511.983994ms) Jan 12 15:52:13 ip-10-50-1-29 nomad[1436]: http: Request /v1/allocations?index=696204®ion=us-east-1&stale=&wait=120000ms (511.983994ms) Jan 12 15:52:13 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:13.608652 [DEBUG] http: Request /v1/agent/health?type=server (605.496µs) Jan 12 15:52:13 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (605.496µs) Jan 12 15:52:14 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (186.597µs) Jan 12 15:52:14 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:14.362698 [DEBUG] http: Request /v1/agent/health?type=client (186.597µs) Jan 12 15:52:14 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:14 [DEBUG] memberlist: TCP connection from=10.50.1.12:33592 Jan 12 15:52:14 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.1.12:33592 Jan 12 15:52:15 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:15.242575 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (195.512µs) Jan 12 15:52:15 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (195.512µs) Jan 12 15:52:15 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:15.283239 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (212.225µs) Jan 12 15:52:15 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (212.225µs) Jan 12 15:52:15 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:15 [DEBUG] memberlist: TCP connection from=10.50.2.47:35958 Jan 12 15:52:15 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.2.47:35958 Jan 12 15:52:16 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:16.532754 [DEBUG] http: Request /v1/agent/health?type=server (1.184321ms) Jan 12 15:52:16 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (1.184321ms) Jan 12 15:52:16 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:16.908213 [DEBUG] http: Request /v1/agent/health?type=client (227.075µs) Jan 12 15:52:16 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (227.075µs) Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:17.086415 [DEBUG] http: Request /v1/status/leader (1.422652ms) Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: http: Request /v1/status/leader (1.422652ms) Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:17.099573 [DEBUG] http: Request /v1/agent/health?type=client (116.697µs) Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (116.697µs) Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (218.146µs) Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:17.187443 [DEBUG] http: Request /v1/agent/health?type=client (218.146µs) Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:17.191839 [DEBUG] http: Request /v1/agent/health?type=server (2.148459ms) Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (2.148459ms) Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:17.212093 [DEBUG] http: Request /v1/agent/health?type=server (8.229039ms) Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (8.229039ms) Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52738 Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:17 [DEBUG] memberlist: TCP connection from=10.50.5.8:52738 Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:18 [DEBUG] memberlist: TCP connection from=10.50.5.8:52742 Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52742 Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/vault-ui/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (637.426µs) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:18.672077 [DEBUG] http: Request /v1/job/vault-ui/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (637.426µs) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:18.674354 [DEBUG] http: Request /v1/job/kibana-production/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (410.793µs) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/kibana-production/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (410.793µs) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:18.684499 [DEBUG] http: Request /v1/job/mitochondrion-staging/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (10.800316ms) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/mitochondrion-staging/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (10.800316ms) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:18.690559 [DEBUG] http: Request /v1/job/consul-snapshot/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (245.419µs) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/consul-snapshot/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (245.419µs) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:18.762328 [DEBUG] http: Request /v1/job/symbiosis-uat/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (6.315185ms) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/symbiosis-uat/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (6.315185ms) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:18.772827 [DEBUG] http: Request /v1/job/symbiosis-production/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (2.709907ms) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/symbiosis-production/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (2.709907ms) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:18.776125 [DEBUG] http: Request /v1/job/symbiosis-staging/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (19.587042ms) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/symbiosis-staging/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (19.587042ms) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:18.776803 [DEBUG] http: Request /v1/job/tricorder-production/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (2.357293ms) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/tricorder-production/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (2.357293ms) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:18.777889 [DEBUG] http: Request /v1/job/kibana-uat/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (413.159µs) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/kibana-uat/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (413.159µs) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:18.778473 [DEBUG] http: Request /v1/job/scribe-production/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (664.869µs) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/scribe-production/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (664.869µs) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:18.781360 [DEBUG] http: Request /v1/job/scribe-uat/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (2.509439ms) Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/scribe-uat/allocations?all=false®ion=us-east-1&stale=&wait=120000ms (2.509439ms) Jan 12 15:52:19 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:19.958151 [DEBUG] http: Request /v1/regions?region=us-east-1&wait=60000ms (634.289µs) Jan 12 15:52:19 ip-10-50-1-29 nomad[1436]: http: Request /v1/regions?region=us-east-1&wait=60000ms (634.289µs) Jan 12 15:52:19 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:19.964389 [DEBUG] http: Request /v1/status/leader?region=us-east-1&wait=60000ms (1.085318ms) Jan 12 15:52:19 ip-10-50-1-29 nomad[1436]: http: Request /v1/status/leader?region=us-east-1&wait=60000ms (1.085318ms) Jan 12 15:52:20 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:20.529149 [DEBUG] http: Request /v1/agent/health?type=client (851.152µs) Jan 12 15:52:20 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (851.152µs) Jan 12 15:52:20 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:20.725620 [DEBUG] http: Request /v1/allocations?index=696209®ion=us-east-1&stale=&wait=120000ms (2.578614755s) Jan 12 15:52:20 ip-10-50-1-29 nomad[1436]: http: Request /v1/allocations?index=696209®ion=us-east-1&stale=&wait=120000ms (2.578614755s) Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:21.247595 [DEBUG] http: Request /v1/agent/health?type=server (1.077007ms) Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (1.077007ms) Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:21.820220 [DEBUG] http: Request /v1/agent/health?type=server (567.847µs) Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (567.847µs) Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:21 [DEBUG] memberlist: TCP connection from=10.50.4.247:35628 Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35628 Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:21 [DEBUG] memberlist: TCP connection from=10.50.4.247:35630 Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35630 Jan 12 15:52:22 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:22.599732 [DEBUG] http: Request /v1/agent/health?type=client (237.127µs) Jan 12 15:52:22 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (237.127µs) Jan 12 15:52:23 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:23.612269 [DEBUG] http: Request /v1/agent/health?type=server (586.58µs) Jan 12 15:52:23 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (586.58µs) Jan 12 15:52:23 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:23.626731 [DEBUG] http: Request /v1/job/sftp-s3/versions?diffs=false&index=695801®ion=us-east-1&stale=&wait=120000ms (2m6.976314067s) Jan 12 15:52:23 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/sftp-s3/versions?diffs=false&index=695801®ion=us-east-1&stale=&wait=120000ms (2m6.976314067s) Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:24.171698 [DEBUG] http: Request /v1/system/gc (509.72µs) Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]: http: Request /v1/system/gc (509.72µs) Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:24.365619 [DEBUG] http: Request /v1/agent/health?type=client (238.222µs) Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (238.222µs) Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:24 [DEBUG] memberlist: TCP connection from=10.50.1.12:33660 Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.1.12:33660 Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:24.955752 [DEBUG] http: Request /v1/regions?region=us-east-1&wait=60000ms (182.903µs) Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]: http: Request /v1/regions?region=us-east-1&wait=60000ms (182.903µs) Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:25.253542 [DEBUG] http: Request /v1/allocations?index=696212®ion=us-east-1&stale=&wait=120000ms (2.876001562s) Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: http: Request /v1/allocations?index=696212®ion=us-east-1&stale=&wait=120000ms (2.876001562s) Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:25.537420 [DEBUG] http: Request /v1/nodes?index=696031®ion=us-east-1&stale=&wait=10000ms (10.247298219s) Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: http: Request /v1/nodes?index=696031®ion=us-east-1&stale=&wait=10000ms (10.247298219s) Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:25.570706 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (183.179µs) Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (183.179µs) Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:25.611524 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (122.086µs) Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (122.086µs) Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:25 [DEBUG] memberlist: TCP connection from=10.50.2.47:36032 Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.2.47:36032 Jan 12 15:52:26 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:26.535868 [DEBUG] http: Request /v1/agent/health?type=server (622.076µs) Jan 12 15:52:26 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (622.076µs) Jan 12 15:52:26 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:26.911992 [DEBUG] http: Request /v1/agent/health?type=client (1.223915ms) Jan 12 15:52:26 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (1.223915ms) Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:27.101542 [DEBUG] http: Request /v1/agent/health?type=client (291.829µs) Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (291.829µs) Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (337.031µs) Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:27.189988 [DEBUG] http: Request /v1/agent/health?type=client (337.031µs) Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:27.194208 [DEBUG] http: Request /v1/agent/health?type=server (628.072µs) Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (628.072µs) Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:27.214834 [DEBUG] http: Request /v1/agent/health?type=server (475.46µs) Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (475.46µs) Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52814 Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:27 [DEBUG] memberlist: TCP connection from=10.50.5.8:52814 Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.1.95:36852 Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:28 [DEBUG] memberlist: TCP connection from=10.50.1.95:36852 Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/tricorder-production/versions?diffs=false®ion=us-east-1&stale=&wait=120000ms (883.538µs) Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:28.337701 [DEBUG] http: Request /v1/job/tricorder-production/versions?diffs=false®ion=us-east-1&stale=&wait=120000ms (883.538µs) Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:28.339358 [DEBUG] http: Request /v1/job/tricorder-production?region=us-east-1&stale=&wait=120000ms (316.702µs) Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/tricorder-production?region=us-east-1&stale=&wait=120000ms (316.702µs) Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52816 Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:28 [DEBUG] memberlist: TCP connection from=10.50.5.8:52816 Jan 12 15:52:29 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:29.992566 [DEBUG] http: Request /v1/regions?region=us-east-1&wait=60000ms (626.969µs) Jan 12 15:52:29 ip-10-50-1-29 nomad[1436]: http: Request /v1/regions?region=us-east-1&wait=60000ms (626.969µs) Jan 12 15:52:30 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:30.531714 [DEBUG] http: Request /v1/agent/health?type=client (701.375µs) Jan 12 15:52:30 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (701.375µs) Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:31.250444 [DEBUG] http: Request /v1/agent/health?type=server (952.828µs) Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (952.828µs) Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:31.827289 [DEBUG] http: Request /v1/agent/health?type=server (4.21674ms) Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (4.21674ms) Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:31 [DEBUG] memberlist: TCP connection from=10.50.4.247:35700 Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35700 Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:31.884365 [DEBUG] http: Request /v1/allocations?index=696214®ion=us-east-1&stale=&wait=120000ms (924.854623ms) Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: http: Request /v1/allocations?index=696214®ion=us-east-1&stale=&wait=120000ms (924.854623ms) Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35702 Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:31 [DEBUG] memberlist: TCP connection from=10.50.4.247:35702 Jan 12 15:52:32 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:32.088790 [DEBUG] http: Request /v1/status/leader (516.314µs) Jan 12 15:52:32 ip-10-50-1-29 nomad[1436]: http: Request /v1/status/leader (516.314µs) Jan 12 15:52:32 ip-10-50-1-29 nomad[1436]: memberlist: Initiating push/pull sync with: 10.50.1.95:4648 Jan 12 15:52:32 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:32 [DEBUG] memberlist: Initiating push/pull sync with: 10.50.1.95:4648 Jan 12 15:52:32 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (259.764µs) Jan 12 15:52:32 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:32.602329 [DEBUG] http: Request /v1/agent/health?type=client (259.764µs) Jan 12 15:52:33 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:33.616261 [DEBUG] http: Request /v1/agent/health?type=server (1.036229ms) Jan 12 15:52:33 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (1.036229ms) Jan 12 15:52:34 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:34.368295 [DEBUG] http: Request /v1/agent/health?type=client (188.306µs) Jan 12 15:52:34 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (188.306µs) Jan 12 15:52:34 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:34 [DEBUG] memberlist: TCP connection from=10.50.1.12:33714 Jan 12 15:52:34 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.1.12:33714 Jan 12 15:52:35 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:35 [DEBUG] memberlist: TCP connection from=10.50.2.47:36096 Jan 12 15:52:35 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.2.47:36096 Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:36.025499 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (322.979µs) Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (322.979µs) Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:36.050205 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (200.549µs) Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (200.549µs) Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:36.538979 [DEBUG] http: Request /v1/agent/health?type=server (563.906µs) Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (563.906µs) Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:36.914877 [DEBUG] http: Request /v1/agent/health?type=client (214.858µs) Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (214.858µs) Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:37.103962 [DEBUG] http: Request /v1/agent/health?type=client (198.812µs) Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (198.812µs) Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (323.954µs) Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:37.192553 [DEBUG] http: Request /v1/agent/health?type=client (323.954µs) Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:37.198934 [DEBUG] http: Request /v1/agent/health?type=server (2.856772ms) Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (2.856772ms) Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:37.234092 [DEBUG] http: Request /v1/agent/health?type=server (16.242204ms) Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (16.242204ms) Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:37.414136 [DEBUG] http: Request /v1/job/tricorder-uat/allocations?all=false&index=695870®ion=us-east-1&stale=&wait=120000ms (2m5.217383171s) Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/tricorder-uat/allocations?all=false&index=695870®ion=us-east-1&stale=&wait=120000ms (2m5.217383171s) Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:37 [DEBUG] memberlist: TCP connection from=10.50.5.8:52886 Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52886 Jan 12 15:52:38 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:38.045488 [DEBUG] http: Request /v1/job/docbot-production/allocations?all=false&index=695870®ion=us-east-1&stale=&wait=120000ms (2m6.455937814s) Jan 12 15:52:38 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/docbot-production/allocations?all=false&index=695870®ion=us-east-1&stale=&wait=120000ms (2m6.455937814s) Jan 12 15:52:38 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:38 [DEBUG] memberlist: TCP connection from=10.50.5.8:52888 Jan 12 15:52:38 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52888 Jan 12 15:52:38 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:38.469131 [DEBUG] http: Request /v1/job/sftp-s3-test?index=695811®ion=us-east-1&stale=&wait=120000ms (2m7.087101371s) Jan 12 15:52:38 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/sftp-s3-test?index=695811®ion=us-east-1&stale=&wait=120000ms (2m7.087101371s) Jan 12 15:52:39 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:39.959955 [DEBUG] http: Request /v1/regions?region=us-east-1&wait=60000ms (620.782µs) Jan 12 15:52:39 ip-10-50-1-29 nomad[1436]: http: Request /v1/regions?region=us-east-1&wait=60000ms (620.782µs) Jan 12 15:52:39 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:39.972180 [DEBUG] http: Request /v1/status/leader?region=us-east-1&wait=60000ms (955.891µs) Jan 12 15:52:39 ip-10-50-1-29 nomad[1436]: http: Request /v1/status/leader?region=us-east-1&wait=60000ms (955.891µs) Jan 12 15:52:40 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:40.173322 [DEBUG] http: Request /v1/job/scribe-staging/allocations?all=false&index=695870®ion=us-east-1&stale=&wait=120000ms (2m3.673234376s) Jan 12 15:52:40 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/scribe-staging/allocations?all=false&index=695870®ion=us-east-1&stale=&wait=120000ms (2m3.673234376s) Jan 12 15:52:40 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:40.367549 [DEBUG] http: Request /v1/job/scribe-production/allocations?all=false&index=695870®ion=us-east-1&stale=&wait=120000ms (2m5.160696839s) Jan 12 15:52:40 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/scribe-production/allocations?all=false&index=695870®ion=us-east-1&stale=&wait=120000ms (2m5.160696839s) Jan 12 15:52:40 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (211.394µs) Jan 12 15:52:40 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:40.533214 [DEBUG] http: Request /v1/agent/health?type=client (211.394µs) Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:41.257815 [DEBUG] http: Request /v1/agent/health?type=server (5.086311ms) Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (5.086311ms) Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:41.542672 [DEBUG] http: Request /v1/job/docbot-production?index=694504®ion=us-east-1&stale=&wait=120000ms (2m6.159973379s) Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/docbot-production?index=694504®ion=us-east-1&stale=&wait=120000ms (2m6.159973379s) Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:41.579486 [DEBUG] http: Request /v1/job/symbiosis-staging/allocations?all=false&index=695873®ion=us-east-1&stale=&wait=120000ms (2m0.429229659s) Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/symbiosis-staging/allocations?all=false&index=695873®ion=us-east-1&stale=&wait=120000ms (2m0.429229659s) Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:41.592937 [DEBUG] http: Request /v1/job/docbot-production/allocations?all=false&index=696128®ion=us-east-1&stale=&wait=120000ms (12.495412ms) Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/docbot-production/allocations?all=false&index=696128®ion=us-east-1&stale=&wait=120000ms (12.495412ms) Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:41.758829 [DEBUG] http: Request /v1/allocations?index=696222®ion=us-east-1&stale=&wait=120000ms (562.7799ms) Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: http: Request /v1/allocations?index=696222®ion=us-east-1&stale=&wait=120000ms (562.7799ms) Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:41.830815 [DEBUG] http: Request /v1/agent/health?type=server (1.076224ms) Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (1.076224ms) Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:41 [DEBUG] memberlist: TCP connection from=10.50.4.247:35788 Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35788 Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:41 [DEBUG] memberlist: TCP connection from=10.50.4.247:35790 Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35790 Jan 12 15:52:42 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:42.605002 [DEBUG] http: Request /v1/agent/health?type=client (301.485µs) Jan 12 15:52:42 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (301.485µs) Jan 12 15:52:43 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:43.445880 [DEBUG] http: Request /v1/job/kibana-staging/allocations?all=false&index=695870®ion=us-east-1&stale=&wait=120000ms (2m6.532032772s) Jan 12 15:52:43 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/kibana-staging/allocations?all=false&index=695870®ion=us-east-1&stale=&wait=120000ms (2m6.532032772s) Jan 12 15:52:43 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:43.593212 [DEBUG] http: Request /v1/job/hashi-ui/allocations?all=false&index=695870®ion=us-east-1&stale=&wait=120000ms (2m7.471101758s) Jan 12 15:52:43 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/hashi-ui/allocations?all=false&index=695870®ion=us-east-1&stale=&wait=120000ms (2m7.471101758s) Jan 12 15:52:43 ip-10-50-1-29 nomad[1436]: 2018/01/12 15:52:43.619118 [DEBUG] http: Request /v1/agent/health?type=server (542.983µs) Jan 12 15:52:43 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (542.983µs) Jan 12 15:52:44 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.1.12:33792 ```

Server2:

``` 3ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.538494 [DEBUG] http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (1.373583ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.539073 [DEBUG] http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (2.160888ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.539625 [DEBUG] http: Request /v1/node/ec20657b-ac9a-c80f-9a87-4098c5d9d665?region=us-east-1&wait=60000ms (1.600722ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (2.160888ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.540221 [DEBUG] http: Request /v1/node/5daf562c-9ccb-f55e-3fda-37fd0bfdd74c?region=us-east-1&wait=60000ms (2.318732ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.541177 [DEBUG] http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (595.869µs) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec20657b-ac9a-c80f-9a87-4098c5d9d665?region=us-east-1&wait=60000ms (1.600722ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.541717 [DEBUG] http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (879.934µs) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/5daf562c-9ccb-f55e-3fda-37fd0bfdd74c?region=us-east-1&wait=60000ms (2.318732ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.542231 [DEBUG] http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (1.517322ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (595.869µs) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.544732 [DEBUG] http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (706.517µs) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.545276 [DEBUG] http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (2.729114ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.545784 [DEBUG] http: Request /v1/node/cfc3db5b-563e-9c80-4e1b-557ecfb1a977?region=us-east-1&wait=60000ms (2.781095ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.546292 [DEBUG] http: Request /v1/node/2bb7c609-f7f5-de03-9609-51ed67e76470?region=us-east-1&wait=60000ms (2.636082ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (879.934µs) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.546814 [DEBUG] http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (3.0296ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.547341 [DEBUG] http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (3.429119ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (1.517322ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.547879 [DEBUG] http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (3.609681ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (706.517µs) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (2.729114ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.548396 [DEBUG] http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (4.242371ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/cfc3db5b-563e-9c80-4e1b-557ecfb1a977?region=us-east-1&wait=60000ms (2.781095ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/2bb7c609-f7f5-de03-9609-51ed67e76470?region=us-east-1&wait=60000ms (2.636082ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (3.0296ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (3.429119ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (3.609681ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (4.242371ms) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.557410 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (121.993µs) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (121.993µs) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.595895 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (152.568µs) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (152.568µs) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25.676433 [DEBUG] http: Request /v1/agent/health?type=server (562.73µs) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (562.73µs) Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25 [DEBUG] memberlist: TCP connection from=10.50.5.8:59084 Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59084 Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.2.47:36366 Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:25 [DEBUG] memberlist: TCP connection from=10.50.2.47:36366 Jan 12 15:52:26 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:26.973049 [DEBUG] http: Request /v1/agent/health?type=server (567.303µs) Jan 12 15:52:26 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (567.303µs) Jan 12 15:52:29 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (209.751µs) Jan 12 15:52:29 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:29.574723 [DEBUG] http: Request /v1/agent/health?type=client (209.751µs) Jan 12 15:52:29 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:29.685405 [DEBUG] http: Request /v1/agent/health?type=server (590.67µs) Jan 12 15:52:29 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (590.67µs) Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:30.217809 [DEBUG] http: Request /v1/agent/health?type=client (646.895µs) Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (646.895µs) Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:30.242089 [DEBUG] http: Request /v1/agent/health?type=client (487.638µs) Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (487.638µs) Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:30 [DEBUG] memberlist: TCP connection from=10.50.4.247:39998 Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:39998 Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:30 [DEBUG] memberlist: TCP connection from=10.50.4.247:40000 Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40000 Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (272.493µs) Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:30.767758 [DEBUG] http: Request /v1/agent/health?type=client (272.493µs) Jan 12 15:52:32 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (234.9µs) Jan 12 15:52:32 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:32.119432 [DEBUG] http: Request /v1/agent/health?type=client (234.9µs) Jan 12 15:52:32 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (497.767µs) Jan 12 15:52:32 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:32.241450 [DEBUG] http: Request /v1/agent/health?type=server (497.767µs) Jan 12 15:52:33 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:33.146783 [DEBUG] http: Request /v1/allocations?index=696215®ion=us-east-1&stale=&wait=120000ms (705.424314ms) Jan 12 15:52:33 ip-10-50-1-101 nomad[25850]: http: Request /v1/allocations?index=696215®ion=us-east-1&stale=&wait=120000ms (705.424314ms) Jan 12 15:52:33 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:33.487380 [DEBUG] http: Request /v1/agent/health?type=server (1.343953ms) Jan 12 15:52:33 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (1.343953ms) Jan 12 15:52:33 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:33.853861 [DEBUG] http: Request /v1/agent/health?type=server (912.993µs) Jan 12 15:52:33 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (912.993µs) Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:34.231842 [DEBUG] http: Request /v1/allocations?index=696216®ion=us-east-1&stale=&wait=120000ms (495.16528ms) Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]: http: Request /v1/allocations?index=696216®ion=us-east-1&stale=&wait=120000ms (495.16528ms) Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (323.833µs) Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:34.543238 [DEBUG] http: Request /v1/agent/health?type=client (323.833µs) Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:34.937950 [DEBUG] http: Request /v1/agent/members?region=us-east-1&wait=60000ms (423.906µs) Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/members?region=us-east-1&wait=60000ms (423.906µs) Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:34.953501 [DEBUG] http: Request /v1/status/leader?region=us-east-1&wait=60000ms (437.547µs) Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]: http: Request /v1/status/leader?region=us-east-1&wait=60000ms (437.547µs) Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:35 [DEBUG] memberlist: TCP connection from=10.50.5.8:59152 Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59152 Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:35.166523 [DEBUG] http: Request /v1/allocations?index=696217®ion=us-east-1&stale=&wait=120000ms (349.708925ms) Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: http: Request /v1/allocations?index=696217®ion=us-east-1&stale=&wait=120000ms (349.708925ms) Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:35.170989 [DEBUG] http: Request /v1/status/leader (984.958µs) Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: http: Request /v1/status/leader (984.958µs) Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.1.12:39768 Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:35 [DEBUG] memberlist: TCP connection from=10.50.1.12:39768 Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:35.680598 [DEBUG] http: Request /v1/agent/health?type=server (921.487µs) Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (921.487µs) Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:35 [DEBUG] memberlist: TCP connection from=10.50.5.8:59154 Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59154 Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:35 [DEBUG] memberlist: TCP connection from=10.50.2.47:36430 Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.2.47:36430 Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.000353 [DEBUG] http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (1.553256ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.001122 [DEBUG] http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (1.248687ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.001662 [DEBUG] http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (904.883µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (1.553256ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (1.248687ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.002575 [DEBUG] http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (600.338µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (904.883µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (600.338µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/5daf562c-9ccb-f55e-3fda-37fd0bfdd74c?region=us-east-1&wait=60000ms (911.034µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.003140 [DEBUG] http: Request /v1/node/5daf562c-9ccb-f55e-3fda-37fd0bfdd74c?region=us-east-1&wait=60000ms (911.034µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.003914 [DEBUG] http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (1.808776ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (1.808776ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.007128 [DEBUG] http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (1.015616ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (1.015616ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.008281 [DEBUG] http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (3.830074ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (3.830074ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (4.207059ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.009001 [DEBUG] http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (4.207059ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.009643 [DEBUG] http: Request /v1/node/ec20657b-ac9a-c80f-9a87-4098c5d9d665?region=us-east-1&wait=60000ms (4.669875ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.010135 [DEBUG] http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (2.251914ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.010242 [DEBUG] http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (4.565396ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.010799 [DEBUG] http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (4.95177ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.010819 [DEBUG] http: Request /v1/node/2bb7c609-f7f5-de03-9609-51ed67e76470?region=us-east-1&wait=60000ms (3.275272ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.011289 [DEBUG] http: Request /v1/node/cfc3db5b-563e-9c80-4e1b-557ecfb1a977?region=us-east-1&wait=60000ms (1.768325ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec20657b-ac9a-c80f-9a87-4098c5d9d665?region=us-east-1&wait=60000ms (4.669875ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (2.251914ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (4.565396ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (4.95177ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/2bb7c609-f7f5-de03-9609-51ed67e76470?region=us-east-1&wait=60000ms (3.275272ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/cfc3db5b-563e-9c80-4e1b-557ecfb1a977?region=us-east-1&wait=60000ms (1.768325ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.013597 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (174.675µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (174.675µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (604.553µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.022745 [DEBUG] http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (604.553µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.023300 [DEBUG] http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (1.019073ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.023813 [DEBUG] http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (1.40664ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.025023 [DEBUG] http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (701.607µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.026284 [DEBUG] http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (716.549µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.027415 [DEBUG] http: Request /v1/node/5daf562c-9ccb-f55e-3fda-37fd0bfdd74c?region=us-east-1&wait=60000ms (2.02512ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.029125 [DEBUG] http: Request /v1/node/cfc3db5b-563e-9c80-4e1b-557ecfb1a977?region=us-east-1&wait=60000ms (1.536367ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.030307 [DEBUG] http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (4.556565ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.030771 [DEBUG] http: Request /v1/node/2bb7c609-f7f5-de03-9609-51ed67e76470?region=us-east-1&wait=60000ms (798.779µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.030929 [DEBUG] http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (2.05353ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.031354 [DEBUG] http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (2.143559ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.031486 [DEBUG] http: Request /v1/node/ec20657b-ac9a-c80f-9a87-4098c5d9d665?region=us-east-1&wait=60000ms (3.568656ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.031982 [DEBUG] http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (2.456053ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.032065 [DEBUG] http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (3.688143ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (1.019073ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.032696 [DEBUG] http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (3.039851ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (1.40664ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (701.607µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (716.549µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/5daf562c-9ccb-f55e-3fda-37fd0bfdd74c?region=us-east-1&wait=60000ms (2.02512ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/cfc3db5b-563e-9c80-4e1b-557ecfb1a977?region=us-east-1&wait=60000ms (1.536367ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (4.556565ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/2bb7c609-f7f5-de03-9609-51ed67e76470?region=us-east-1&wait=60000ms (798.779µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (2.05353ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (2.143559ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec20657b-ac9a-c80f-9a87-4098c5d9d665?region=us-east-1&wait=60000ms (3.568656ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (2.456053ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (3.688143ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (3.039851ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.037591 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (228.116µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (228.116µs) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:36.976002 [DEBUG] http: Request /v1/agent/health?type=server (1.133916ms) Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (1.133916ms) Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:39.313165 [DEBUG] http: Request /v1/job/helix2-emitter-production/versions?diffs=false®ion=us-east-1&stale=&wait=120000ms (4.643344ms) Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/helix2-emitter-production/versions?diffs=false®ion=us-east-1&stale=&wait=120000ms (4.643344ms) Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:39.327666 [DEBUG] http: Request /v1/job/helix2-emitter-production?region=us-east-1&stale=&wait=120000ms (321.544µs) Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/helix2-emitter-production?region=us-east-1&stale=&wait=120000ms (321.544µs) Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (234.403µs) Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:39.580685 [DEBUG] http: Request /v1/agent/health?type=client (234.403µs) Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:39.675826 [DEBUG] http: Request /v1/allocations?index=696220®ion=us-east-1&stale=&wait=120000ms (661.649061ms) Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: http: Request /v1/allocations?index=696220®ion=us-east-1&stale=&wait=120000ms (661.649061ms) Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:39.688672 [DEBUG] http: Request /v1/agent/health?type=server (459.85µs) Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (459.85µs) Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:40.219282 [DEBUG] http: Request /v1/agent/health?type=client (220.826µs) Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (220.826µs) Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:40.244322 [DEBUG] http: Request /v1/agent/health?type=client (208.558µs) Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (208.558µs) Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:40 [DEBUG] memberlist: TCP connection from=10.50.4.247:40080 Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40080 Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:40 [DEBUG] memberlist: TCP connection from=10.50.4.247:40082 Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40082 Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:40.610226 [DEBUG] http: Request /v1/allocations?index=696221®ion=us-east-1&stale=&wait=120000ms (352.745161ms) Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: http: Request /v1/allocations?index=696221®ion=us-east-1&stale=&wait=120000ms (352.745161ms) Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:40.770709 [DEBUG] http: Request /v1/agent/health?type=client (195.904µs) Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (195.904µs) Jan 12 15:52:42 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:42.122468 [DEBUG] http: Request /v1/agent/health?type=client (592.037µs) Jan 12 15:52:42 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (592.037µs) Jan 12 15:52:42 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:42.244712 [DEBUG] http: Request /v1/agent/health?type=server (524.794µs) Jan 12 15:52:42 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (524.794µs) Jan 12 15:52:43 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:43.491105 [DEBUG] http: Request /v1/agent/health?type=server (909.135µs) Jan 12 15:52:43 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (909.135µs) Jan 12 15:52:43 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:43.858387 [DEBUG] http: Request /v1/agent/health?type=server (949.047µs) Jan 12 15:52:43 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (949.047µs) Jan 12 15:52:44 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:44.545823 [DEBUG] http: Request /v1/agent/health?type=client (189.834µs) Jan 12 15:52:44 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (189.834µs) Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:45 [DEBUG] memberlist: TCP connection from=10.50.5.8:59244 Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59244 Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:45 [DEBUG] memberlist: TCP connection from=10.50.1.12:39848 Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.1.12:39848 Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:45.684107 [DEBUG] http: Request /v1/agent/health?type=server (773.292µs) Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (773.292µs) Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59246 Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:45 [DEBUG] memberlist: TCP connection from=10.50.5.8:59246 Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:45.774979 [DEBUG] http: Request /v1/job/wfoc-staging/allocations?all=false&index=696129®ion=us-east-1&stale=&wait=120000ms (20.477897ms) Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/wfoc-staging/allocations?all=false&index=696129®ion=us-east-1&stale=&wait=120000ms (20.477897ms) Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:45 [DEBUG] memberlist: TCP connection from=10.50.2.47:36522 Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.2.47:36522 Jan 12 15:52:46 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:46.286834 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (224.692µs) Jan 12 15:52:46 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (224.692µs) Jan 12 15:52:46 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:46.329205 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (200.015µs) Jan 12 15:52:46 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (200.015µs) Jan 12 15:52:46 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:46.978314 [DEBUG] http: Request /v1/agent/health?type=server (509.54µs) Jan 12 15:52:46 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (509.54µs) Jan 12 15:52:47 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:47.515690 [DEBUG] http: Request /v1/job/mitochondrion-staging?index=663818®ion=us-east-1&stale=&wait=120000ms (2m1.194231162s) Jan 12 15:52:47 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/mitochondrion-staging?index=663818®ion=us-east-1&stale=&wait=120000ms (2m1.194231162s) Jan 12 15:52:47 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:47.535438 [DEBUG] http: Request /v1/job/mitochondrion-staging/allocations?all=false&index=696134®ion=us-east-1&stale=&wait=120000ms (10.323071ms) Jan 12 15:52:47 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/mitochondrion-staging/allocations?all=false&index=696134®ion=us-east-1&stale=&wait=120000ms (10.323071ms) Jan 12 15:52:49 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:49.584270 [DEBUG] http: Request /v1/agent/health?type=client (667.487µs) Jan 12 15:52:49 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (667.487µs) Jan 12 15:52:49 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (559.558µs) Jan 12 15:52:49 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:49.692211 [DEBUG] http: Request /v1/agent/health?type=server (559.558µs) Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:50.174508 [DEBUG] http: Request /v1/status/leader (535.344µs) Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: http: Request /v1/status/leader (535.344µs) Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:50.223139 [DEBUG] http: Request /v1/agent/health?type=client (192.179µs) Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (192.179µs) Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:50.247697 [DEBUG] http: Request /v1/agent/health?type=client (177.957µs) Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (177.957µs) Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:50.337689 [DEBUG] http: Request /v1/evaluations?region=us-east-1&stale=&wait=120000ms (192.138891ms) Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: http: Request /v1/evaluations?region=us-east-1&stale=&wait=120000ms (192.138891ms) Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40176 Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:50 [DEBUG] memberlist: TCP connection from=10.50.4.247:40176 Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:50 [DEBUG] memberlist: TCP connection from=10.50.4.247:40178 Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40178 Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:50.773755 [DEBUG] http: Request /v1/agent/health?type=client (217.207µs) Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (217.207µs) Jan 12 15:52:52 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:52.125688 [DEBUG] http: Request /v1/agent/health?type=client (607.93µs) Jan 12 15:52:52 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (607.93µs) Jan 12 15:52:52 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (550.386µs) Jan 12 15:52:52 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:52.246526 [DEBUG] http: Request /v1/agent/health?type=server (550.386µs) Jan 12 15:52:53 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (454.884µs) Jan 12 15:52:53 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:53.493752 [DEBUG] http: Request /v1/agent/health?type=server (454.884µs) Jan 12 15:52:53 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:53.861673 [DEBUG] http: Request /v1/agent/health?type=server (572.724µs) Jan 12 15:52:53 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (572.724µs) Jan 12 15:52:54 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:54.548297 [DEBUG] http: Request /v1/agent/health?type=client (171.695µs) Jan 12 15:52:54 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (171.695µs) Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:55 [DEBUG] memberlist: TCP connection from=10.50.5.8:59324 Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59324 Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:55 [DEBUG] memberlist: TCP connection from=10.50.1.12:39906 Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.1.12:39906 Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:55.690939 [DEBUG] http: Request /v1/agent/health?type=server (3.959848ms) Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (3.959848ms) Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:55 [DEBUG] memberlist: TCP connection from=10.50.5.8:59326 Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59326 Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:55 [DEBUG] memberlist: TCP connection from=10.50.2.47:36592 Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.2.47:36592 Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:56.330196 [DEBUG] http: Request /v1/job/tricorder-production/allocations?all=false&index=695876®ion=us-east-1&stale=&wait=120000ms (2.238855ms) Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/tricorder-production/allocations?all=false&index=695876®ion=us-east-1&stale=&wait=120000ms (2.238855ms) Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:56.878232 [DEBUG] http: Request /v1/nodes?index=696031®ion=us-east-1&stale=&wait=10000ms (10.497649584s) Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: http: Request /v1/nodes?index=696031®ion=us-east-1&stale=&wait=10000ms (10.497649584s) Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:56.904322 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (272.713µs) Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (272.713µs) Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:56.925012 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (186.196µs) Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (186.196µs) Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:56.981661 [DEBUG] http: Request /v1/agent/health?type=server (1.194177ms) Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (1.194177ms) Jan 12 15:52:57 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.50.1.95:4648 Jan 12 15:52:57 ip-10-50-1-101 nomad[25850]: memberlist: Initiating push/pull sync with: 10.50.1.95:4648 Jan 12 15:52:59 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:59.587301 [DEBUG] http: Request /v1/agent/health?type=client (226.539µs) Jan 12 15:52:59 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (226.539µs) Jan 12 15:52:59 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:59.696220 [DEBUG] http: Request /v1/agent/health?type=server (752.867µs) Jan 12 15:52:59 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (752.867µs) Jan 12 15:52:59 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/members?region=us-east-1&wait=60000ms (480.612µs) Jan 12 15:52:59 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:52:59.938879 [DEBUG] http: Request /v1/agent/members?region=us-east-1&wait=60000ms (480.612µs) Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (382.354µs) Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:53:00.225118 [DEBUG] http: Request /v1/agent/health?type=client (382.354µs) Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:53:00.249966 [DEBUG] http: Request /v1/agent/health?type=client (174.588µs) Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (174.588µs) Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:53:00 [DEBUG] memberlist: TCP connection from=10.50.4.247:40258 Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:53:00 [DEBUG] memberlist: TCP connection from=10.50.4.247:40256 Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40258 Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40256 Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:53:00.777635 [DEBUG] http: Request /v1/agent/health?type=client (590.016µs) Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (590.016µs) Jan 12 15:53:01 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:53:01.340877 [DEBUG] http: Request /v1/job/ingestion-uat?index=693233®ion=us-east-1&stale=&wait=120000ms (2m4.841526643s) Jan 12 15:53:01 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/ingestion-uat?index=693233®ion=us-east-1&stale=&wait=120000ms (2m4.841526643s) Jan 12 15:53:01 ip-10-50-1-101 nomad[25850]: 2018/01/12 15:53:01.542596 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (190.684µs) ```

Server3:

``` Jan 12 15:50:06 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:09 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:09.906778 [INFO] client.gc: garbage collecting allocation fb7ccb1a-3ca1-79dd-b4a0-68c458ee656e due to new allocations and over max (50) Jan 12 15:50:09 ip-10-50-1-95 nomad[21952]: client.gc: garbage collecting allocation fb7ccb1a-3ca1-79dd-b4a0-68c458ee656e due to new allocations and over max (50) Jan 12 15:50:09 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:09.952000 [INFO] client.gc: garbage collecting allocation 4384639f-51b9-7204-e2d7-81c2b504bd97 due to new allocations and over max (50) Jan 12 15:50:09 ip-10-50-1-95 nomad[21952]: client.gc: garbage collecting allocation 4384639f-51b9-7204-e2d7-81c2b504bd97 due to new allocations and over max (50) Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:10 [INFO] (runner) creating new runner (dry: false, once: false) Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: (runner) creating new runner (dry: false, once: false) Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:10 [INFO] (runner) creating watcher Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: (runner) creating watcher Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:10.447404 [ERR] client: alloc "2c650187-be72-ec50-1197-5fef0a26fe44", task "mmx" failed to build task's template manager: failed to read template: open /var/nomad/alloc/2c650187-be72-ec50-1197-5fef0a26fe44/mmx/local/config.ctmpl: no such file or directory Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: client: alloc "2c650187-be72-ec50-1197-5fef0a26fe44", task "mmx" failed to build task's template manager: failed to read template: open /var/nomad/alloc/2c650187-be72-ec50-1197-5fef0a26fe44/mmx/local/config.ctmpl: no such file or directory Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:10.447705 [INFO] client.gc: marking allocation 2c650187-be72-ec50-1197-5fef0a26fe44 for GC Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: client.gc: marking allocation 2c650187-be72-ec50-1197-5fef0a26fe44 for GC Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:10.635318 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:10.672615 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:10.718613 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:10 [ERR] raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40094->10.50.1.29:4647: i/o timeout Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40094->10.50.1.29:4647: i/o timeout Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:10.774199 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:10.815519 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:11 [INFO] (runner) creating new runner (dry: false, once: false) Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:11 [INFO] (runner) creating watcher Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]: (runner) creating new runner (dry: false, once: false) Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]: (runner) creating watcher Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:11.248048 [ERR] client: alloc "d47252ba-d489-069c-063a-74ca1e16380e", task "mmx" failed to build task's template manager: failed to read template: open /var/nomad/alloc/d47252ba-d489-069c-063a-74ca1e16380e/mmx/local/config.ctmpl: no such file or directory Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]: client: alloc "d47252ba-d489-069c-063a-74ca1e16380e", task "mmx" failed to build task's template manager: failed to read template: open /var/nomad/alloc/d47252ba-d489-069c-063a-74ca1e16380e/mmx/local/config.ctmpl: no such file or directory Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:11.248443 [INFO] client.gc: marking allocation d47252ba-d489-069c-063a-74ca1e16380e for GC Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]: client.gc: marking allocation d47252ba-d489-069c-063a-74ca1e16380e for GC Jan 12 15:50:19 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:19.781007 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:19 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:19 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:19.835517 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:19 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:20 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:20 [ERR] raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40096->10.50.1.29:4647: i/o timeout Jan 12 15:50:20 ip-10-50-1-95 nomad[21952]: raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40096->10.50.1.29:4647: i/o timeout Jan 12 15:50:30 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:30 [ERR] raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40130->10.50.1.29:4647: i/o timeout Jan 12 15:50:30 ip-10-50-1-95 nomad[21952]: raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40130->10.50.1.29:4647: i/o timeout Jan 12 15:50:40 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:40 [ERR] raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40164->10.50.1.29:4647: i/o timeout Jan 12 15:50:40 ip-10-50-1-95 nomad[21952]: raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40164->10.50.1.29:4647: i/o timeout Jan 12 15:50:40 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:40.982217 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:40 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:41 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:41.017821 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:41 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation Jan 12 15:50:47 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:47.538499 [WARN] nomad: eval reached delivery limit, marking as failed Jan 12 15:50:47 ip-10-50-1-95 nomad[21952]: nomad: eval reached delivery limit, marking as failed Jan 12 15:50:50 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:50 [ERR] raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40172->10.50.1.29:4647: i/o timeout Jan 12 15:50:50 ip-10-50-1-95 nomad[21952]: raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40172->10.50.1.29:4647: i/o timeout Jan 12 15:50:56 ip-10-50-1-95 nomad[21952]: 2018/01/12 15:50:56 [INFO] raft: pipelining replication to peer {Voter 10.50.1.29:4647 10.50.1.29:4647} Jan 12 15:50:56 ip-10-50-1-95 nomad[21952]: raft: pipelining replication to peer {Voter 10.50.1.29:4647 10.50.1.29:4647} ```

I only have debug enabled on the first two, sorry. Probably makes it easier to read at least.

@dadgar direct ping as you closed it, you may be able to open it :)

welp.txt

Added this as a txt file because it's too big, but this is the allocs on a single master node, all unable to GC. You asked for this on #3604 where two other people are experiencing this.

I have a similar (though not identical) issue about nodes not collecting garbage correctly.

I am using Nomad 0.7.1/Consul 1.0.2/CentOS 7.4+/Docker CE.

My issue is that "/var/lib/docker" bloats up over time (I have image cleanup purposely set to 'false').

This fills up the disk and nomad reports the machine "ready" but no allocation go the machine as the disk capacity is exhausted.

I don't have the logs with me right now, but it keeps emitting a message along the following line: disk usage of 98 more than the threshold of 80 but not triggering a gc as no allocations are terminal.

So I have a node which is basically unusable but not being indicated as such by Nomad.

I know that '/var/lib/docker' bloating up is not really Nomad's problem, but just wanted to put this out there.

Regards, Shantanu

@apenney Thanks for the additional logs! In the future please attach them as text files or link a gist to make browsing this issue easier. We'll almost always want to download them to search/grep/etc.

@shantanugadgil As far as we can tell right now @apenney's issue is related to the servers GCing dead jobs. Your issue is specific to clients. Ensure you have not set docker.cleanup.image=true in your configs and reopen #3560 if you think it's the same issue. There may be something we're missing and closed too soon!

@apenney Hey I am sorry but I just can't reproduce this. I will reopen when there are clear reproduction steps. The reproduction steps will have to include the configuration and how to get to this state from a fresh cluster.

I have even tested by creating a periodic job that fails in the same fashion as your attached allocations but as soon as I hit the GC endpoint, the jobs and allocations get removed.

@apenney And if you post logs can you please post the Nomad Server logs and not the client's logs. After you do the curl -XPUT .../v1/system/gc you should see logs like:

    2018/01/16 23:10:11.956340 [DEBUG] http: Request /v1/system/gc (71.546µs)
    2018/01/16 23:10:11.956464 [DEBUG] worker: dequeued evaluation fd8b7e4c-ca04-0781-91f9-b3b847638f8d
    2018/01/16 23:10:11.956494 [DEBUG] sched.core: forced job GC
    2018/01/16 23:10:11.956653 [DEBUG] sched.core: job GC: 13 jobs, 13 evaluations, 13 allocs eligible
    2018/01/16 23:10:11.962373 [DEBUG] sched.core: forced eval GC
    2018/01/16 23:10:11.962441 [DEBUG] sched.core: forced deployment GC
    2018/01/16 23:10:11.962450 [DEBUG] sched.core: forced node GC

This should be logged if you are in DEBUG level logs regardless of if anything gets garbage collected. So if you don't see that something is odd with your setup.

@dadgar thanks to the wonderful @jippi we've fixed this!

On our server nodes we had:

    enabled_schedulers = ["service","batch","system"]
    num_schedulers = 2

Removing these two lines immediately fixed the issue, with it starting to delete thousands of piled up batch jobs. We don't really understand -why-, unless it was starved by a low num_schedulers stopping it ever getting to gc?

I just wanted to let you know, and let the world know, in case someone else stumbles over this bug report.

@apenney glad this was fixed for you.

And you guessed right - see the docs for num_schedulers. By setting it to 2, the ability of nomad servers to be able to run scheduler workers in parallel was severely reduced. Garbage collection is done as an internal scheduled job as well so that explains why.

Had the exact same problem, running only 1 scheduler, on 1vCPU test nodes. After removing both lines mentioned above, gc started immediately. After putting both config lines back, GC still works.

On 1vCPU, the default is still 1 scheduler, so it doesn't seem to be the num_schedulers that's creating the issue. (or not alone)

My test setup was updated in-place/with rolling redeploys from 5.6 and did almost all versions until 7.1.

Haven't restarted all my nodes yet, but I'll report back here if I'm able to reproduce the issue after restarting all nodes again.

@dadgar @preetapan I was able to reproduce the problem again in my setup by performing a rolling re-deploy.

Removing the enabled_schedulers setting on one of the nodes and restarting nomad fixed the issue again.

When the issue is active, server logs (in debug mode) show the ../system/gc call coming in, but there are no other log entries about GC.

I think I found the issue, the default value for enabled_schedulers adds the _core scheduler type.

https://github.com/hashicorp/nomad/blob/b3ec943a2e2c0f98db4be0fc092cde5b072f5ec5/nomad/config.go#L363

But if a custom value is provided, the _core scheduler type is never added to this list by the config parser.

@groggemans Great find. This was definitely not intended and we will get this fixed up for 0.8

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / nomad