search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.82k stars 1.94k forks source link

DEAD Task (desired run) after Power Off #8893

Closed jozef-slezak closed 3 years ago

jozef-slezak commented 4 years ago

Nomad version

Nomad v0.12.3 (2db8abd9620dd41cb7bfe399551ba0f7824b3f61)

Operating system and Environment details

Centos 7

Issue

DEAD job/task after power of restart

Reproduction steps

Repeat power off a Virutal Box VM with Nomad and running Jobs

Alloc Status

nomad alloc status 3fd2dce4-6ac0-a889-d03a-8259643ba3d9
ID                  = 3fd2dce4-6ac0-a889-d03a-8259643ba3d9
Eval ID             = 5c0f1af8
Name                = web-worker[0]
Node ID             = f904bc63
Node Name           = demo-77
Job ID              = web-worker
Job Version         = 0
Client Status       = complete
Client Description  = All tasks have completed
Desired Status      = run
Desired Description = <none>
Created             = 24m28s ago
Modified            = 7m55s ago
Deployment ID       = c61ded2c
Deployment Health   = unhealthy

Task "worker" is "dead"
Task Resources
CPU         Memory           Disk     Addresses
28/100 MHz  242 MiB/100 MiB  150 MiB  WEB-WORKER: 127.0.0.1:3202

Task Events:
Started At     = 2020-09-16T08:51:53Z
Finished At    = 2020-09-16T09:01:58Z
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                  Type                   Description
2020-09-16T09:01:58Z  Killed                 Task successfully killed
2020-09-16T09:01:58Z  Terminated             Exit Code: 0, Exit Message: "executor: error waiting on process: rpc error: code = Canceled desc = grpc: the client connection is closing"
2020-09-16T09:01:58Z  Killing                Sent interrupt. Waiting 5s before force killing
2020-09-16T09:01:25Z  Started                Task started by client
2020-09-16T09:01:23Z  Task Setup             Building Task Directory
2020-09-16T08:56:51Z  Alloc Unhealthy        Task not running for min_healthy_time of 10s by deadline
2020-09-16T08:51:53Z  Started                Task started by client
2020-09-16T08:51:50Z  Task Setup             Building Task Directory
2020-09-16T08:51:50Z  Received               Task received by client
2020-09-16T08:51:47Z  Failed Restoring Task  failed to restore task; will not run until server is contacted
nomad alloc status ee0839ea-ddee-de61-8fbe-6923ced634b2
ID                  = ee0839ea-ddee-de61-8fbe-6923ced634b2
Eval ID             = 6db0785e
Name                = web-delayed-job-web-cache.web-delayed-job-web-cache[0]
Node ID             = f904bc63
Node Name           = demo-inno77
Job ID              = web-delayed-job-web-cache
Job Version         = 0
Client Status       = complete
Client Description  = All tasks have completed
Desired Status      = run
Desired Description = <none>
Created             = 35m32s ago
Modified            = 19m2s ago
Deployment ID       = f49dfcb6
Deployment Health   = unhealthy

Task "web-delayed-job-web-cache" is "dead"
Task Resources
CPU         Memory           Disk     Addresses
28/100 MHz  282 MiB/100 MiB  150 MiB  WEB_DELAYED_JOB_WEB_CACHE: 127.0.0.1:3203

Task Events:
Started At     = 2020-09-16T08:51:54Z
Finished At    = 2020-09-16T09:01:59Z
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                  Type                   Description
2020-09-16T09:01:59Z  Killed                 Task successfully killed
2020-09-16T09:01:59Z  Terminated             Exit Code: 0, Exit Message: "executor: error waiting on process: rpc error: code = Canceled desc = grpc: the client connection is closing"
2020-09-16T09:01:58Z  Killing                Sent interrupt. Waiting 5s before force killing
2020-09-16T09:01:25Z  Started                Task started by client
2020-09-16T09:01:23Z  Task Setup             Building Task Directory
2020-09-16T08:56:52Z  Alloc Unhealthy        Task not running for min_healthy_time of 10s by deadline
2020-09-16T08:51:54Z  Started                Task started by client
2020-09-16T08:51:51Z  Task Setup             Building Task Directory
2020-09-16T08:51:51Z  Received               Task received by client
2020-09-16T08:51:47Z  Failed Restoring Task  failed to restore task; will not run until server is contacted
nomad alloc status a8f0bb1e-2e17-5279-025d-17d79ca1d164
ID                  = a8f0bb1e-2e17-5279-025d-17d79ca1d164
Eval ID             = 55e81068
Name                = web-workflow-worker.web-workflow-worker[0]
Node ID             = f904bc63
Node Name           = demo-inno77
Job ID              = web-workflow-worker
Job Version         = 0
Client Status       = complete
Client Description  = All tasks have completed
Desired Status      = run
Desired Description = <none>
Created             = 36m20s ago
Modified            = 19m58s ago
Deployment ID       = 73b6d45d
Deployment Health   = unhealthy

Task "web-mq-workflow-worker" is "dead"
Task Resources
CPU        Memory           Disk     Addresses
0/100 MHz  246 MiB/100 MiB  150 MiB  WEB_MQ_WORKFLOW: 127.0.0.1:3200

Task Events:
Started At     = 2020-09-16T08:51:53Z
Finished At    = 2020-09-16T09:01:45Z
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                  Type                   Description
2020-09-16T09:01:44Z  Killed                 Task successfully killed
2020-09-16T09:01:44Z  Terminated             Exit Code: 0, Exit Message: "executor: error waiting on process: rpc error: code = Canceled desc = grpc: the client connection is closing"
2020-09-16T09:01:33Z  Killing                Sent interrupt. Waiting 5s before force killing
2020-09-16T09:01:25Z  Started                Task started by client
2020-09-16T09:01:23Z  Task Setup             Building Task Directory
2020-09-16T08:56:52Z  Alloc Unhealthy        Task not running for min_healthy_time of 10s by deadline
2020-09-16T08:51:53Z  Started                Task started by client
2020-09-16T08:51:51Z  Task Setup             Building Task Directory
2020-09-16T08:51:50Z  Received               Task received by client
2020-09-16T08:51:47Z  Failed Restoring Task  failed to restore task; will not run until server is contacted

Client logs

``` node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=7f81adeb-6e79-c377-4c77-d47eb255eb91 2020-09-16T08:59:50.196Z [DEBUG] worker: submitted plan for evaluation: eval_id=7f81adeb-6e79-c377-4c77-d47eb255eb91 2020-09-16T08:59:50.196Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=7f81adeb-6e79-c377-4c77-d47eb255eb91 2020-09-16T08:59:50.196Z [DEBUG] worker.service_sched: refresh forced: eval_id=7f81adeb-6e79-c377-4c77-d47eb255eb91 job_id=pre-extractor namespace=default 2020-09-16T08:59:50.199Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T08:59:50.200Z [DEBUG] worker.service_sched: setting eval status: eval_id=7f81adeb-6e79-c377-4c77-d47eb255eb91 job_id=pre-extractor namespace=default status=failed 2020-09-16T08:59:50.201Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T08:59:50.201Z [DEBUG] worker: ack evaluation: eval_id=7f81adeb-6e79-c377-4c77-d47eb255eb91 2020-09-16T08:59:50.201Z [DEBUG] worker: dequeued evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.201Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default algorithm=binpack 2020-09-16T08:59:50.201Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.202Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.202Z [DEBUG] worker: submitted plan for evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.202Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.202Z [DEBUG] worker.service_sched: refresh forced: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default 2020-09-16T08:59:50.202Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default algorithm=binpack 2020-09-16T08:59:50.202Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.202Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.202Z [DEBUG] worker: submitted plan for evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.202Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.202Z [DEBUG] worker.service_sched: refresh forced: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default 2020-09-16T08:59:50.202Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default algorithm=binpack 2020-09-16T08:59:50.202Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.203Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.203Z [DEBUG] worker: submitted plan for evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.203Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.203Z [DEBUG] worker.service_sched: refresh forced: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default 2020-09-16T08:59:50.203Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default algorithm=binpack 2020-09-16T08:59:50.203Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.204Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.204Z [DEBUG] worker: submitted plan for evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.204Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.204Z [DEBUG] worker.service_sched: refresh forced: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default 2020-09-16T08:59:50.204Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default algorithm=binpack 2020-09-16T08:59:50.204Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.204Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.204Z [DEBUG] worker: submitted plan for evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.204Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.204Z [DEBUG] worker.service_sched: refresh forced: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default 2020-09-16T08:59:50.206Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T08:59:50.206Z [DEBUG] worker.service_sched: setting eval status: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default status=failed 2020-09-16T08:59:50.209Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T08:59:50.209Z [DEBUG] worker: ack evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.209Z [DEBUG] worker: dequeued evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.209Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default algorithm=binpack 2020-09-16T08:59:50.209Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.210Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.210Z [DEBUG] worker: submitted plan for evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.210Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.210Z [DEBUG] worker.service_sched: refresh forced: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default 2020-09-16T08:59:50.210Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default algorithm=binpack 2020-09-16T08:59:50.210Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.210Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.210Z [DEBUG] worker: submitted plan for evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.210Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.210Z [DEBUG] worker.service_sched: refresh forced: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default 2020-09-16T08:59:50.210Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default algorithm=binpack 2020-09-16T08:59:50.211Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.211Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.211Z [DEBUG] worker: submitted plan for evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.211Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.211Z [DEBUG] worker.service_sched: refresh forced: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default 2020-09-16T08:59:50.211Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default algorithm=binpack 2020-09-16T08:59:50.211Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.212Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.212Z [DEBUG] worker: submitted plan for evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.212Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.212Z [DEBUG] worker.service_sched: refresh forced: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default 2020-09-16T08:59:50.212Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default algorithm=binpack 2020-09-16T08:59:50.212Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.212Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.212Z [DEBUG] worker: submitted plan for evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.212Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.212Z [DEBUG] worker.service_sched: refresh forced: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default 2020-09-16T08:59:50.214Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T08:59:50.214Z [DEBUG] worker.service_sched: setting eval status: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default status=failed 2020-09-16T08:59:50.216Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T08:59:50.216Z [DEBUG] worker: ack evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:51.202Z [TRACE] client: next heartbeat: period=19.909570784s 2020-09-16T08:59:51.329Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/summary?index=280 duration=5m6.68468646s 2020-09-16T08:59:53.320Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=175.091µs 2020-09-16T08:59:54.260Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43600 2020-09-16T08:59:54.388Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=195.797µs 2020-09-16T08:59:58.623Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/allocations?index=1177 duration=5m15.979758964s 2020-09-16T09:00:03.326Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=133.113µs 2020-09-16T09:00:04.265Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43626 2020-09-16T09:00:04.396Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=211.937µs 2020-09-16T09:00:07.637Z [DEBUG] http: request complete: method=GET path=/v1/allocation/3a870e8d-7a49-1810-594b-cbc1a05f594e?index=512 duration=5m3.828755715s 2020-09-16T09:00:11.113Z [TRACE] client: next heartbeat: period=19.51271536s 2020-09-16T09:00:13.328Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=96.55µs 2020-09-16T09:00:14.266Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43648 2020-09-16T09:00:14.398Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=158.839µs 2020-09-16T09:00:14.444Z [DEBUG] http: request complete: method=GET path=/v1/allocation/e02922da-1390-dc79-e9a2-0519e2d2bc19?index=357 duration=5m11.327258984s 2020-09-16T09:00:23.330Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=111.659µs 2020-09-16T09:00:24.266Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43672 2020-09-16T09:00:24.400Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=207.749µs 2020-09-16T09:00:29.440Z [DEBUG] http: request complete: method=GET path=/v1/allocation/e02922da-1390-dc79-e9a2-0519e2d2bc19?index=357 duration=5m15.578093878s 2020-09-16T09:00:30.626Z [TRACE] client: next heartbeat: period=13.643576207s 2020-09-16T09:00:33.332Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=96.962µs 2020-09-16T09:00:34.267Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43696 2020-09-16T09:00:34.403Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=118.814µs 2020-09-16T09:00:38.705Z [DEBUG] http: request complete: method=GET path=/v1/job/web-delayed-job/evaluations?index=1202 duration=5m16.517219227s 2020-09-16T09:00:43.334Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=193.542µs 2020-09-16T09:00:44.101Z [DEBUG] http: request complete: method=GET path=/v1/job/web-delayed-job/versions?diffs=true&index=141 duration=5m8.60691639s 2020-09-16T09:00:44.268Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43720 2020-09-16T09:00:44.271Z [TRACE] client: next heartbeat: period=15.676066348s 2020-09-16T09:00:44.405Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=121.203µs 2020-09-16T09:00:50.174Z [DEBUG] worker: dequeued evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.174Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default algorithm=binpack 2020-09-16T09:00:50.174Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "activemq": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.175Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker: submitted plan for evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker.service_sched: refresh forced: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default 2020-09-16T09:00:50.175Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default algorithm=binpack 2020-09-16T09:00:50.175Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "activemq": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.175Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker: submitted plan for evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker.service_sched: refresh forced: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default 2020-09-16T09:00:50.175Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default algorithm=binpack 2020-09-16T09:00:50.175Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "activemq": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.175Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker: submitted plan for evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker.service_sched: refresh forced: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default 2020-09-16T09:00:50.175Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default algorithm=binpack 2020-09-16T09:00:50.175Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "activemq": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.176Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.176Z [DEBUG] worker: submitted plan for evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.176Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.176Z [DEBUG] worker.service_sched: refresh forced: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default 2020-09-16T09:00:50.176Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default algorithm=binpack 2020-09-16T09:00:50.176Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "activemq": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.176Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.177Z [DEBUG] worker: submitted plan for evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.177Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.177Z [DEBUG] worker.service_sched: refresh forced: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default 2020-09-16T09:00:50.181Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T09:00:50.181Z [DEBUG] worker.service_sched: setting eval status: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default status=failed 2020-09-16T09:00:50.183Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T09:00:50.183Z [DEBUG] worker: ack evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.183Z [DEBUG] worker: dequeued evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.183Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default algorithm=binpack 2020-09-16T09:00:50.183Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "web-admin-worker": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.184Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.184Z [DEBUG] worker: submitted plan for evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.184Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.184Z [DEBUG] worker.service_sched: refresh forced: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default 2020-09-16T09:00:50.184Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default algorithm=binpack 2020-09-16T09:00:50.184Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "web-admin-worker": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.184Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.184Z [DEBUG] worker: submitted plan for evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.184Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.184Z [DEBUG] worker.service_sched: refresh forced: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default 2020-09-16T09:00:50.184Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default algorithm=binpack 2020-09-16T09:00:50.184Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "web-admin-worker": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.185Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker: submitted plan for evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker.service_sched: refresh forced: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default 2020-09-16T09:00:50.185Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default algorithm=binpack 2020-09-16T09:00:50.185Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "web-admin-worker": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.185Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker: submitted plan for evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker.service_sched: refresh forced: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default 2020-09-16T09:00:50.185Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default algorithm=binpack 2020-09-16T09:00:50.185Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "web-admin-worker": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.185Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker: submitted plan for evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker.service_sched: refresh forced: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default 2020-09-16T09:00:50.188Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T09:00:50.188Z [DEBUG] worker.service_sched: setting eval status: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default status=failed 2020-09-16T09:00:50.190Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T09:00:50.190Z [DEBUG] worker: ack evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.190Z [DEBUG] worker: dequeued evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.190Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default algorithm=binpack 2020-09-16T09:00:50.190Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "pre-extractor": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.190Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.190Z [DEBUG] worker: submitted plan for evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.190Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.190Z [DEBUG] worker.service_sched: refresh forced: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default 2020-09-16T09:00:50.190Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default algorithm=binpack 2020-09-16T09:00:50.190Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "pre-extractor": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.191Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker: submitted plan for evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker.service_sched: refresh forced: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default 2020-09-16T09:00:50.191Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default algorithm=binpack 2020-09-16T09:00:50.191Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "pre-extractor": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.191Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker: submitted plan for evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker.service_sched: refresh forced: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default 2020-09-16T09:00:50.191Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default algorithm=binpack 2020-09-16T09:00:50.191Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "pre-extractor": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.191Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker: submitted plan for evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker.service_sched: refresh forced: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default 2020-09-16T09:00:50.191Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default algorithm=binpack 2020-09-16T09:00:50.192Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "pre-extractor": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.192Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.192Z [DEBUG] worker: submitted plan for evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.192Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.192Z [DEBUG] worker.service_sched: refresh forced: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default 2020-09-16T09:00:50.194Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T09:00:50.194Z [DEBUG] worker.service_sched: setting eval status: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default status=failed 2020-09-16T09:00:50.195Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T09:00:50.195Z [DEBUG] worker: ack evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.196Z [DEBUG] worker: dequeued evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default algorithm=binpack 2020-09-16T09:00:50.196Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.196Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [DEBUG] worker: submitted plan for evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [DEBUG] worker.service_sched: refresh forced: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default 2020-09-16T09:00:50.196Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default algorithm=binpack 2020-09-16T09:00:50.196Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.196Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [DEBUG] worker: submitted plan for evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [DEBUG] worker.service_sched: refresh forced: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default 2020-09-16T09:00:50.196Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default algorithm=binpack 2020-09-16T09:00:50.196Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.197Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.197Z [DEBUG] worker: submitted plan for evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.197Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.197Z [DEBUG] worker.service_sched: refresh forced: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default 2020-09-16T09:00:50.197Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default algorithm=binpack 2020-09-16T09:00:50.197Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.197Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.197Z [DEBUG] worker: submitted plan for evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.197Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.197Z [DEBUG] worker.service_sched: refresh forced: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default 2020-09-16T09:00:50.197Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default algorithm=binpack 2020-09-16T09:00:50.197Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.198Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.198Z [DEBUG] worker: submitted plan for evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.198Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.198Z [DEBUG] worker.service_sched: refresh forced: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default 2020-09-16T09:00:50.200Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T09:00:50.200Z [DEBUG] worker.service_sched: setting eval status: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default status=failed 2020-09-16T09:00:50.203Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T09:00:50.203Z [DEBUG] worker: ack evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.203Z [DEBUG] worker: dequeued evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.203Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default algorithm=binpack 2020-09-16T09:00:50.203Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.203Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.203Z [DEBUG] worker: submitted plan for evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.203Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.203Z [DEBUG] worker.service_sched: refresh forced: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default 2020-09-16T09:00:50.203Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default algorithm=binpack 2020-09-16T09:00:50.203Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.204Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.204Z [DEBUG] worker: submitted plan for evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.204Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.204Z [DEBUG] worker.service_sched: refresh forced: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default 2020-09-16T09:00:50.204Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default algorithm=binpack 2020-09-16T09:00:50.204Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.204Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.204Z [DEBUG] worker: submitted plan for evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.204Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.204Z [DEBUG] worker.service_sched: refresh forced: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default 2020-09-16T09:00:50.204Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default algorithm=binpack 2020-09-16T09:00:50.204Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.205Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.205Z [DEBUG] worker: submitted plan for evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.205Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.205Z [DEBUG] worker.service_sched: refresh forced: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default 2020-09-16T09:00:50.205Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default algorithm=binpack 2020-09-16T09:00:50.205Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.205Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.205Z [DEBUG] worker: submitted plan for evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.205Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.205Z [DEBUG] worker.service_sched: refresh forced: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default 2020-09-16T09:00:50.207Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T09:00:50.207Z [DEBUG] worker.service_sched: setting eval status: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default status=failed 2020-09-16T09:00:50.210Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T09:00:50.210Z [DEBUG] worker: ack evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:53.337Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=95.941µs 2020-09-16T09:00:54.268Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43744 2020-09-16T09:00:54.407Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=122.387µs ```

server logs

``` node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=7f81adeb-6e79-c377-4c77-d47eb255eb91 2020-09-16T08:59:50.196Z [DEBUG] worker: submitted plan for evaluation: eval_id=7f81adeb-6e79-c377-4c77-d47eb255eb91 2020-09-16T08:59:50.196Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=7f81adeb-6e79-c377-4c77-d47eb255eb91 2020-09-16T08:59:50.196Z [DEBUG] worker.service_sched: refresh forced: eval_id=7f81adeb-6e79-c377-4c77-d47eb255eb91 job_id=pre-extractor namespace=default 2020-09-16T08:59:50.199Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T08:59:50.200Z [DEBUG] worker.service_sched: setting eval status: eval_id=7f81adeb-6e79-c377-4c77-d47eb255eb91 job_id=pre-extractor namespace=default status=failed 2020-09-16T08:59:50.201Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T08:59:50.201Z [DEBUG] worker: ack evaluation: eval_id=7f81adeb-6e79-c377-4c77-d47eb255eb91 2020-09-16T08:59:50.201Z [DEBUG] worker: dequeued evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.201Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default algorithm=binpack 2020-09-16T08:59:50.201Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.202Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.202Z [DEBUG] worker: submitted plan for evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.202Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.202Z [DEBUG] worker.service_sched: refresh forced: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default 2020-09-16T08:59:50.202Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default algorithm=binpack 2020-09-16T08:59:50.202Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.202Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.202Z [DEBUG] worker: submitted plan for evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.202Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.202Z [DEBUG] worker.service_sched: refresh forced: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default 2020-09-16T08:59:50.202Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default algorithm=binpack 2020-09-16T08:59:50.202Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.203Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.203Z [DEBUG] worker: submitted plan for evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.203Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.203Z [DEBUG] worker.service_sched: refresh forced: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default 2020-09-16T08:59:50.203Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default algorithm=binpack 2020-09-16T08:59:50.203Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.204Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.204Z [DEBUG] worker: submitted plan for evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.204Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.204Z [DEBUG] worker.service_sched: refresh forced: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default 2020-09-16T08:59:50.204Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default algorithm=binpack 2020-09-16T08:59:50.204Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.204Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.204Z [DEBUG] worker: submitted plan for evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.204Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.204Z [DEBUG] worker.service_sched: refresh forced: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default 2020-09-16T08:59:50.206Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T08:59:50.206Z [DEBUG] worker.service_sched: setting eval status: eval_id=27656af3-fceb-891a-c410-c5d038322c27 job_id=fast-service namespace=default status=failed 2020-09-16T08:59:50.209Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T08:59:50.209Z [DEBUG] worker: ack evaluation: eval_id=27656af3-fceb-891a-c410-c5d038322c27 2020-09-16T08:59:50.209Z [DEBUG] worker: dequeued evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.209Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default algorithm=binpack 2020-09-16T08:59:50.209Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.210Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.210Z [DEBUG] worker: submitted plan for evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.210Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.210Z [DEBUG] worker.service_sched: refresh forced: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default 2020-09-16T08:59:50.210Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default algorithm=binpack 2020-09-16T08:59:50.210Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.210Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.210Z [DEBUG] worker: submitted plan for evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.210Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.210Z [DEBUG] worker.service_sched: refresh forced: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default 2020-09-16T08:59:50.210Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default algorithm=binpack 2020-09-16T08:59:50.211Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.211Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.211Z [DEBUG] worker: submitted plan for evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.211Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.211Z [DEBUG] worker.service_sched: refresh forced: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default 2020-09-16T08:59:50.211Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default algorithm=binpack 2020-09-16T08:59:50.211Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.212Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.212Z [DEBUG] worker: submitted plan for evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.212Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.212Z [DEBUG] worker.service_sched: refresh forced: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default 2020-09-16T08:59:50.212Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default algorithm=binpack 2020-09-16T08:59:50.212Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T08:59:50.212Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.212Z [DEBUG] worker: submitted plan for evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.212Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:50.212Z [DEBUG] worker.service_sched: refresh forced: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default 2020-09-16T08:59:50.214Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T08:59:50.214Z [DEBUG] worker.service_sched: setting eval status: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b job_id=monitoring namespace=default status=failed 2020-09-16T08:59:50.216Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T08:59:50.216Z [DEBUG] worker: ack evaluation: eval_id=d9fbd289-d489-6924-1be0-5469884f6e9b 2020-09-16T08:59:51.202Z [TRACE] client: next heartbeat: period=19.909570784s 2020-09-16T08:59:51.329Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/summary?index=280 duration=5m6.68468646s 2020-09-16T08:59:53.320Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=175.091µs 2020-09-16T08:59:54.260Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43600 2020-09-16T08:59:54.388Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=195.797µs 2020-09-16T08:59:58.623Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/allocations?index=1177 duration=5m15.979758964s 2020-09-16T09:00:03.326Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=133.113µs 2020-09-16T09:00:04.265Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43626 2020-09-16T09:00:04.396Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=211.937µs 2020-09-16T09:00:07.637Z [DEBUG] http: request complete: method=GET path=/v1/allocation/3a870e8d-7a49-1810-594b-cbc1a05f594e?index=512 duration=5m3.828755715s 2020-09-16T09:00:11.113Z [TRACE] client: next heartbeat: period=19.51271536s 2020-09-16T09:00:13.328Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=96.55µs 2020-09-16T09:00:14.266Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43648 2020-09-16T09:00:14.398Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=158.839µs 2020-09-16T09:00:14.444Z [DEBUG] http: request complete: method=GET path=/v1/allocation/e02922da-1390-dc79-e9a2-0519e2d2bc19?index=357 duration=5m11.327258984s 2020-09-16T09:00:23.330Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=111.659µs 2020-09-16T09:00:24.266Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43672 2020-09-16T09:00:24.400Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=207.749µs 2020-09-16T09:00:29.439Z [DEBUG] http: request complete: method=GET path=/v1/allocation/e02922da-1390-dc79-e9a2-0519e2d2bc19?index=357 duration=5m15.578093878s 2020-09-16T09:00:30.626Z [TRACE] client: next heartbeat: period=13.643576207s 2020-09-16T09:00:33.332Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=96.962µs 2020-09-16T09:00:34.267Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43696 2020-09-16T09:00:34.403Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=118.814µs 2020-09-16T09:00:38.705Z [DEBUG] http: request complete: method=GET path=/v1/job/web-delayed-job/evaluations?index=1202 duration=5m16.517219227s 2020-09-16T09:00:43.334Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=193.542µs 2020-09-16T09:00:44.101Z [DEBUG] http: request complete: method=GET path=/v1/job/web-delayed-job/versions?diffs=true&index=141 duration=5m8.60691639s 2020-09-16T09:00:44.268Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43720 2020-09-16T09:00:44.271Z [TRACE] client: next heartbeat: period=15.676066348s 2020-09-16T09:00:44.405Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=121.203µs 2020-09-16T09:00:50.174Z [DEBUG] worker: dequeued evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.174Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default algorithm=binpack 2020-09-16T09:00:50.174Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "activemq": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.175Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker: submitted plan for evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker.service_sched: refresh forced: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default 2020-09-16T09:00:50.175Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default algorithm=binpack 2020-09-16T09:00:50.175Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "activemq": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.175Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker: submitted plan for evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker.service_sched: refresh forced: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default 2020-09-16T09:00:50.175Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default algorithm=binpack 2020-09-16T09:00:50.175Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "activemq": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.175Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker: submitted plan for evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.175Z [DEBUG] worker.service_sched: refresh forced: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default 2020-09-16T09:00:50.175Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default algorithm=binpack 2020-09-16T09:00:50.175Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "activemq": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.176Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.176Z [DEBUG] worker: submitted plan for evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.176Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.176Z [DEBUG] worker.service_sched: refresh forced: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default 2020-09-16T09:00:50.176Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default algorithm=binpack 2020-09-16T09:00:50.176Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "activemq": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.177Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.177Z [DEBUG] worker: submitted plan for evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.177Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.177Z [DEBUG] worker.service_sched: refresh forced: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default 2020-09-16T09:00:50.181Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T09:00:50.181Z [DEBUG] worker.service_sched: setting eval status: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 job_id=activemq namespace=default status=failed 2020-09-16T09:00:50.183Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T09:00:50.183Z [DEBUG] worker: ack evaluation: eval_id=3f3c635b-f3ec-0e76-5625-7ea3690f78a5 2020-09-16T09:00:50.183Z [DEBUG] worker: dequeued evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.183Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default algorithm=binpack 2020-09-16T09:00:50.183Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "web-admin-worker": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.184Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.184Z [DEBUG] worker: submitted plan for evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.184Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.184Z [DEBUG] worker.service_sched: refresh forced: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default 2020-09-16T09:00:50.184Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default algorithm=binpack 2020-09-16T09:00:50.184Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "web-admin-worker": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.184Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.184Z [DEBUG] worker: submitted plan for evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.184Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.184Z [DEBUG] worker.service_sched: refresh forced: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default 2020-09-16T09:00:50.184Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default algorithm=binpack 2020-09-16T09:00:50.184Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "web-admin-worker": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.185Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker: submitted plan for evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker.service_sched: refresh forced: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default 2020-09-16T09:00:50.185Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default algorithm=binpack 2020-09-16T09:00:50.185Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "web-admin-worker": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.185Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker: submitted plan for evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker.service_sched: refresh forced: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default 2020-09-16T09:00:50.185Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default algorithm=binpack 2020-09-16T09:00:50.185Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "web-admin-worker": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.185Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker: submitted plan for evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.185Z [DEBUG] worker.service_sched: refresh forced: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default 2020-09-16T09:00:50.188Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T09:00:50.188Z [DEBUG] worker.service_sched: setting eval status: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc job_id=web-admin-worker namespace=default status=failed 2020-09-16T09:00:50.190Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T09:00:50.190Z [DEBUG] worker: ack evaluation: eval_id=16eda51d-afcd-67c9-f5e7-8a606d8db6dc 2020-09-16T09:00:50.190Z [DEBUG] worker: dequeued evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.190Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default algorithm=binpack 2020-09-16T09:00:50.190Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "pre-extractor": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.190Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.190Z [DEBUG] worker: submitted plan for evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.190Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.190Z [DEBUG] worker.service_sched: refresh forced: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default 2020-09-16T09:00:50.190Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default algorithm=binpack 2020-09-16T09:00:50.190Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "pre-extractor": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.191Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker: submitted plan for evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker.service_sched: refresh forced: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default 2020-09-16T09:00:50.191Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default algorithm=binpack 2020-09-16T09:00:50.191Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "pre-extractor": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.191Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker: submitted plan for evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker.service_sched: refresh forced: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default 2020-09-16T09:00:50.191Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default algorithm=binpack 2020-09-16T09:00:50.191Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "pre-extractor": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.191Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker: submitted plan for evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.191Z [DEBUG] worker.service_sched: refresh forced: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default 2020-09-16T09:00:50.191Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default algorithm=binpack 2020-09-16T09:00:50.191Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "pre-extractor": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.192Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.192Z [DEBUG] worker: submitted plan for evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.192Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.192Z [DEBUG] worker.service_sched: refresh forced: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default 2020-09-16T09:00:50.194Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T09:00:50.194Z [DEBUG] worker.service_sched: setting eval status: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 job_id=pre-extractor namespace=default status=failed 2020-09-16T09:00:50.195Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T09:00:50.195Z [DEBUG] worker: ack evaluation: eval_id=9323c495-5c1c-ca1d-9d26-8055d408aa52 2020-09-16T09:00:50.196Z [DEBUG] worker: dequeued evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default algorithm=binpack 2020-09-16T09:00:50.196Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.196Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [DEBUG] worker: submitted plan for evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [DEBUG] worker.service_sched: refresh forced: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default 2020-09-16T09:00:50.196Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default algorithm=binpack 2020-09-16T09:00:50.196Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.196Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [DEBUG] worker: submitted plan for evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.196Z [DEBUG] worker.service_sched: refresh forced: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default 2020-09-16T09:00:50.196Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default algorithm=binpack 2020-09-16T09:00:50.196Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.197Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.197Z [DEBUG] worker: submitted plan for evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.197Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.197Z [DEBUG] worker.service_sched: refresh forced: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default 2020-09-16T09:00:50.197Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default algorithm=binpack 2020-09-16T09:00:50.197Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.197Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.197Z [DEBUG] worker: submitted plan for evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.197Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.197Z [DEBUG] worker.service_sched: refresh forced: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default 2020-09-16T09:00:50.197Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default algorithm=binpack 2020-09-16T09:00:50.197Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "fast-service": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.198Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.198Z [DEBUG] worker: submitted plan for evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.198Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.198Z [DEBUG] worker.service_sched: refresh forced: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default 2020-09-16T09:00:50.200Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T09:00:50.200Z [DEBUG] worker.service_sched: setting eval status: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a job_id=fast-service namespace=default status=failed 2020-09-16T09:00:50.203Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T09:00:50.203Z [DEBUG] worker: ack evaluation: eval_id=ac681597-b7ff-69cd-26a5-3b9094807c8a 2020-09-16T09:00:50.203Z [DEBUG] worker: dequeued evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.203Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default algorithm=binpack 2020-09-16T09:00:50.203Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.203Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.203Z [DEBUG] worker: submitted plan for evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.203Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.203Z [DEBUG] worker.service_sched: refresh forced: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default 2020-09-16T09:00:50.203Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default algorithm=binpack 2020-09-16T09:00:50.203Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.204Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.204Z [DEBUG] worker: submitted plan for evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.204Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.204Z [DEBUG] worker.service_sched: refresh forced: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default 2020-09-16T09:00:50.204Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default algorithm=binpack 2020-09-16T09:00:50.204Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.204Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.204Z [DEBUG] worker: submitted plan for evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.204Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.204Z [DEBUG] worker.service_sched: refresh forced: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default 2020-09-16T09:00:50.204Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default algorithm=binpack 2020-09-16T09:00:50.204Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.205Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.205Z [DEBUG] worker: submitted plan for evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.205Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.205Z [DEBUG] worker.service_sched: refresh forced: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default 2020-09-16T09:00:50.205Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default algorithm=binpack 2020-09-16T09:00:50.205Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "monitoring": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 0) (canary 0)" 2020-09-16T09:00:50.205Z [DEBUG] nomad: plan for node rejected: node_id=f904bc63-d590-81b9-0bf7-55c76891673d reason="reserved port collision" eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.205Z [DEBUG] worker: submitted plan for evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.205Z [DEBUG] worker: refreshing state: refresh_index=1231 eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:50.205Z [DEBUG] worker.service_sched: refresh forced: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default 2020-09-16T09:00:50.207Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T09:00:50.207Z [DEBUG] worker.service_sched: setting eval status: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 job_id=monitoring namespace=default status=failed 2020-09-16T09:00:50.210Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T09:00:50.210Z [DEBUG] worker: ack evaluation: eval_id=362d379c-e904-d883-06da-e5d1353a9a47 2020-09-16T09:00:53.337Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=95.941µs 2020-09-16T09:00:54.268Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:43744 2020-09-16T09:00:54.407Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=122.387µs ```
jozef-slezak commented 4 years ago

Root cause of this problem (see following app logs) - nomad started the task but we do not see it Nomad console:

Errno::EADDRINUSE: Address already in use - bind(2) for "0.0.0.0" port 3200
systemctl status nomad | grep 3200
           ├─8812 puma 3.12.0 (tcp://0.0.0.0:3200) [workflow worker:3200]                                                                                                                                              
           ├─8813 /bin/bash -c cd /opt/web/current; exec &> >(tee -i -a /var/log/innrm/web_mq_workflow_worker-127.0.0.1-3200.log); exec /bin/bash ./bin/production_run_worker_mq.sh --type workflow
           ├─8814 tee -i -a /var/log/innrm/web_mq_workflow_worker-127.0.0.1-3200.log

Nomad console displays duplicit alocation that will never succeed a therefore it is DEAD.

jozef-slezak commented 4 years ago

Nomad config:

addresses = {
  http = "0.0.0.0"
}
advertise = {
  http = "127.0.0.1"
  rpc = "127.0.0.1"
  serf = "127.0.0.1"
}
bind_addr = "127.0.0.1"
client = {
  enabled = true
  network_interface = "lo"
  options = {
    driver.raw_exec.enable = 1
    driver.raw_exec.no_cgroups = 1
  }
}
data_dir = "/var/lib/nomad"
datacenter = "dc1"
disable_update_check = true
log_level = "INFO"
consul {
  server_auto_join    = false
}
server = {
bootstrap_expect = 1
  enabled = true
  encrypt = "..."
  server_join {
    retry_join = [ "127.0.0.1" ]
    retry_max = 0 # infinite
    retry_interval = "3s"
  }
}

Systemd nomad.service

systemctl show nomad.service | grep Kill
KillMode=control-group
KillSignal=15
jozef-slezak commented 4 years ago

Relates to #5919

tgross commented 4 years ago

Hi @jozef-slezak! From what I can see here, the sequence seems like it was:

A few things we could use from you:

jozef-slezak commented 4 years ago

I needed to repeat same test again and found #8899. Checked eligible using web console and reproduce same behaviour - address already in use (wrong reconciliation after restart using Virtual Box power off). I modified also nomad.hcl and nomad.service.

Application log (different job on port 3201

Errno::EADDRINUSE: Address already in use - bind(2) for "0.0.0.0" port 3201
/opt/web/web_customization/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/binder.rb:272:in `initialize'
/opt/web/web_customization/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/binder.rb:272:in `new'
/opt/web/web_customization/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/binder.rb:272:in `add_tcp_listener'
/opt/web/web_customization/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/binder.rb:105:in `block in parse'
/opt/web/web_customization/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/binder.rb:88:in `each'
/opt/web/web_customization/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/binder.rb:88:in `parse'
/opt/web/web_customization/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/runner.rb:151:in `load_and_bind'
/opt/web/web_customization/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/single.rb:96:in `run'
/opt/web/web_customization/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/launcher.rb:184:in `run'
/opt/web/web_customization/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/cli.rb:78:in `run'
/opt/web/web_core/lib/tasks/workers.rake:6:in `block (2 levels) in <top (required)>'
/opt/web/web_core/lib/tasks/workers.rake:14:in `block (2 levels) in <top (required)>'
/opt/web/web_core/lib/tasks/run.rake:25:in `block (3 levels) in <top (required)>'
/opt/web/web_customization/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/opt/web/web_customization/vendor/bundle/ruby/2.6.0/bin/ruby_executable_hooks:24:in `eval'
/opt/web/web_customization/vendor/bundle/ruby/2.6.0/bin/ruby_executable_hooks:24:in `<main>'
Tasks: TOP => workers:run
(See full trace by running task with --trace)

/usr/lib/systemd/system/nomad.service

  GNU nano 2.3.1                                                  File: /usr/lib/systemd/system/nomad.service

[Unit]
Description=HashiCorp Nomad
After=network-online.target
After=consul.service
Wants=consul.service
Requires=network-online.target

[Service]
Type=simple
ExecStart=/usr/sbin/nomad agent -config=/etc/nomad.d
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
LimitNOFILE=65536
LimitNPROC=65536

KillMode=process
KillSignal=SIGINT

[Install]
WantedBy=multi-user.target

/etc/nomad.d

addresses = {
  http = "0.0.0.0"
}
advertise = {
  http = "127.0.0.1"
  rpc = "127.0.0.1"
  serf = "127.0.0.1"
}
bind_addr = "127.0.0.1"
client = {
  enabled = true
  network_interface = "lo"
  options = {
    driver.raw_exec.enable = 1
    driver.raw_exec.no_cgroups = 0
  }
}
data_dir = "/var/lib/nomad"
datacenter = "dc1"
disable_update_check = true
log_level = "INFO"
consul {
  server_auto_join    = false
}
server = {
bootstrap_expect = 1
  enabled = true
  encrypt = "WvSG0VnCVqZ8j8CnZNYfpA=="
  server_join {
    retry_join = [ "127.0.0.1" ]
    retry_max = 0 # infinite
    retry_interval = "3s"
  }
}

Client logs:

``` te: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:34:17.520Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:34:17.526Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:34:17.526Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:34:17.531Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:34:17.531Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:34:22.948Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:55876 2020-09-16T14:34:23.075Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=232.834µs 2020-09-16T14:34:23.138Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=136.604µs 2020-09-16T14:34:26.823Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:34:26.823Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=running desired_status= 2020-09-16T14:34:26.825Z [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker error="rpc error: code = Unavailable desc = transport is closing" 2020-09-16T14:34:26.825Z [ERROR] client.alloc_runner.task_runner.task_hook.stats_hook: failed to start stats collection for task: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker error="rpc error: code = Canceled desc = grpc: the client connection is closing" 2020-09-16T14:34:26.826Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin process exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad pid=4342 2020-09-16T14:34:26.827Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker 2020-09-16T14:34:26.827Z [INFO] client.alloc_runner.task_runner: restarting task: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker reason="Restart within policy" delay=15.563263625s 2020-09-16T14:34:26.827Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker state=pending event=Restarting 2020-09-16T14:34:26.831Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:34:26.831Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=pending desired_status= 2020-09-16T14:34:26.838Z [DEBUG] consul.sync: sync complete: registered_services=0 deregistered_services=1 registered_checks=0 deregistered_checks=1 2020-09-16T14:34:27.017Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1514 duration=42.388254481s 2020-09-16T14:34:27.017Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1514 duration=25.749448496s 2020-09-16T14:34:27.017Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1514 duration=22.561463315s 2020-09-16T14:34:27.019Z [TRACE] client: next heartbeat: period=19.682077063s 2020-09-16T14:34:27.020Z [DEBUG] client: updated allocations: index=1523 total=34 pulled=6 filtered=28 2020-09-16T14:34:27.020Z [DEBUG] client: allocation updates: added=0 removed=0 updated=6 ignored=28 2020-09-16T14:34:27.021Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:34:27.021Z [TRACE] client: next heartbeat: period=13.258669766s 2020-09-16T14:34:27.022Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:34:27.022Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:34:27.024Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:34:27.024Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:34:27.026Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:34:27.026Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:34:27.028Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:34:27.028Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:34:27.029Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:34:27.029Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:34:27.031Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:34:27.031Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:34:31.581Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1514 duration=481.175µs 2020-09-16T14:34:32.954Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:55922 2020-09-16T14:34:33.077Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=245.9µs 2020-09-16T14:34:33.141Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=108.629µs 2020-09-16T14:34:35.551Z [DEBUG] http: request complete: method=GET path=/v1/regions duration=195.701µs 2020-09-16T14:34:35.557Z [DEBUG] http: request failed: method=GET path=/v1/acl/token/self error="RPC Error:: 400,ACL support disabled" code=400 2020-09-16T14:34:35.557Z [DEBUG] http: request complete: method=GET path=/v1/acl/token/self duration=174.64µs 2020-09-16T14:34:35.563Z [DEBUG] http: request complete: method=GET path=/v1/agent/members duration=196.213µs 2020-09-16T14:34:35.609Z [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501 2020-09-16T14:34:35.609Z [DEBUG] http: request complete: method=GET path=/v1/namespaces duration=246.832µs 2020-09-16T14:34:35.775Z [DEBUG] http: request complete: method=GET path=/v1/jobs duration=517.323µs 2020-09-16T14:34:40.280Z [TRACE] client: next heartbeat: period=18.334886555s 2020-09-16T14:34:42.395Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=validate 2020-09-16T14:34:42.396Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker path=/var/lib/nomad/alloc/0757fa38-d4ae-a692-a729-be89391f36eb/alloc/logs/.web-admin-worker.stdout.fifo @module=logmon timestamp=2020-09-16T14:34:42.396Z 2020-09-16T14:34:42.396Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker @module=logmon path=/var/lib/nomad/alloc/0757fa38-d4ae-a692-a729-be89391f36eb/alloc/logs/.web-admin-worker.stderr.fifo timestamp=2020-09-16T14:34:42.396Z 2020-09-16T14:34:42.396Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=dispatch_payload 2020-09-16T14:34:42.396Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=artifacts 2020-09-16T14:34:42.396Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=devices 2020-09-16T14:34:42.397Z [INFO] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/web/current; exec &> >(tee -i -a /var/log/innrm/web_admin_worker-127.0.0.1-3201.log); exec /bin/bash ./bin/production_run_worker_mq.sh --type admin]}" 2020-09-16T14:34:42.397Z [DEBUG] client.driver_mgr.raw_exec.executor: starting plugin: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad args=[/usr/sbin/nomad, executor, {"LogFile":"/var/lib/nomad/alloc/0757fa38-d4ae-a692-a729-be89391f36eb/web-admin-worker/executor.out","LogLevel":"debug","FSIsolation":false}] 2020-09-16T14:34:42.403Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin started: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad pid=4732 2020-09-16T14:34:42.403Z [DEBUG] client.driver_mgr.raw_exec.executor: waiting for RPC address: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad 2020-09-16T14:34:42.434Z [DEBUG] client.driver_mgr.raw_exec.executor: using plugin: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker version=2 2020-09-16T14:34:42.453Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker state=running event=Started 2020-09-16T14:34:42.455Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:34:42.455Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=running desired_status= 2020-09-16T14:34:42.455Z [DEBUG] consul.health: watching check: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker check="web-admin-worker check" 2020-09-16T14:34:42.491Z [DEBUG] consul.sync: sync complete: registered_services=1 deregistered_services=0 registered_checks=1 deregistered_checks=0 2020-09-16T14:34:42.618Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1523 duration=6.80022775s 2020-09-16T14:34:42.618Z [TRACE] client: next heartbeat: period=10.146008816s 2020-09-16T14:34:42.620Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1523 duration=9.036481634s 2020-09-16T14:34:42.622Z [DEBUG] client: updated allocations: index=1525 total=34 pulled=6 filtered=28 2020-09-16T14:34:42.622Z [DEBUG] client: allocation updates: added=0 removed=0 updated=6 ignored=28 2020-09-16T14:34:42.622Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:34:42.623Z [TRACE] client: next heartbeat: period=11.407754856s 2020-09-16T14:34:42.625Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:34:42.625Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:34:42.627Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:34:42.627Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:34:42.635Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:34:42.635Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:34:42.639Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:34:42.639Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:34:42.642Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:34:42.642Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:34:42.658Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:34:42.659Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:34:42.955Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:55962 2020-09-16T14:34:43.079Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=123.894µs 2020-09-16T14:34:43.142Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=168.481µs 2020-09-16T14:34:52.955Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:56012 2020-09-16T14:34:53.080Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=105.788µs 2020-09-16T14:34:53.145Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=91.339µs 2020-09-16T14:34:54.032Z [TRACE] client: next heartbeat: period=10.05999192s 2020-09-16T14:34:55.492Z [DEBUG] http: request complete: method=GET path=/v1/regions duration=191.044µs 2020-09-16T14:34:55.502Z [DEBUG] http: request failed: method=GET path=/v1/acl/token/self error="RPC Error:: 400,ACL support disabled" code=400 2020-09-16T14:34:55.502Z [DEBUG] http: request complete: method=GET path=/v1/acl/token/self duration=231.027µs 2020-09-16T14:34:55.511Z [DEBUG] http: request complete: method=GET path=/v1/agent/members duration=111.201µs 2020-09-16T14:34:55.587Z [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501 2020-09-16T14:34:55.587Z [DEBUG] http: request complete: method=GET path=/v1/namespaces duration=137.039µs 2020-09-16T14:34:55.852Z [DEBUG] http: request complete: method=GET path=/v1/jobs duration=490.889µs 2020-09-16T14:34:59.669Z [DEBUG] http: request complete: method=GET path=/v1/node/9ee95205-0945-8494-15b6-74400d24799e?index=638 duration=277.538µs 2020-09-16T14:34:59.685Z [DEBUG] http: request complete: method=GET path=/v1/node/9ee95205-0945-8494-15b6-74400d24799e/allocations?index=1461 duration=9.963991ms 2020-09-16T14:34:59.714Z [DEBUG] http: request complete: method=GET path=/v1/client/stats?node_id=9ee95205-0945-8494-15b6-74400d24799e duration=267.388µs 2020-09-16T14:35:00.218Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker duration=212.077µs 2020-09-16T14:35:00.220Z [DEBUG] http: request complete: method=GET path=/v1/job/message-processor duration=250.846µs 2020-09-16T14:35:00.232Z [DEBUG] http: request complete: method=GET path=/v1/job/fast-service duration=221.729µs 2020-09-16T14:35:00.233Z [DEBUG] http: request complete: method=GET path=/v1/job/pre-extractor duration=214.666µs 2020-09-16T14:35:00.238Z [DEBUG] http: request complete: method=GET path=/v1/job/jaeger-collector duration=221.796µs 2020-09-16T14:35:00.238Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0 duration=150.022µs 2020-09-16T14:35:00.250Z [DEBUG] http: request complete: method=GET path=/v1/job/activemq duration=234.163µs 2020-09-16T14:35:00.250Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/cd3019c0-6679-26b5-b733-bca9a18670f4/stats duration=294.852µs 2020-09-16T14:35:00.315Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/0757fa38-d4ae-a692-a729-be89391f36eb/stats duration=162.678µs 2020-09-16T14:35:00.336Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/19368a06-1e30-432b-3b25-4f8a0609b8fe/stats duration=253.256µs 2020-09-16T14:35:00.353Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/fe958507-4ffc-d1d1-ae4e-e9d0cc939f05/stats duration=159.549µs 2020-09-16T14:35:00.368Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/03e6e81f-909e-daa2-8f68-afeecd5cefa5/stats duration=178.828µs 2020-09-16T14:35:00.385Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/98e0b801-c484-68df-1dc6-bff0855ceb13/stats duration=160.166µs 2020-09-16T14:35:00.398Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4/stats duration=190.072µs 2020-09-16T14:35:00.441Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/29883c59-0cdd-02fd-08e5-df0c35292aa0/stats duration=267.455µs 2020-09-16T14:35:01.509Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:35:01.509Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=running desired_status= 2020-09-16T14:35:01.511Z [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker error="rpc error: code = Unavailable desc = transport is closing" 2020-09-16T14:35:01.511Z [ERROR] client.alloc_runner.task_runner.task_hook.stats_hook: failed to start stats collection for task: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker error="rpc error: code = Unavailable desc = connection error: desc = "transport: error while dialing: dial unix /tmp/plugin022961690: connect: no such file or directory"" 2020-09-16T14:35:01.513Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin process exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad pid=4732 2020-09-16T14:35:01.513Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker 2020-09-16T14:35:01.513Z [INFO] client.alloc_runner.task_runner: restarting task: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker reason="Restart within policy" delay=18.577465537s 2020-09-16T14:35:01.513Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker state=pending event=Restarting 2020-09-16T14:35:01.516Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:35:01.516Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=pending desired_status= 2020-09-16T14:35:01.520Z [DEBUG] consul.sync: sync complete: registered_services=0 deregistered_services=1 registered_checks=0 deregistered_checks=1 2020-09-16T14:35:01.621Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1525 duration=5.687205364s 2020-09-16T14:35:01.621Z [TRACE] client: next heartbeat: period=18.671029218s 2020-09-16T14:35:01.622Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1525 duration=18.877735593s 2020-09-16T14:35:01.623Z [TRACE] client: next heartbeat: period=12.087664123s 2020-09-16T14:35:01.624Z [DEBUG] client: updated allocations: index=1526 total=34 pulled=6 filtered=28 2020-09-16T14:35:01.624Z [DEBUG] client: allocation updates: added=0 removed=0 updated=6 ignored=28 2020-09-16T14:35:01.624Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:35:01.625Z [TRACE] client: next heartbeat: period=16.57067239s 2020-09-16T14:35:01.626Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:35:01.626Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:35:01.628Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:35:01.628Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:35:01.629Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:35:01.629Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:35:01.631Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:35:01.631Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:35:01.632Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:35:01.632Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:35:01.634Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:35:01.634Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:35:01.682Z [DEBUG] http: request complete: method=GET path=/v1/node/9ee95205-0945-8494-15b6-74400d24799e/allocations?index=1525 duration=9.012097ms 2020-09-16T14:35:01.882Z [DEBUG] http: request complete: method=GET path=/v1/client/stats?node_id=9ee95205-0945-8494-15b6-74400d24799e duration=220.807µs 2020-09-16T14:35:02.796Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/cd3019c0-6679-26b5-b733-bca9a18670f4/stats duration=128.96µs 2020-09-16T14:35:02.956Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:56060 2020-09-16T14:35:02.983Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/19368a06-1e30-432b-3b25-4f8a0609b8fe/stats duration=209.622µs 2020-09-16T14:35:02.993Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/fe958507-4ffc-d1d1-ae4e-e9d0cc939f05/stats duration=237.036µs 2020-09-16T14:35:02.995Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/03e6e81f-909e-daa2-8f68-afeecd5cefa5/stats duration=333.699µs 2020-09-16T14:35:03.003Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/98e0b801-c484-68df-1dc6-bff0855ceb13/stats duration=326.709µs 2020-09-16T14:35:03.011Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4/stats duration=142.912µs 2020-09-16T14:35:03.012Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/29883c59-0cdd-02fd-08e5-df0c35292aa0/stats duration=188.261µs 2020-09-16T14:35:03.082Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=124.223µs 2020-09-16T14:35:03.147Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=109.281µs 2020-09-16T14:35:03.913Z [DEBUG] http: request complete: method=GET path=/v1/client/stats?node_id=9ee95205-0945-8494-15b6-74400d24799e duration=283.897µs 2020-09-16T14:35:05.304Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/cd3019c0-6679-26b5-b733-bca9a18670f4/stats duration=144.62µs 2020-09-16T14:35:05.490Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/19368a06-1e30-432b-3b25-4f8a0609b8fe/stats duration=153.992µs 2020-09-16T14:35:05.502Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/fe958507-4ffc-d1d1-ae4e-e9d0cc939f05/stats duration=180.744µs 2020-09-16T14:35:05.521Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/03e6e81f-909e-daa2-8f68-afeecd5cefa5/stats duration=161.101µs 2020-09-16T14:35:05.522Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/98e0b801-c484-68df-1dc6-bff0855ceb13/stats duration=197.162µs 2020-09-16T14:35:05.527Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4/stats duration=164.957µs 2020-09-16T14:35:05.528Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/29883c59-0cdd-02fd-08e5-df0c35292aa0/stats duration=174.221µs 2020-09-16T14:35:05.948Z [DEBUG] http: request complete: method=GET path=/v1/client/stats?node_id=9ee95205-0945-8494-15b6-74400d24799e duration=211.388µs 2020-09-16T14:35:07.811Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/cd3019c0-6679-26b5-b733-bca9a18670f4/stats duration=135.024µs 2020-09-16T14:35:07.976Z [DEBUG] http: request complete: method=GET path=/v1/client/stats?node_id=9ee95205-0945-8494-15b6-74400d24799e duration=148.311µs 2020-09-16T14:35:08.010Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/19368a06-1e30-432b-3b25-4f8a0609b8fe/stats duration=251.496µs 2020-09-16T14:35:08.026Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/fe958507-4ffc-d1d1-ae4e-e9d0cc939f05/stats duration=188.187µs 2020-09-16T14:35:08.031Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/03e6e81f-909e-daa2-8f68-afeecd5cefa5/stats duration=396.884µs 2020-09-16T14:35:08.050Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/98e0b801-c484-68df-1dc6-bff0855ceb13/stats duration=290.73µs 2020-09-16T14:35:08.053Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4/stats duration=288.635µs 2020-09-16T14:35:08.065Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/29883c59-0cdd-02fd-08e5-df0c35292aa0/stats duration=214.987µs 2020-09-16T14:35:08.312Z [DEBUG] http: request complete: method=GET path=/v1/jobs duration=278.977µs 2020-09-16T14:35:08.371Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/allocations?index=1 duration=487.738µs 2020-09-16T14:35:08.372Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/evaluations?index=1 duration=228.603µs 2020-09-16T14:35:08.377Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/deployment?index=1 duration=224.744µs 2020-09-16T14:35:08.390Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/summary?index=1 duration=148.307µs 2020-09-16T14:35:08.472Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/evaluations duration=125.608µs 2020-09-16T14:35:08.475Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/deployment duration=151.068µs 2020-09-16T14:35:08.505Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/allocations duration=275.169µs 2020-09-16T14:35:08.557Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/deployment duration=209.059µs 2020-09-16T14:35:08.564Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/versions?diffs=true duration=196.359µs 2020-09-16T14:35:08.642Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0757fa38-d4ae-a692-a729-be89391f36eb duration=332.003µs 2020-09-16T14:35:08.645Z [DEBUG] http: request complete: method=GET path=/v1/allocation/81b56d77-aef0-1a03-c265-2d1cda1ef2e8 duration=276.45µs 2020-09-16T14:35:12.957Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:56100 2020-09-16T14:35:13.083Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=226.598µs 2020-09-16T14:35:13.148Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=139.839µs 2020-09-16T14:35:18.196Z [TRACE] client: next heartbeat: period=16.551341703s 2020-09-16T14:35:20.093Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=validate 2020-09-16T14:35:20.094Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker @module=logmon path=/var/lib/nomad/alloc/0757fa38-d4ae-a692-a729-be89391f36eb/alloc/logs/.web-admin-worker.stdout.fifo timestamp=2020-09-16T14:35:20.094Z 2020-09-16T14:35:20.094Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker path=/var/lib/nomad/alloc/0757fa38-d4ae-a692-a729-be89391f36eb/alloc/logs/.web-admin-worker.stderr.fifo @module=logmon timestamp=2020-09-16T14:35:20.094Z 2020-09-16T14:35:20.094Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=dispatch_payload 2020-09-16T14:35:20.095Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=artifacts 2020-09-16T14:35:20.095Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=devices 2020-09-16T14:35:20.096Z [INFO] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/web/current; exec &> >(tee -i -a /var/log/innrm/web_admin_worker-127.0.0.1-3201.log); exec /bin/bash ./bin/production_run_worker_mq.sh --type admin]}" 2020-09-16T14:35:20.096Z [DEBUG] client.driver_mgr.raw_exec.executor: starting plugin: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad args=[/usr/sbin/nomad, executor, {"LogFile":"/var/lib/nomad/alloc/0757fa38-d4ae-a692-a729-be89391f36eb/web-admin-worker/executor.out","LogLevel":"debug","FSIsolation":false}] 2020-09-16T14:35:20.102Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin started: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad pid=4812 2020-09-16T14:35:20.102Z [DEBUG] client.driver_mgr.raw_exec.executor: waiting for RPC address: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad 2020-09-16T14:35:20.121Z [DEBUG] client.driver_mgr.raw_exec.executor: using plugin: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker version=2 2020-09-16T14:35:20.128Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker state=running event=Started 2020-09-16T14:35:20.129Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:35:20.129Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=running desired_status= 2020-09-16T14:35:20.130Z [DEBUG] consul.health: watching check: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker check="web-admin-worker check" 2020-09-16T14:35:20.147Z [DEBUG] consul.sync: sync complete: registered_services=1 deregistered_services=0 registered_checks=1 deregistered_checks=0 2020-09-16T14:35:20.217Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/summary?index=1526 duration=9.845663476s 2020-09-16T14:35:20.217Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/allocations?index=1526 duration=9.84522413s 2020-09-16T14:35:20.218Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0757fa38-d4ae-a692-a729-be89391f36eb?index=1526 duration=3.755084563s 2020-09-16T14:35:20.219Z [TRACE] client: next heartbeat: period=18.569214961s 2020-09-16T14:35:20.221Z [DEBUG] client: updated allocations: index=1527 total=34 pulled=6 filtered=28 2020-09-16T14:35:20.221Z [TRACE] client: next heartbeat: period=10.698688515s 2020-09-16T14:35:20.221Z [DEBUG] client: allocation updates: added=0 removed=0 updated=6 ignored=28 2020-09-16T14:35:20.221Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:35:20.224Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:35:20.224Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:35:20.226Z [DEBUG] http: request complete: method=GET path=/v1/node/9ee95205-0945-8494-15b6-74400d24799e/allocations?index=1526 duration=16.549687477s 2020-09-16T14:35:20.226Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:35:20.226Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:35:20.229Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:35:20.229Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:35:20.231Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:35:20.231Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:35:20.233Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:35:20.233Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:35:20.237Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:35:20.237Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:35:20.265Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/0757fa38-d4ae-a692-a729-be89391f36eb/stats duration=202.642µs 2020-09-16T14:35:22.958Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:56142 2020-09-16T14:35:23.085Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=114.542µs 2020-09-16T14:35:23.149Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=102.586µs 2020-09-16T14:35:30.921Z [TRACE] client: next heartbeat: period=15.794855327s 2020-09-16T14:35:32.959Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:56194 2020-09-16T14:35:33.087Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=136.573µs 2020-09-16T14:35:33.153Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=216.848µs 2020-09-16T14:35:36.442Z [DEBUG] worker: dequeued evaluation: eval_id=f55c60b4-48cc-8453-5f1e-2e5e4335e892 2020-09-16T14:35:36.442Z [DEBUG] core.sched: eval GC scanning before cutoff index: index=0 eval_gc_threshold=1h0m0s 2020-09-16T14:35:36.442Z [DEBUG] worker: ack evaluation: eval_id=f55c60b4-48cc-8453-5f1e-2e5e4335e892 2020-09-16T14:35:36.442Z [DEBUG] worker: dequeued evaluation: eval_id=7497c939-0468-9e9b-6633-4c9bc9dd422e 2020-09-16T14:35:36.442Z [DEBUG] core.sched: node GC scanning before cutoff index: index=0 node_gc_threshold=24h0m0s 2020-09-16T14:35:36.442Z [DEBUG] worker: ack evaluation: eval_id=7497c939-0468-9e9b-6633-4c9bc9dd422e 2020-09-16T14:35:36.442Z [DEBUG] worker: dequeued evaluation: eval_id=a3fbf31e-8449-0e09-806f-7f28094e5181 2020-09-16T14:35:36.442Z [TRACE] core.sched: garbage collecting unclaimed CSI volume claims: eval.JobID=csi-volume-claim-gc 2020-09-16T14:35:36.442Z [DEBUG] core.sched: CSI volume claim GC scanning before cutoff index: index=1457 csi_volume_claim_gc_threshold=5m0s 2020-09-16T14:35:36.442Z [DEBUG] worker: ack evaluation: eval_id=a3fbf31e-8449-0e09-806f-7f28094e5181 2020-09-16T14:35:36.442Z [DEBUG] worker: dequeued evaluation: eval_id=257cd9c4-c808-230d-a4ee-f4bdcd335a66 2020-09-16T14:35:36.442Z [DEBUG] core.sched: deployment GC scanning before cutoff index: index=0 deployment_gc_threshold=1h0m0s 2020-09-16T14:35:36.442Z [DEBUG] worker: ack evaluation: eval_id=257cd9c4-c808-230d-a4ee-f4bdcd335a66 2020-09-16T14:35:36.442Z [DEBUG] worker: dequeued evaluation: eval_id=8be59d96-ff4c-8bfc-ec8d-e8fe11936643 2020-09-16T14:35:36.442Z [DEBUG] core.sched: CSI plugin GC scanning before cutoff index: index=0 csi_plugin_gc_threshold=1h0m0s 2020-09-16T14:35:36.442Z [DEBUG] worker: ack evaluation: eval_id=8be59d96-ff4c-8bfc-ec8d-e8fe11936643 2020-09-16T14:35:36.442Z [DEBUG] worker: dequeued evaluation: eval_id=f766a826-49da-e38f-8527-762d36ea90ba 2020-09-16T14:35:36.442Z [DEBUG] core.sched: job GC scanning before cutoff index: index=0 job_gc_threshold=4h0m0s 2020-09-16T14:35:36.442Z [DEBUG] worker: ack evaluation: eval_id=f766a826-49da-e38f-8527-762d36ea90ba 2020-09-16T14:35:37.952Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:35:37.952Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=running desired_status= 2020-09-16T14:35:37.954Z [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker error="rpc error: code = Unavailable desc = transport is closing" 2020-09-16T14:35:37.954Z [ERROR] client.alloc_runner.task_runner.task_hook.stats_hook: failed to start stats collection for task: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker error="rpc error: code = Canceled desc = grpc: the client connection is closing" 2020-09-16T14:35:37.957Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin process exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad pid=4812 2020-09-16T14:35:37.957Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker 2020-09-16T14:35:37.957Z [INFO] client.alloc_runner.task_runner: not restarting task: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail"" 2020-09-16T14:35:37.957Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker state=dead event="Not Restarting" 2020-09-16T14:35:37.961Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:35:37.965Z [INFO] client.gc: marking allocation for GC: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb 2020-09-16T14:35:37.965Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=failed desired_status= 2020-09-16T14:35:37.965Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:35:37.965Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=failed desired_status= 2020-09-16T14:35:37.972Z [DEBUG] consul.sync: sync complete: registered_services=0 deregistered_services=1 registered_checks=0 deregistered_checks=1 2020-09-16T14:35:37.975Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker path=/usr/sbin/nomad pid=4305 2020-09-16T14:35:37.975Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker 2020-09-16T14:35:37.975Z [DEBUG] client.alloc_runner.task_runner: task run loop exiting: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker 2020-09-16T14:35:37.976Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=true 2020-09-16T14:35:37.976Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=failed desired_status= 2020-09-16T14:35:38.015Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0757fa38-d4ae-a692-a729-be89391f36eb?index=1527 duration=17.765881292s 2020-09-16T14:35:38.020Z [DEBUG] client: updated allocations: index=1529 total=34 pulled=6 filtered=28 2020-09-16T14:35:38.020Z [TRACE] client: next heartbeat: period=12.480484912s 2020-09-16T14:35:38.020Z [DEBUG] client: allocation updates: added=0 removed=0 updated=6 ignored=28 2020-09-16T14:35:38.020Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:35:38.022Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:35:38.022Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:35:38.024Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:35:38.024Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:35:38.026Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:35:38.026Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:35:38.028Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:35:38.028Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:35:38.029Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:35:38.029Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:35:38.031Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:35:38.031Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:35:38.212Z [DEBUG] nomad.client: adding evaluations for rescheduling failed allocations: num_evals=1 2020-09-16T14:35:38.216Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker?index=1492 duration=29.828076969s 2020-09-16T14:35:38.216Z [TRACE] client: next heartbeat: period=18.775546362s 2020-09-16T14:35:38.216Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/evaluations?index=1506 duration=27.843150784s 2020-09-16T14:35:38.217Z [DEBUG] worker: dequeued evaluation: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd 2020-09-16T14:35:38.217Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd job_id=web-admin-worker namespace=default algorithm=binpack 2020-09-16T14:35:38.217Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd job_id=web-admin-worker namespace=default results="Total changes: (place 0) (destructive 0) (inplace 0) (stop 0) Desired Changes for "web-admin-worker": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 1) (canary 0)" 2020-09-16T14:35:38.217Z [TRACE] client: next heartbeat: period=15.945243284s 2020-09-16T14:35:38.220Z [DEBUG] client: updated allocations: index=1530 total=34 pulled=6 filtered=28 2020-09-16T14:35:38.220Z [DEBUG] client: allocation updates: added=0 removed=0 updated=6 ignored=28 2020-09-16T14:35:38.220Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:35:38.220Z [TRACE] client: next heartbeat: period=10.178164909s 2020-09-16T14:35:38.222Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T14:35:38.222Z [DEBUG] worker.service_sched: found reschedulable allocs, followup eval created: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd job_id=web-admin-worker namespace=default followup_eval_id=ba6fab80-c937-6c84-b125-b32664bf0433 2020-09-16T14:35:38.222Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:35:38.222Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:35:38.226Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:35:38.226Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:35:38.228Z [DEBUG] worker: submitted plan for evaluation: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd 2020-09-16T14:35:38.228Z [DEBUG] worker.service_sched: setting eval status: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd job_id=web-admin-worker namespace=default status=complete 2020-09-16T14:35:38.228Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:35:38.228Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:35:38.230Z [TRACE] client: next heartbeat: period=18.62996492s 2020-09-16T14:35:38.232Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:35:38.232Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:35:38.233Z [DEBUG] client: updated allocations: index=1532 total=34 pulled=7 filtered=27 2020-09-16T14:35:38.233Z [TRACE] client: next heartbeat: period=14.405656497s 2020-09-16T14:35:38.234Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T14:35:38.234Z [DEBUG] worker: ack evaluation: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd 2020-09-16T14:35:38.235Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:35:38.235Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:35:38.238Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:35:38.238Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:35:38.238Z [DEBUG] client: allocation updates: added=0 removed=0 updated=7 ignored=27 2020-09-16T14:35:38.238Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:35:38.241Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:35:38.241Z [TRACE] client: updating alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb index=1532 2020-09-16T14:35:38.242Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb modify_index=1532 2020-09-16T14:35:38.242Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:35:38.244Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:35:38.244Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:35:38.245Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:35:38.245Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:35:38.247Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:35:38.247Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:35:38.248Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:35:38.248Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:35:38.251Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:35:38.251Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=7 ignored=27 errors=0 2020-09-16T14:37:14.813Z [TRACE] client: next heartbeat: period=15.542385354s ```

Server logs:

``` 1da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:34:17.526Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:34:17.526Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:34:17.531Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:34:17.531Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:34:22.948Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:55876 2020-09-16T14:34:23.075Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=232.834µs 2020-09-16T14:34:23.138Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=136.604µs 2020-09-16T14:34:26.823Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:34:26.823Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=running desired_status= 2020-09-16T14:34:26.825Z [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker error="rpc error: code = Unavailable desc = transport is closing" 2020-09-16T14:34:26.825Z [ERROR] client.alloc_runner.task_runner.task_hook.stats_hook: failed to start stats collection for task: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker error="rpc error: code = Canceled desc = grpc: the client connection is closing" 2020-09-16T14:34:26.826Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin process exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad pid=4342 2020-09-16T14:34:26.827Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker 2020-09-16T14:34:26.827Z [INFO] client.alloc_runner.task_runner: restarting task: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker reason="Restart within policy" delay=15.563263625s 2020-09-16T14:34:26.827Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker state=pending event=Restarting 2020-09-16T14:34:26.831Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:34:26.831Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=pending desired_status= 2020-09-16T14:34:26.838Z [DEBUG] consul.sync: sync complete: registered_services=0 deregistered_services=1 registered_checks=0 deregistered_checks=1 2020-09-16T14:34:27.017Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1514 duration=42.388254481s 2020-09-16T14:34:27.017Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1514 duration=25.749448496s 2020-09-16T14:34:27.017Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1514 duration=22.561463315s 2020-09-16T14:34:27.019Z [TRACE] client: next heartbeat: period=19.682077063s 2020-09-16T14:34:27.020Z [DEBUG] client: updated allocations: index=1523 total=34 pulled=6 filtered=28 2020-09-16T14:34:27.020Z [DEBUG] client: allocation updates: added=0 removed=0 updated=6 ignored=28 2020-09-16T14:34:27.021Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:34:27.021Z [TRACE] client: next heartbeat: period=13.258669766s 2020-09-16T14:34:27.022Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:34:27.022Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:34:27.024Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:34:27.024Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:34:27.026Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:34:27.026Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:34:27.028Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:34:27.028Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:34:27.029Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:34:27.029Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:34:27.031Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:34:27.031Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:34:31.581Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1514 duration=481.175µs 2020-09-16T14:34:32.954Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:55922 2020-09-16T14:34:33.077Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=245.9µs 2020-09-16T14:34:33.141Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=108.629µs 2020-09-16T14:34:35.551Z [DEBUG] http: request complete: method=GET path=/v1/regions duration=195.701µs 2020-09-16T14:34:35.557Z [DEBUG] http: request failed: method=GET path=/v1/acl/token/self error="RPC Error:: 400,ACL support disabled" code=400 2020-09-16T14:34:35.557Z [DEBUG] http: request complete: method=GET path=/v1/acl/token/self duration=174.64µs 2020-09-16T14:34:35.563Z [DEBUG] http: request complete: method=GET path=/v1/agent/members duration=196.213µs 2020-09-16T14:34:35.609Z [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501 2020-09-16T14:34:35.609Z [DEBUG] http: request complete: method=GET path=/v1/namespaces duration=246.832µs 2020-09-16T14:34:35.775Z [DEBUG] http: request complete: method=GET path=/v1/jobs duration=517.323µs 2020-09-16T14:34:40.280Z [TRACE] client: next heartbeat: period=18.334886555s 2020-09-16T14:34:42.395Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=validate 2020-09-16T14:34:42.396Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker path=/var/lib/nomad/alloc/0757fa38-d4ae-a692-a729-be89391f36eb/alloc/logs/.web-admin-worker.stdout.fifo @module=logmon timestamp=2020-09-16T14:34:42.396Z 2020-09-16T14:34:42.396Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker @module=logmon path=/var/lib/nomad/alloc/0757fa38-d4ae-a692-a729-be89391f36eb/alloc/logs/.web-admin-worker.stderr.fifo timestamp=2020-09-16T14:34:42.396Z 2020-09-16T14:34:42.396Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=dispatch_payload 2020-09-16T14:34:42.396Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=artifacts 2020-09-16T14:34:42.396Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=devices 2020-09-16T14:34:42.397Z [INFO] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/web/current; exec &> >(tee -i -a /var/log/innrm/web_admin_worker-127.0.0.1-3201.log); exec /bin/bash ./bin/production_run_worker_mq.sh --type admin]}" 2020-09-16T14:34:42.397Z [DEBUG] client.driver_mgr.raw_exec.executor: starting plugin: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad args=[/usr/sbin/nomad, executor, {"LogFile":"/var/lib/nomad/alloc/0757fa38-d4ae-a692-a729-be89391f36eb/web-admin-worker/executor.out","LogLevel":"debug","FSIsolation":false}] 2020-09-16T14:34:42.403Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin started: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad pid=4732 2020-09-16T14:34:42.403Z [DEBUG] client.driver_mgr.raw_exec.executor: waiting for RPC address: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad 2020-09-16T14:34:42.434Z [DEBUG] client.driver_mgr.raw_exec.executor: using plugin: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker version=2 2020-09-16T14:34:42.453Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker state=running event=Started 2020-09-16T14:34:42.455Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:34:42.455Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=running desired_status= 2020-09-16T14:34:42.455Z [DEBUG] consul.health: watching check: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker check="web-admin-worker check" 2020-09-16T14:34:42.491Z [DEBUG] consul.sync: sync complete: registered_services=1 deregistered_services=0 registered_checks=1 deregistered_checks=0 2020-09-16T14:34:42.618Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1523 duration=6.80022775s 2020-09-16T14:34:42.618Z [TRACE] client: next heartbeat: period=10.146008816s 2020-09-16T14:34:42.620Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1523 duration=9.036481634s 2020-09-16T14:34:42.622Z [DEBUG] client: updated allocations: index=1525 total=34 pulled=6 filtered=28 2020-09-16T14:34:42.622Z [DEBUG] client: allocation updates: added=0 removed=0 updated=6 ignored=28 2020-09-16T14:34:42.622Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:34:42.623Z [TRACE] client: next heartbeat: period=11.407754856s 2020-09-16T14:34:42.624Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:34:42.625Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:34:42.627Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:34:42.627Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:34:42.635Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:34:42.635Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:34:42.639Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:34:42.639Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:34:42.642Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:34:42.642Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:34:42.658Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:34:42.659Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:34:42.955Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:55962 2020-09-16T14:34:43.079Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=123.894µs 2020-09-16T14:34:43.142Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=168.481µs 2020-09-16T14:34:52.955Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:56012 2020-09-16T14:34:53.080Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=105.788µs 2020-09-16T14:34:53.145Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=91.339µs 2020-09-16T14:34:54.032Z [TRACE] client: next heartbeat: period=10.05999192s 2020-09-16T14:34:55.492Z [DEBUG] http: request complete: method=GET path=/v1/regions duration=191.044µs 2020-09-16T14:34:55.502Z [DEBUG] http: request failed: method=GET path=/v1/acl/token/self error="RPC Error:: 400,ACL support disabled" code=400 2020-09-16T14:34:55.502Z [DEBUG] http: request complete: method=GET path=/v1/acl/token/self duration=231.027µs 2020-09-16T14:34:55.511Z [DEBUG] http: request complete: method=GET path=/v1/agent/members duration=111.201µs 2020-09-16T14:34:55.587Z [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501 2020-09-16T14:34:55.587Z [DEBUG] http: request complete: method=GET path=/v1/namespaces duration=137.039µs 2020-09-16T14:34:55.852Z [DEBUG] http: request complete: method=GET path=/v1/jobs duration=490.889µs 2020-09-16T14:34:59.669Z [DEBUG] http: request complete: method=GET path=/v1/node/9ee95205-0945-8494-15b6-74400d24799e?index=638 duration=277.538µs 2020-09-16T14:34:59.685Z [DEBUG] http: request complete: method=GET path=/v1/node/9ee95205-0945-8494-15b6-74400d24799e/allocations?index=1461 duration=9.963991ms 2020-09-16T14:34:59.714Z [DEBUG] http: request complete: method=GET path=/v1/client/stats?node_id=9ee95205-0945-8494-15b6-74400d24799e duration=267.388µs 2020-09-16T14:35:00.218Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker duration=212.077µs 2020-09-16T14:35:00.220Z [DEBUG] http: request complete: method=GET path=/v1/job/message-processor duration=250.846µs 2020-09-16T14:35:00.232Z [DEBUG] http: request complete: method=GET path=/v1/job/fast-service duration=221.729µs 2020-09-16T14:35:00.233Z [DEBUG] http: request complete: method=GET path=/v1/job/pre-extractor duration=214.666µs 2020-09-16T14:35:00.238Z [DEBUG] http: request complete: method=GET path=/v1/job/jaeger-collector duration=221.796µs 2020-09-16T14:35:00.238Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0 duration=150.022µs 2020-09-16T14:35:00.250Z [DEBUG] http: request complete: method=GET path=/v1/job/activemq duration=234.163µs 2020-09-16T14:35:00.250Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/cd3019c0-6679-26b5-b733-bca9a18670f4/stats duration=294.852µs 2020-09-16T14:35:00.315Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/0757fa38-d4ae-a692-a729-be89391f36eb/stats duration=162.678µs 2020-09-16T14:35:00.336Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/19368a06-1e30-432b-3b25-4f8a0609b8fe/stats duration=253.256µs 2020-09-16T14:35:00.353Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/fe958507-4ffc-d1d1-ae4e-e9d0cc939f05/stats duration=159.549µs 2020-09-16T14:35:00.368Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/03e6e81f-909e-daa2-8f68-afeecd5cefa5/stats duration=178.828µs 2020-09-16T14:35:00.385Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/98e0b801-c484-68df-1dc6-bff0855ceb13/stats duration=160.166µs 2020-09-16T14:35:00.398Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4/stats duration=190.072µs 2020-09-16T14:35:00.441Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/29883c59-0cdd-02fd-08e5-df0c35292aa0/stats duration=267.455µs 2020-09-16T14:35:01.509Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:35:01.509Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=running desired_status= 2020-09-16T14:35:01.511Z [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker error="rpc error: code = Unavailable desc = transport is closing" 2020-09-16T14:35:01.511Z [ERROR] client.alloc_runner.task_runner.task_hook.stats_hook: failed to start stats collection for task: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker error="rpc error: code = Unavailable desc = connection error: desc = "transport: error while dialing: dial unix /tmp/plugin022961690: connect: no such file or directory"" 2020-09-16T14:35:01.513Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin process exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad pid=4732 2020-09-16T14:35:01.513Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker 2020-09-16T14:35:01.513Z [INFO] client.alloc_runner.task_runner: restarting task: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker reason="Restart within policy" delay=18.577465537s 2020-09-16T14:35:01.513Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker state=pending event=Restarting 2020-09-16T14:35:01.516Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:35:01.516Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=pending desired_status= 2020-09-16T14:35:01.520Z [DEBUG] consul.sync: sync complete: registered_services=0 deregistered_services=1 registered_checks=0 deregistered_checks=1 2020-09-16T14:35:01.621Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1525 duration=5.687205364s 2020-09-16T14:35:01.621Z [TRACE] client: next heartbeat: period=18.671029218s 2020-09-16T14:35:01.622Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1525 duration=18.877735593s 2020-09-16T14:35:01.623Z [TRACE] client: next heartbeat: period=12.087664123s 2020-09-16T14:35:01.624Z [DEBUG] client: updated allocations: index=1526 total=34 pulled=6 filtered=28 2020-09-16T14:35:01.624Z [DEBUG] client: allocation updates: added=0 removed=0 updated=6 ignored=28 2020-09-16T14:35:01.624Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:35:01.625Z [TRACE] client: next heartbeat: period=16.57067239s 2020-09-16T14:35:01.626Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:35:01.626Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:35:01.628Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:35:01.628Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:35:01.629Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:35:01.629Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:35:01.631Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:35:01.631Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:35:01.632Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:35:01.632Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:35:01.634Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:35:01.634Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:35:01.682Z [DEBUG] http: request complete: method=GET path=/v1/node/9ee95205-0945-8494-15b6-74400d24799e/allocations?index=1525 duration=9.012097ms 2020-09-16T14:35:01.882Z [DEBUG] http: request complete: method=GET path=/v1/client/stats?node_id=9ee95205-0945-8494-15b6-74400d24799e duration=220.807µs 2020-09-16T14:35:02.796Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/cd3019c0-6679-26b5-b733-bca9a18670f4/stats duration=128.96µs 2020-09-16T14:35:02.956Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:56060 2020-09-16T14:35:02.982Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/19368a06-1e30-432b-3b25-4f8a0609b8fe/stats duration=209.622µs 2020-09-16T14:35:02.993Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/fe958507-4ffc-d1d1-ae4e-e9d0cc939f05/stats duration=237.036µs 2020-09-16T14:35:02.995Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/03e6e81f-909e-daa2-8f68-afeecd5cefa5/stats duration=333.699µs 2020-09-16T14:35:03.003Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/98e0b801-c484-68df-1dc6-bff0855ceb13/stats duration=326.709µs 2020-09-16T14:35:03.011Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4/stats duration=142.912µs 2020-09-16T14:35:03.012Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/29883c59-0cdd-02fd-08e5-df0c35292aa0/stats duration=188.261µs 2020-09-16T14:35:03.082Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=124.223µs 2020-09-16T14:35:03.147Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=109.281µs 2020-09-16T14:35:03.913Z [DEBUG] http: request complete: method=GET path=/v1/client/stats?node_id=9ee95205-0945-8494-15b6-74400d24799e duration=283.897µs 2020-09-16T14:35:05.304Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/cd3019c0-6679-26b5-b733-bca9a18670f4/stats duration=144.62µs 2020-09-16T14:35:05.490Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/19368a06-1e30-432b-3b25-4f8a0609b8fe/stats duration=153.992µs 2020-09-16T14:35:05.502Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/fe958507-4ffc-d1d1-ae4e-e9d0cc939f05/stats duration=180.744µs 2020-09-16T14:35:05.521Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/03e6e81f-909e-daa2-8f68-afeecd5cefa5/stats duration=161.101µs 2020-09-16T14:35:05.522Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/98e0b801-c484-68df-1dc6-bff0855ceb13/stats duration=197.162µs 2020-09-16T14:35:05.527Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4/stats duration=164.957µs 2020-09-16T14:35:05.528Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/29883c59-0cdd-02fd-08e5-df0c35292aa0/stats duration=174.221µs 2020-09-16T14:35:05.948Z [DEBUG] http: request complete: method=GET path=/v1/client/stats?node_id=9ee95205-0945-8494-15b6-74400d24799e duration=211.388µs 2020-09-16T14:35:07.811Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/cd3019c0-6679-26b5-b733-bca9a18670f4/stats duration=135.024µs 2020-09-16T14:35:07.976Z [DEBUG] http: request complete: method=GET path=/v1/client/stats?node_id=9ee95205-0945-8494-15b6-74400d24799e duration=148.311µs 2020-09-16T14:35:08.010Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/19368a06-1e30-432b-3b25-4f8a0609b8fe/stats duration=251.496µs 2020-09-16T14:35:08.026Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/fe958507-4ffc-d1d1-ae4e-e9d0cc939f05/stats duration=188.187µs 2020-09-16T14:35:08.031Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/03e6e81f-909e-daa2-8f68-afeecd5cefa5/stats duration=396.884µs 2020-09-16T14:35:08.050Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/98e0b801-c484-68df-1dc6-bff0855ceb13/stats duration=290.73µs 2020-09-16T14:35:08.053Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4/stats duration=288.635µs 2020-09-16T14:35:08.065Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/29883c59-0cdd-02fd-08e5-df0c35292aa0/stats duration=214.987µs 2020-09-16T14:35:08.312Z [DEBUG] http: request complete: method=GET path=/v1/jobs duration=278.977µs 2020-09-16T14:35:08.371Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/allocations?index=1 duration=487.738µs 2020-09-16T14:35:08.372Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/evaluations?index=1 duration=228.603µs 2020-09-16T14:35:08.377Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/deployment?index=1 duration=224.744µs 2020-09-16T14:35:08.390Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/summary?index=1 duration=148.307µs 2020-09-16T14:35:08.472Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/evaluations duration=125.608µs 2020-09-16T14:35:08.475Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/deployment duration=151.068µs 2020-09-16T14:35:08.505Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/allocations duration=275.169µs 2020-09-16T14:35:08.557Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/deployment duration=209.059µs 2020-09-16T14:35:08.564Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/versions?diffs=true duration=196.359µs 2020-09-16T14:35:08.642Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0757fa38-d4ae-a692-a729-be89391f36eb duration=332.003µs 2020-09-16T14:35:08.645Z [DEBUG] http: request complete: method=GET path=/v1/allocation/81b56d77-aef0-1a03-c265-2d1cda1ef2e8 duration=276.45µs 2020-09-16T14:35:12.957Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:56100 2020-09-16T14:35:13.083Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=226.598µs 2020-09-16T14:35:13.148Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=139.839µs 2020-09-16T14:35:18.196Z [TRACE] client: next heartbeat: period=16.551341703s 2020-09-16T14:35:20.093Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=validate 2020-09-16T14:35:20.094Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker @module=logmon path=/var/lib/nomad/alloc/0757fa38-d4ae-a692-a729-be89391f36eb/alloc/logs/.web-admin-worker.stdout.fifo timestamp=2020-09-16T14:35:20.094Z 2020-09-16T14:35:20.094Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker path=/var/lib/nomad/alloc/0757fa38-d4ae-a692-a729-be89391f36eb/alloc/logs/.web-admin-worker.stderr.fifo @module=logmon timestamp=2020-09-16T14:35:20.094Z 2020-09-16T14:35:20.094Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=dispatch_payload 2020-09-16T14:35:20.095Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=artifacts 2020-09-16T14:35:20.095Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker name=devices 2020-09-16T14:35:20.096Z [INFO] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/web/current; exec &> >(tee -i -a /var/log/innrm/web_admin_worker-127.0.0.1-3201.log); exec /bin/bash ./bin/production_run_worker_mq.sh --type admin]}" 2020-09-16T14:35:20.096Z [DEBUG] client.driver_mgr.raw_exec.executor: starting plugin: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad args=[/usr/sbin/nomad, executor, {"LogFile":"/var/lib/nomad/alloc/0757fa38-d4ae-a692-a729-be89391f36eb/web-admin-worker/executor.out","LogLevel":"debug","FSIsolation":false}] 2020-09-16T14:35:20.102Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin started: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad pid=4812 2020-09-16T14:35:20.102Z [DEBUG] client.driver_mgr.raw_exec.executor: waiting for RPC address: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad 2020-09-16T14:35:20.121Z [DEBUG] client.driver_mgr.raw_exec.executor: using plugin: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker version=2 2020-09-16T14:35:20.128Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker state=running event=Started 2020-09-16T14:35:20.129Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:35:20.129Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=running desired_status= 2020-09-16T14:35:20.130Z [DEBUG] consul.health: watching check: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker check="web-admin-worker check" 2020-09-16T14:35:20.146Z [DEBUG] consul.sync: sync complete: registered_services=1 deregistered_services=0 registered_checks=1 deregistered_checks=0 2020-09-16T14:35:20.217Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/summary?index=1526 duration=9.845663476s 2020-09-16T14:35:20.217Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/allocations?index=1526 duration=9.84522413s 2020-09-16T14:35:20.218Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0757fa38-d4ae-a692-a729-be89391f36eb?index=1526 duration=3.755084563s 2020-09-16T14:35:20.219Z [TRACE] client: next heartbeat: period=18.569214961s 2020-09-16T14:35:20.221Z [DEBUG] client: updated allocations: index=1527 total=34 pulled=6 filtered=28 2020-09-16T14:35:20.221Z [TRACE] client: next heartbeat: period=10.698688515s 2020-09-16T14:35:20.221Z [DEBUG] client: allocation updates: added=0 removed=0 updated=6 ignored=28 2020-09-16T14:35:20.221Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:35:20.224Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:35:20.224Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:35:20.226Z [DEBUG] http: request complete: method=GET path=/v1/node/9ee95205-0945-8494-15b6-74400d24799e/allocations?index=1526 duration=16.549687477s 2020-09-16T14:35:20.226Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:35:20.226Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:35:20.229Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:35:20.229Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:35:20.231Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:35:20.231Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:35:20.233Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:35:20.233Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:35:20.237Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:35:20.237Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:35:20.265Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/0757fa38-d4ae-a692-a729-be89391f36eb/stats duration=202.642µs 2020-09-16T14:35:22.958Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:56142 2020-09-16T14:35:23.085Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=114.542µs 2020-09-16T14:35:23.149Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=102.586µs 2020-09-16T14:35:30.921Z [TRACE] client: next heartbeat: period=15.794855327s 2020-09-16T14:35:32.959Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:56194 2020-09-16T14:35:33.087Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=136.573µs 2020-09-16T14:35:33.153Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=216.848µs 2020-09-16T14:35:36.442Z [DEBUG] worker: dequeued evaluation: eval_id=f55c60b4-48cc-8453-5f1e-2e5e4335e892 2020-09-16T14:35:36.442Z [DEBUG] core.sched: eval GC scanning before cutoff index: index=0 eval_gc_threshold=1h0m0s 2020-09-16T14:35:36.442Z [DEBUG] worker: ack evaluation: eval_id=f55c60b4-48cc-8453-5f1e-2e5e4335e892 2020-09-16T14:35:36.442Z [DEBUG] worker: dequeued evaluation: eval_id=7497c939-0468-9e9b-6633-4c9bc9dd422e 2020-09-16T14:35:36.442Z [DEBUG] core.sched: node GC scanning before cutoff index: index=0 node_gc_threshold=24h0m0s 2020-09-16T14:35:36.442Z [DEBUG] worker: ack evaluation: eval_id=7497c939-0468-9e9b-6633-4c9bc9dd422e 2020-09-16T14:35:36.442Z [DEBUG] worker: dequeued evaluation: eval_id=a3fbf31e-8449-0e09-806f-7f28094e5181 2020-09-16T14:35:36.442Z [TRACE] core.sched: garbage collecting unclaimed CSI volume claims: eval.JobID=csi-volume-claim-gc 2020-09-16T14:35:36.442Z [DEBUG] core.sched: CSI volume claim GC scanning before cutoff index: index=1457 csi_volume_claim_gc_threshold=5m0s 2020-09-16T14:35:36.442Z [DEBUG] worker: ack evaluation: eval_id=a3fbf31e-8449-0e09-806f-7f28094e5181 2020-09-16T14:35:36.442Z [DEBUG] worker: dequeued evaluation: eval_id=257cd9c4-c808-230d-a4ee-f4bdcd335a66 2020-09-16T14:35:36.442Z [DEBUG] core.sched: deployment GC scanning before cutoff index: index=0 deployment_gc_threshold=1h0m0s 2020-09-16T14:35:36.442Z [DEBUG] worker: ack evaluation: eval_id=257cd9c4-c808-230d-a4ee-f4bdcd335a66 2020-09-16T14:35:36.442Z [DEBUG] worker: dequeued evaluation: eval_id=8be59d96-ff4c-8bfc-ec8d-e8fe11936643 2020-09-16T14:35:36.442Z [DEBUG] core.sched: CSI plugin GC scanning before cutoff index: index=0 csi_plugin_gc_threshold=1h0m0s 2020-09-16T14:35:36.442Z [DEBUG] worker: ack evaluation: eval_id=8be59d96-ff4c-8bfc-ec8d-e8fe11936643 2020-09-16T14:35:36.442Z [DEBUG] worker: dequeued evaluation: eval_id=f766a826-49da-e38f-8527-762d36ea90ba 2020-09-16T14:35:36.442Z [DEBUG] core.sched: job GC scanning before cutoff index: index=0 job_gc_threshold=4h0m0s 2020-09-16T14:35:36.442Z [DEBUG] worker: ack evaluation: eval_id=f766a826-49da-e38f-8527-762d36ea90ba 2020-09-16T14:35:37.952Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:35:37.952Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=running desired_status= 2020-09-16T14:35:37.954Z [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker error="rpc error: code = Unavailable desc = transport is closing" 2020-09-16T14:35:37.954Z [ERROR] client.alloc_runner.task_runner.task_hook.stats_hook: failed to start stats collection for task: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker error="rpc error: code = Canceled desc = grpc: the client connection is closing" 2020-09-16T14:35:37.957Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin process exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker path=/usr/sbin/nomad pid=4812 2020-09-16T14:35:37.957Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb driver=raw_exec task_name=web-admin-worker 2020-09-16T14:35:37.957Z [INFO] client.alloc_runner.task_runner: not restarting task: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail"" 2020-09-16T14:35:37.957Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker state=dead event="Not Restarting" 2020-09-16T14:35:37.961Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:35:37.965Z [INFO] client.gc: marking allocation for GC: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb 2020-09-16T14:35:37.965Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=failed desired_status= 2020-09-16T14:35:37.965Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=false 2020-09-16T14:35:37.965Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=failed desired_status= 2020-09-16T14:35:37.972Z [DEBUG] consul.sync: sync complete: registered_services=0 deregistered_services=1 registered_checks=0 deregistered_checks=1 2020-09-16T14:35:37.975Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker path=/usr/sbin/nomad pid=4305 2020-09-16T14:35:37.975Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin exited: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker 2020-09-16T14:35:37.975Z [DEBUG] client.alloc_runner.task_runner: task run loop exiting: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb task=web-admin-worker 2020-09-16T14:35:37.976Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb done=true 2020-09-16T14:35:37.976Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb client_status=failed desired_status= 2020-09-16T14:35:38.015Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0757fa38-d4ae-a692-a729-be89391f36eb?index=1527 duration=17.765881292s 2020-09-16T14:35:38.020Z [DEBUG] client: updated allocations: index=1529 total=34 pulled=6 filtered=28 2020-09-16T14:35:38.020Z [TRACE] client: next heartbeat: period=12.480484912s 2020-09-16T14:35:38.020Z [DEBUG] client: allocation updates: added=0 removed=0 updated=6 ignored=28 2020-09-16T14:35:38.020Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:35:38.022Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:35:38.022Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:35:38.024Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:35:38.024Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:35:38.026Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:35:38.026Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:35:38.028Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:35:38.028Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:35:38.029Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:35:38.029Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:35:38.031Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:35:38.031Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:35:38.212Z [DEBUG] nomad.client: adding evaluations for rescheduling failed allocations: num_evals=1 2020-09-16T14:35:38.216Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker?index=1492 duration=29.828076969s 2020-09-16T14:35:38.216Z [TRACE] client: next heartbeat: period=18.775546362s 2020-09-16T14:35:38.216Z [DEBUG] http: request complete: method=GET path=/v1/job/web-admin-worker/evaluations?index=1506 duration=27.843150784s 2020-09-16T14:35:38.217Z [DEBUG] worker: dequeued evaluation: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd 2020-09-16T14:35:38.217Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd job_id=web-admin-worker namespace=default algorithm=binpack 2020-09-16T14:35:38.217Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd job_id=web-admin-worker namespace=default results="Total changes: (place 0) (destructive 0) (inplace 0) (stop 0) Desired Changes for "web-admin-worker": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 1) (canary 0)" 2020-09-16T14:35:38.217Z [TRACE] client: next heartbeat: period=15.945243284s 2020-09-16T14:35:38.220Z [DEBUG] client: updated allocations: index=1530 total=34 pulled=6 filtered=28 2020-09-16T14:35:38.220Z [DEBUG] client: allocation updates: added=0 removed=0 updated=6 ignored=28 2020-09-16T14:35:38.220Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:35:38.220Z [TRACE] client: next heartbeat: period=10.178164909s 2020-09-16T14:35:38.222Z [DEBUG] worker: created evaluation: eval="" 2020-09-16T14:35:38.222Z [DEBUG] worker.service_sched: found reschedulable allocs, followup eval created: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd job_id=web-admin-worker namespace=default followup_eval_id=ba6fab80-c937-6c84-b125-b32664bf0433 2020-09-16T14:35:38.222Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:35:38.222Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:35:38.226Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:35:38.226Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:35:38.228Z [DEBUG] worker: submitted plan for evaluation: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd 2020-09-16T14:35:38.228Z [DEBUG] worker.service_sched: setting eval status: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd job_id=web-admin-worker namespace=default status=complete 2020-09-16T14:35:38.228Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:35:38.228Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:35:38.230Z [TRACE] client: next heartbeat: period=18.62996492s 2020-09-16T14:35:38.232Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:35:38.232Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:35:38.233Z [DEBUG] client: updated allocations: index=1532 total=34 pulled=7 filtered=27 2020-09-16T14:35:38.233Z [TRACE] client: next heartbeat: period=14.405656497s 2020-09-16T14:35:38.234Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T14:35:38.234Z [DEBUG] worker: ack evaluation: eval_id=459b09ca-e446-322b-ea5f-a0ad960093bd 2020-09-16T14:35:38.235Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:35:38.235Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:35:38.238Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:35:38.238Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=6 ignored=28 errors=0 2020-09-16T14:35:38.238Z [DEBUG] client: allocation updates: added=0 removed=0 updated=7 ignored=27 2020-09-16T14:35:38.238Z [TRACE] client: updating alloc: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 index=1444 2020-09-16T14:35:38.241Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=50991da7-d351-dec7-eab9-e62917223a95 modify_index=1444 2020-09-16T14:35:38.241Z [TRACE] client: updating alloc: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb index=1532 2020-09-16T14:35:38.242Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=0757fa38-d4ae-a692-a729-be89391f36eb modify_index=1532 2020-09-16T14:35:38.242Z [TRACE] client: updating alloc: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c index=1317 2020-09-16T14:35:38.244Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4c036d13-6c3d-09c5-85a3-cc5ad0feb94c modify_index=1317 2020-09-16T14:35:38.244Z [TRACE] client: updating alloc: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 index=1204 2020-09-16T14:35:38.245Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=e7cbbb51-2a6c-942f-caf3-36209d5ae649 modify_index=1204 2020-09-16T14:35:38.245Z [TRACE] client: updating alloc: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e index=658 2020-09-16T14:35:38.247Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=88d9cf3f-1470-cdbf-354e-c7245363188e modify_index=658 2020-09-16T14:35:38.247Z [TRACE] client: updating alloc: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 index=1376 2020-09-16T14:35:38.248Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=32911e9f-2e34-c4b4-f957-a1e728817c61 modify_index=1376 2020-09-16T14:35:38.248Z [TRACE] client: updating alloc: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 index=1333 2020-09-16T14:35:38.251Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=93ca4966-f455-5017-e21a-7a7c54bba1e4 modify_index=1333 2020-09-16T14:35:38.251Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=7 ignored=27 errors=0 2020-09-16T14:35:40.019Z [DEBUG] http: request complete: method=GET path=/v1/agent/monitor?client_id=9ee95205-0945-8494-15b6-74400d24799e&follow=true&log_level=trace&offset=50000&origin=end duration=18m59.326630707s ```
tgross commented 4 years ago

@jozef-slezak I've wrapped your logs in <detail> blocks to make it a little easier to navigate the issue. I'm realizing when I did so that you've got a single node that's running both the server and the client, so there's no need to provide separate logs (they're identical).

Having only a single node means that it's not possible for the server to have marked the client as lost during power shutdown, as they would both be shut down.

I see the task is a raw_exec, and I'm wondering if the workload got orphaned somehow. Can you post a process tree for the process (ex. ps afx)?

After you've got that but before continuing to debug, I would recommend shutting down Nomad (not the host) and verifying that there are no tasks still running that could be binding to that port (check with netstat). At this point you already know the host is in a weird state so we should make sure the previous problem isn't influencing any further tests.

jozef-slezak commented 4 years ago

ps afx (reproduced again)

 nomad alloc status ca0da6da-6856-f79a-6219-662bb1c130fb
ID                  = ca0da6da-6856-f79a-6219-662bb1c130fb
Eval ID             = 66435672
Name                = idispatcher.dispatcher[0]
Node ID             = 9ee95205
Node Name           = demo-inno77
Job ID              = idispatcher
Job Version         = 1
Client Status       = complete
Client Description  = All tasks have completed
Desired Status      = run
Desired Description = <none>
Created             = 1h9m ago
Modified            = 8m49s ago
Deployment ID       = cc9abc15
Deployment Health   = healthy

Task "dispatcher" is "dead"
Task Resources
CPU         Memory           Disk     Addresses
87/100 MHz  254 MiB/100 MiB  150 MiB  DISPATCHER: 127.0.0.1:21233

Task Events:
Started At     = 2020-09-16T13:53:24Z
Finished At    = 2020-09-16T14:53:55Z
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                  Type             Description
2020-09-16T14:53:55Z  Killed           Task successfully killed
2020-09-16T14:53:55Z  Terminated       Exit Code: 0, Exit Message: "executor: error waiting on process: rpc error: code = Canceled desc = grpc: the client connection is closing"
2020-09-16T14:53:55Z  Killing          Sent interrupt. Waiting 30s before force killing
2020-09-16T14:53:12Z  Started          Task started by client
2020-09-16T14:53:11Z  Task Setup       Building Task Directory
2020-09-16T14:20:38Z  Alloc Unhealthy  Task not running for min_healthy_time of 10s by deadline
2020-09-16T14:15:42Z  Started          Task started by client
2020-09-16T14:15:38Z  Task Setup       Building Task Directory
2020-09-16T14:15:37Z  Received         Task received by client
2020-09-16T14:01:42Z  Alloc Unhealthy  Task not running for min_healthy_time of 10s by deadline
[demo@demo-inno77 ~]$ ps aux | grep 21233
e     2112  0.0  0.0 115412   656 ?        S    14:53   0:00 /bin/bash -c cd /opte/natives;  exec &> >(tee -i -a /var/log/e/dispatcher-127.0.0.1-21233.log); exec ./IDispatcher -c /etcconf.d/idispatcher.xml -d 1
e     2117  0.0  0.0 108056   668 ?        S    14:53   0:00 tee -i -a /var/log/e/dispatcher-127.0.0.1-21233.log
demo      4163  0.0  0.0 112816   972 pts/0    S+   15:03   0:00 grep --color=auto 21233
[demo@demo-inno77 ~]$ ps afx 2117
  PID TTY      STAT   TIME COMMAND
    2 ?        S      0:00 [kthreadd]
    4 ?        S<     0:00  \_ [kworker/0:0H]
    6 ?        S      0:00  \_ [ksoftirqd/0]
    7 ?        S      0:00  \_ [migration/0]
    8 ?        S      0:00  \_ [rcu_bh]
    9 ?        S      0:01  \_ [rcu_sched]
   10 ?        S<     0:00  \_ [lru-add-drain]
   11 ?        S      0:00  \_ [watchdog/0]
   12 ?        S      0:00  \_ [watchdog/1]
   13 ?        S      0:00  \_ [migration/1]
   14 ?        S      0:00  \_ [ksoftirqd/1]
   16 ?        S<     0:00  \_ [kworker/1:0H]
   17 ?        S      0:00  \_ [watchdog/2]
   18 ?        S      0:00  \_ [migration/2]
   19 ?        S      0:00  \_ [ksoftirqd/2]
   21 ?        S<     0:00  \_ [kworker/2:0H]
   22 ?        S      0:01  \_ [watchdog/3]
   23 ?        S      0:00  \_ [migration/3]
   24 ?        S      0:00  \_ [ksoftirqd/3]
   26 ?        S<     0:00  \_ [kworker/3:0H]
   28 ?        S      0:00  \_ [kdevtmpfs]
   29 ?        S<     0:00  \_ [netns]
   30 ?        S      0:00  \_ [khungtaskd]
   31 ?        S<     0:00  \_ [writeback]
   32 ?        S<     0:00  \_ [kintegrityd]
   33 ?        S<     0:00  \_ [bioset]
   34 ?        S<     0:00  \_ [bioset]
   35 ?        S<     0:00  \_ [bioset]
   36 ?        S<     0:00  \_ [kblockd]
   37 ?        S<     0:00  \_ [md]
   38 ?        S<     0:00  \_ [edac-poller]
   39 ?        S<     0:00  \_ [watchdogd]
   40 ?        S      0:00  \_ [kworker/3:1]
   45 ?        S      0:00  \_ [kswapd0]
   46 ?        SN     0:00  \_ [ksmd]
   47 ?        SN     0:00  \_ [khugepaged]
   48 ?        S<     0:00  \_ [crypto]
   56 ?        S<     0:00  \_ [kthrotld]
   57 ?        S      0:00  \_ [kworker/u8:1]
   58 ?        S<     0:00  \_ [kmpath_rdacd]
   59 ?        S<     0:00  \_ [kaluad]
   61 ?        S<     0:00  \_ [kpsmoused]
   63 ?        S<     0:00  \_ [ipv6_addrconf]
   76 ?        S<     0:00  \_ [deferwq]
  113 ?        S      0:00  \_ [kauditd]
  229 ?        S      0:00  \_ [kworker/1:1]
  293 ?        S<     0:00  \_ [ata_sff]
  306 ?        S      0:00  \_ [scsi_eh_0]
  307 ?        S<     0:00  \_ [scsi_tmf_0]
  308 ?        S      0:00  \_ [scsi_eh_1]
  309 ?        S<     0:00  \_ [scsi_tmf_1]
  311 ?        S      0:00  \_ [kworker/u8:3]
  312 ?        S      0:00  \_ [scsi_eh_2]
  313 ?        S<     0:00  \_ [scsi_tmf_2]
  342 ?        S<     0:00  \_ [kworker/0:1H]
  346 ?        S<     0:00  \_ [kworker/1:1H]
  386 ?        S<     0:00  \_ [kdmflush]
  387 ?        S<     0:00  \_ [bioset]
  398 ?        S<     0:00  \_ [kdmflush]
  399 ?        S<     0:00  \_ [bioset]
  412 ?        S<     0:00  \_ [bioset]
  413 ?        S<     0:00  \_ [xfsalloc]
  414 ?        S<     0:00  \_ [xfs_mru_cache]
  415 ?        S<     0:00  \_ [xfs-buf/dm-0]
  416 ?        S<     0:00  \_ [xfs-data/dm-0]
  417 ?        S<     0:00  \_ [xfs-conv/dm-0]
  418 ?        S<     0:00  \_ [xfs-cil/dm-0]
  419 ?        S<     0:00  \_ [xfs-reclaim/dm-]
  420 ?        S<     0:00  \_ [xfs-log/dm-0]
  421 ?        S<     0:00  \_ [xfs-eofblocks/d]
  422 ?        S      0:00  \_ [xfsaild/dm-0]
  453 ?        S      0:00  \_ [kworker/1:2]
  612 ?        S      0:00  \_ [kworker/0:3]
  627 ?        S<     0:00  \_ [xfs-buf/sda1]
  628 ?        S<     0:00  \_ [xfs-data/sda1]
  629 ?        S<     0:00  \_ [xfs-conv/sda1]
  630 ?        S<     0:00  \_ [xfs-cil/sda1]
  631 ?        S<     0:00  \_ [xfs-reclaim/sda]
  632 ?        S<     0:00  \_ [xfs-log/sda1]
  633 ?        S<     0:00  \_ [xfs-eofblocks/s]
  634 ?        S      0:00  \_ [xfsaild/sda1]
  714 ?        S<     0:00  \_ [kworker/2:1H]
 1047 ?        S      0:00  \_ [kworker/2:2]
 1070 ?        S<     0:00  \_ [kworker/3:1H]
 3626 ?        S      0:00  \_ [kworker/3:0]
 3627 ?        S      0:00  \_ [kworker/0:0]
 3912 ?        S      0:00  \_ [kworker/2:0]
 4143 ?        S      0:00  \_ [kworker/3:2]
 4144 ?        S      0:00  \_ [kworker/0:1]
    1 ?        Ss     0:01 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
  503 ?        Ss     0:00 /usr/lib/systemd/systemd-journald
  528 ?        Ss     0:00 /usr/sbin/lvmetad -f
  539 ?        Ss     0:00 /usr/lib/systemd/systemd-udevd
  657 ?        S<sl   0:00 /sbin/auditd
  680 ?        Ssl    0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
  683 ?        Ss     0:00 /usr/sbin/irqbalance --foreground
  685 ?        Ss     0:00 /usr/lib/systemd/systemd-logind
  692 ?        Ssl    0:00 /usr/lib/polkit-1/polkitd --no-debug
  693 ?        Ss     0:00 /usr/sbin/abrtd -d -s
  694 ?        Ss     0:00 /usr/bin/abrt-watch-log -F BUG: WARNING: at WARNING: CPU: INFO: possible recursive locking detected ernel BUG at list_del corruption
  699 ?        Ss     0:00 /usr/sbin/crond -n
  703 ?        Ss     0:00 /usr/sbin/acpid
  711 ?        Ss     0:00 login -- demo
 3756 tty1     Ss+    0:00  \_ -bash
 3795 tty1     S+     0:00      \_ /bin/bash /home/demo/.menu.sh
  713 ?        Ss     0:00 /usr/sbin/ntpd -u ntp:ntp -g
  733 ?        Ssl    0:00 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid
  735 ?        Ssl    0:00 /usr/sbin/NetworkManager --no-daemon
  860 ?        S      0:00  \_ /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-enp0s3.pid -lf /var/lib/NetworkManager/dhclient-95ec9
  869 ?        S      0:00  \_ /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/NetworkManager/dnsmasq.pid -
 1041 ?        Ssl    0:00 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
 1044 ?        Ss     0:00 /usr/sbin/sshd -D
 4061 ?        Ss     0:00  \_ sshd: demo [priv]
 4065 ?        S      0:00      \_ sshd: demo@pts/0
 4066 pts/0    Ss     0:00          \_ -bash
 4284 pts/0    R+     0:00              \_ ps afx 2117
 1045 ?        Ssl    0:00 /usr/sbin/rsyslogd -n
 1046 ?        Ssl    0:00 /usr/bin/redis-server 127.0.0.1:6379
 1048 ?        Ssl    0:14 /usr/sbin/consul agent -config-dir=/etcconsul.d -ui-content-path=/consul/ui
 1049 ?        Ssl    0:02 /usr/sbin/traefik --consul --consul.endpoint=127.0.0.1:8500
 1054 ?        Ssl    0:24 /usr/sbin/nomad agent -config=/etcnomad.d
 1947 ?        Sl     0:01  \_ /usr/sbin/nomad logmon
 1970 ?        Sl     0:01  \_ /usr/sbin/nomad logmon
 2000 ?        Sl     0:01  \_ /usr/sbin/nomad logmon
 2035 ?        Ssl    0:12  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/3954c06e-8fbc-c467-e2f0-c15fb0863f5a/jaeger-collector/exec
 2063 ?        Sl     0:00  |   \_ /usr/bin/jaeger-all-in-one --memory.max-traces=10000
 2069 ?        S      0:00  |       \_ /bin/bash -c exec &> >(tee -i -a /var/log/e/jaeger-collector-127.0.0.1-14267.log); exec /usr/bin/jaeger-all-in-one -
 2070 ?        S      0:00  |           \_ tee -i -a /var/log/e/jaeger-collector-127.0.0.1-14267.log
 2088 ?        Ssl    0:11  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/ca0da6da-6856-f79a-6219-662bb1c130fb/dispatcher/executor.o
 2110 ?        Sl     0:20  |   \_ ./IDispatcher -c /etcconf.d/idispatcher.xml -d 1
 2112 ?        S      0:00  |       \_ /bin/bash -c cd /opte/natives;  exec &> >(tee -i -a /var/log/e/dispatcher-127.0.0.1-21233.log); exe
 2117 ?        S      0:00  |           \_ tee -i -a /var/log/e/dispatcher-127.0.0.1-21233.log
 2129 ?        Ssl    0:11  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/efbb3557-0148-9ddc-617b-8eb7eba053d6/web-webserver/exe
 2158 ?        Sl     0:44  |   \_ puma 3.12.0 (tcp:0.0.0.0:3001) [webserver:3001]
 2163 ?        S      0:00  |       \_ /bin/bash -c cd /optweb/current; exec &> >(tee -i -a /var/log/innrm/web_webserver-127.0.0.1-3001.lo
 2167 ?        S      0:00  |           \_ tee -i -a /var/log/innrm/web_webserver-127.0.0.1-3001.log
 2182 ?        Sl     0:02  \_ /usr/sbin/nomad logmon
 2191 ?        Sl     0:01  \_ /usr/sbin/nomad logmon
 2197 ?        Sl     0:01  \_ /usr/sbin/nomad logmon
 2218 ?        Sl     0:01  \_ /usr/sbin/nomad logmon
 2228 ?        Ssl    0:11  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/010bca37-809b-4db6-057a-223bd9e0769e/web-mq-workflow-w
 2246 ?        Sl     0:32  |   \_ puma 3.12.0 (tcp:0.0.0.0:3200) [workflow worker:3200]
 2251 ?        S      0:00  |       \_ /bin/bash -c cd /optweb/current; exec &> >(tee -i -a /var/log/innrm/web_mq_workflow_worker-127.0.0.
 2253 ?        S      0:00  |           \_ tee -i -a /var/log/innrm/web_mq_workflow_worker-127.0.0.1-3200.log
 2236 ?        Ssl    0:14  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/efb90a4d-ba33-3595-3680-f081bd428b54/web-hitlist-worke
 2266 ?        S      0:00  |   \_ /bin/bash ./bin/production_run_webserver.sh
 2293 ?        S      0:00  |       \_ /bin/bash -c cd /optweb/hitlist_worker; exec &> >(tee -i -a /var/log/innrm/web_hitlist_worker-127.0
 2297 ?        S      0:00  |       |   \_ tee -i -a /var/log/innrm/web_hitlist_worker-127.0.0.1-3400.log
 2302 ?        Sl     0:10  |       \_ puma 3.12.0 (tcp:0.0.0.0:3400) [hitlist_worker]
 2239 ?        Sl     0:01  \_ /usr/sbin/nomad logmon
 2260 ?        Ssl    0:11  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/da054000-3d76-7eb1-8b57-5e580849e3de/web-delayed-job/e
 2281 ?        Sl     0:42  |   \_ puma 3.12.0 (tcp:0.0.0.0:3300) [delayed_job:3300]
 2284 ?        S      0:00  |       \_ /bin/bash -c cd /optweb/current; exec &> >(tee -i -a /var/log/innrm/web_delayed_job-127.0.0.1-3300.
 2286 ?        S      0:00  |           \_ tee -i -a /var/log/innrm/web_delayed_job-127.0.0.1-3300.log
 2287 ?        Ssl    0:12  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/0027b62a-6483-5a64-d746-ceb8e8b13e86/web--worker/e
 2315 ?        Sl     0:32  |   \_ puma 3.12.0 (tcp:0.0.0.0:3202) [ worker:3202]
 2321 ?        S      0:00  |       \_ /bin/bash -c cd /optweb/current; exec &> >(tee -i -a /var/log/innrm/web__worker-127.0.0.1-3202.
 2323 ?        S      0:00  |           \_ tee -i -a /var/log/innrm/web__worker-127.0.0.1-3202.log
 2306 ?        Ssl    0:11  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/54c0d11a-06ea-46df-49c9-3487cd77452a/web-delayed-job-w
 2334 ?        Sl     0:41  |   \_ puma 3.12.0 (tcp:0.0.0.0:3203) [delayed_job:3203]
 2339 ?        S      0:00  |       \_ /bin/bash -c cd /optweb/web_customization && exec &> >(tee -i -a /var/log/innrm/WEB_DELAYED_JOB
 2342 ?        S      0:00  |           \_ tee -i -a /var/log/innrm/WEB_DELAYED_JOB_WEB_CACHE-127.0.0.1-3203.log
 2352 ?        Sl     0:02  \_ /usr/sbin/nomad logmon
 2356 ?        Sl     0:01  \_ /usr/sbin/nomad logmon
 2376 ?        Sl     0:01  \_ /usr/sbin/nomad logmon
 2391 ?        Sl     0:02  \_ /usr/sbin/nomad logmon
 2423 ?        Ssl    0:11  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/19368a06-1e30-432b-3b25-4f8a0609b8fe/message-processor/exe
 2471 ?        Sl     1:04  |   \_ java -Xmx1024m -Djava.net.preferIPv4Stack=true -XX:-OmitStackTraceInFastThrow -D.cluster_name=DEMO-CLUSTER -D.rest_h
 2482 ?        S      0:00  |       \_ /bin/bash -c exec &> >(tee -i -a /var/log/e/e-service-127.0.0.1-8585.log); exec /bin/bash /opte
 2488 ?        S      0:00  |           \_ tee -i -a /var/log/e/e-service-127.0.0.1-8585.log
 2429 ?        Ssl    0:11  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/60c5e684-7314-5859-6ac1-2e66b4d2b943/fast-service/executor
 2477 ?        Sl     0:15  |   \_ java -Xmx512m -Djava.net.preferIPv4Stack=true -XX:-OmitStackTraceInFastThrow -XX:+ExitOnOutOfMemoryError -Djava.awt.headless
 2481 ?        S      0:00  |       \_ /bin/bash -c exec &> >(tee -i -a  /var/log/e/fast-service-127.0.0.1-22907.log); exec /bin/bash /opteafi
 2487 ?        S      0:00  |           \_ tee -i -a /var/log/e/fast-service-127.0.0.1-22907.log
 2435 ?        Ssl    0:11  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/cabc6570-da2e-a842-d6de-e84874e5c6a4/matcher-numa0/executo
 2480 ?        Sl     0:00  |   \_ ./INode -c /etcconf.d/inode0.xml -d 1
 2493 ?        S      0:00  |       \_ /bin/bash -c cd /opte/natives; exec &> >(tee -i -a /var/log/e/matcher-127.0.0.1-25780.log); exec ./
 2496 ?        S      0:00  |           \_ tee -i -a /var/log/e/matcher-127.0.0.1-25780.log
 2458 ?        Ssl    0:11  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/650c389a-686c-497c-f62b-80bd26c6a642/pre-extractor/executo
 2489 ?        Sl     0:15  |   \_ java -Xmx512m -Djava.net.preferIPv4Stack=true -XX:-OmitStackTraceInFastThrow -XX:+ExitOnOutOfMemoryError -Djava.awt.headless
 2497 ?        S      0:00  |       \_ /bin/bash -c exec &> >(tee -i -a  /var/log/e/fast-service-pre-extractor-127.0.0.1-20189.log); exec /bin/bash /opt/in
 2504 ?        S      0:00  |           \_ tee -i -a /var/log/e/fast-service-pre-extractor-127.0.0.1-20189.log
 3055 ?        Sl     0:01  \_ /usr/sbin/nomad logmon
 3076 ?        Ssl    0:10  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/bf0705b4-7ad5-3b35-8071-31642cfb21ac/activemq/executor.out
 3085 ?        Sl     0:28  |   \_ /usr/bin/java -Xmx2048M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties
 3086 ?        S      0:00  |       \_ /bin/bash -c exec &> >(tee -i -a /var/log/activemq/activemq-127.0.0.1-61616.log); exec /bin/bash /optextern
 3089 ?        S      0:00  |           \_ tee -i -a /var/log/activemq/activemq-127.0.0.1-61616.log
 3094 ?        Sl     0:01  \_ /usr/sbin/nomad logmon
 3113 ?        Ssl    0:10  \_ /usr/sbin/nomad executor {"LogFile":"/var/libnomad/alloc/b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b/monitoring/execu
 3126 ?        Sl     0:07      \_ java -Xms64m -Xmx256m -Djava.net.preferIPv4Stack=true -XX:-OmitStackTraceInFastThrow -XX:+ExitOnOutOfMemoryError -Djava.awt.
 3129 ?        S      0:00          \_ /bin/bash -c exec &> >(tee -i -a /var/log/innrm/monitoring-127.0.0.1-8787.log); exec /bin/bash /optabi
 3133 ?        S      0:00              \_ tee -i -a /var/log/innrm/monitoring-127.0.0.1-8787.log
 1096 ?        S      0:00 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
 1158 ?        Ss     0:00  \_ postgres: logger process
 1614 ?        Ss     0:00  \_ postgres: checkpointer process
 1615 ?        Ss     0:00  \_ postgres: writer process
 1616 ?        Ss     0:00  \_ postgres: wal writer process
 1617 ?        Ss     0:00  \_ postgres: autovacuum launcher process
 1618 ?        Ss     0:00  \_ postgres: stats collector process
 2702 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(57638) idle
 2703 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(57640) idle
 2704 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(57642) idle
 2706 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(57644) idle
 2707 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(57646) idle
 2708 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(57648) idle
 2714 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(57650) idle
 2717 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(57652) idle
 2965 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(57892) idle
 2966 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(57894) idle
 2967 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(57896) idle
 3022 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(58020) idle
 3023 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(58022) idle
 3024 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(58024) idle
 3025 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(58026) idle
 3026 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(58028) idle
 3037 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(58052) idle
 3386 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(58366) idle
 3400 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(58368) idle
 3937 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(60100) idle
 4230 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(33442) idle
 4231 ?        Ss     0:00  \_ postgres: e e 127.0.0.1(33444) idle
 1370 ?        Ss     0:00 /usr/libexec/postfix/master -w
 1394 ?        S      0:00  \_ pickup -l -t unix -u
 1395 ?        S      0:00  \_ qmgr -l -t unix -u

The process is definitely orphaned. What about the wrapping shell script - could it cause some trouble? Basically every time I power off I can reproduce the problem but with different job.

I am going to reproduce that again using your advice: a. shutting down nomad (not host) b. netstat c. power off

jozef-slezak commented 4 years ago

I stopped nomad using systemclt and then killed all nomad, java and puma processes. Process tree look like this and ports are not opened anymore.

ps afx 2117
  PID TTY      STAT   TIME COMMAND
    2 ?        S      0:00 [kthreadd]
    4 ?        S<     0:00  \_ [kworker/0:0H]
    6 ?        S      0:00  \_ [ksoftirqd/0]
    7 ?        S      0:00  \_ [migration/0]
    8 ?        S      0:00  \_ [rcu_bh]
    9 ?        S      0:03  \_ [rcu_sched]
   10 ?        S<     0:00  \_ [lru-add-drain]
   11 ?        S      0:00  \_ [watchdog/0]
   12 ?        S      0:00  \_ [watchdog/1]
   13 ?        S      0:00  \_ [migration/1]
   14 ?        S      0:00  \_ [ksoftirqd/1]
   16 ?        S<     0:00  \_ [kworker/1:0H]
   17 ?        S      0:00  \_ [watchdog/2]
   18 ?        S      0:00  \_ [migration/2]
   19 ?        S      0:00  \_ [ksoftirqd/2]
   21 ?        S<     0:00  \_ [kworker/2:0H]
   22 ?        S      0:01  \_ [watchdog/3]
   23 ?        S      0:00  \_ [migration/3]
   24 ?        S      0:00  \_ [ksoftirqd/3]
   26 ?        S<     0:00  \_ [kworker/3:0H]
   28 ?        S      0:00  \_ [kdevtmpfs]
   29 ?        S<     0:00  \_ [netns]
   30 ?        S      0:00  \_ [khungtaskd]
   31 ?        S<     0:00  \_ [writeback]
   32 ?        S<     0:00  \_ [kintegrityd]
   33 ?        S<     0:00  \_ [bioset]
   34 ?        S<     0:00  \_ [bioset]
   35 ?        S<     0:00  \_ [bioset]
   36 ?        S<     0:00  \_ [kblockd]
   37 ?        S<     0:00  \_ [md]
   38 ?        S<     0:00  \_ [edac-poller]
   39 ?        S<     0:00  \_ [watchdogd]
   45 ?        S      0:00  \_ [kswapd0]
   46 ?        SN     0:00  \_ [ksmd]
   47 ?        SN     0:00  \_ [khugepaged]
   48 ?        S<     0:00  \_ [crypto]
   56 ?        S<     0:00  \_ [kthrotld]
   57 ?        S      0:00  \_ [kworker/u8:1]
   58 ?        S<     0:00  \_ [kmpath_rdacd]
   59 ?        S<     0:00  \_ [kaluad]
   61 ?        S<     0:00  \_ [kpsmoused]
   63 ?        S<     0:00  \_ [ipv6_addrconf]
   76 ?        S<     0:00  \_ [deferwq]
  113 ?        S      0:00  \_ [kauditd]
  293 ?        S<     0:00  \_ [ata_sff]
  306 ?        S      0:00  \_ [scsi_eh_0]
  307 ?        S<     0:00  \_ [scsi_tmf_0]
  308 ?        S      0:00  \_ [scsi_eh_1]
  309 ?        S<     0:00  \_ [scsi_tmf_1]
  311 ?        S      0:00  \_ [kworker/u8:3]
  312 ?        S      0:00  \_ [scsi_eh_2]
  313 ?        S<     0:00  \_ [scsi_tmf_2]
  342 ?        S<     0:00  \_ [kworker/0:1H]
  346 ?        S<     0:00  \_ [kworker/1:1H]
  386 ?        S<     0:00  \_ [kdmflush]
  387 ?        S<     0:00  \_ [bioset]
  398 ?        S<     0:00  \_ [kdmflush]
  399 ?        S<     0:00  \_ [bioset]
  412 ?        S<     0:00  \_ [bioset]
  413 ?        S<     0:00  \_ [xfsalloc]
  414 ?        S<     0:00  \_ [xfs_mru_cache]
  415 ?        S<     0:00  \_ [xfs-buf/dm-0]
  416 ?        S<     0:00  \_ [xfs-data/dm-0]
  417 ?        S<     0:00  \_ [xfs-conv/dm-0]
  418 ?        S<     0:00  \_ [xfs-cil/dm-0]
  419 ?        S<     0:00  \_ [xfs-reclaim/dm-]
  420 ?        S<     0:00  \_ [xfs-log/dm-0]
  421 ?        S<     0:00  \_ [xfs-eofblocks/d]
  422 ?        S      0:00  \_ [xfsaild/dm-0]
  453 ?        S      0:00  \_ [kworker/1:2]
  612 ?        S      0:00  \_ [kworker/0:3]
  627 ?        S<     0:00  \_ [xfs-buf/sda1]
  628 ?        S<     0:00  \_ [xfs-data/sda1]
  629 ?        S<     0:00  \_ [xfs-conv/sda1]
  630 ?        S<     0:00  \_ [xfs-cil/sda1]
  631 ?        S<     0:00  \_ [xfs-reclaim/sda]
  632 ?        S<     0:00  \_ [xfs-log/sda1]
  633 ?        S<     0:00  \_ [xfs-eofblocks/s]
  634 ?        S      0:00  \_ [xfsaild/sda1]
  714 ?        S<     0:00  \_ [kworker/2:1H]
 1047 ?        S      0:00  \_ [kworker/2:2]
 1070 ?        S<     0:00  \_ [kworker/3:1H]
 3912 ?        S      0:00  \_ [kworker/2:0]
 4353 ?        S      0:00  \_ [kworker/3:1]
 4393 ?        S      0:00  \_ [kworker/1:0]
 4494 ?        S      0:00  \_ [kworker/0:1]
 4540 ?        S      0:00  \_ [kworker/3:0]
 4611 ?        S      0:00  \_ [kworker/0:0]
 4653 ?        S      0:00  \_ [kworker/3:2]
 4663 ?        S      0:00  \_ [kworker/2:1]
    1 ?        Ss     0:01 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
  503 ?        Ss     0:01 /usr/lib/systemd/systemd-journald
  528 ?        Ss     0:00 /usr/sbin/lvmetad -f
  539 ?        Ss     0:00 /usr/lib/systemd/systemd-udevd
  657 ?        S<sl   0:00 /sbin/auditd
  680 ?        Ssl    0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
  683 ?        Ss     0:00 /usr/sbin/irqbalance --foreground
  685 ?        Ss     0:00 /usr/lib/systemd/systemd-logind
  692 ?        Ssl    0:00 /usr/lib/polkit-1/polkitd --no-debug
  693 ?        Ss     0:00 /usr/sbin/abrtd -d -s
  694 ?        Ss     0:00 /usr/bin/abrt-watch-log -F BUG: WARNING: at WARNING: CPU: INFO: possible recursive locking detected ernel BUG at list_del corruption list_add corruption do_IRQ: stack overflow: ear stack overflow (cur: eneral p
  699 ?        Ss     0:00 /usr/sbin/crond -n
  703 ?        Ss     0:00 /usr/sbin/acpid
  711 ?        Ss     0:00 login -- demo
 3756 tty1     Ss+    0:00  \_ -bash
 3795 tty1     S+     0:00      \_ /bin/bash /home/demo/.menu.sh
  713 ?        Ss     0:00 /usr/sbin/ntpd -u ntp:ntp -g
  733 ?        Ssl    0:00 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid
  735 ?        Ssl    0:00 /usr/sbin/NetworkManager --no-daemon
  860 ?        S      0:00  \_ /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-enp0s3.pid -lf /var/lib/NetworkManager/dhclient-95ec9a00-abd1-4be9-a1e7-6f6d34d6cce7-enp0s3.lease -cf /var/lib/NetworkManager/dhcli
  869 ?        S      0:00  \_ /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/NetworkManager/dnsmasq.pid --listen-address=127.0.0.1 --cache-size=400 --clear-on-reload --conf-file=/dev/
 1041 ?        Ssl    0:00 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
 1044 ?        Ss     0:00 /usr/sbin/sshd -D
 4061 ?        Ss     0:00  \_ sshd: demo [priv]
 4065 ?        S      0:00      \_ sshd: demo@pts/0
 4066 pts/0    Ss     0:00          \_ -bash
 4676 pts/0    R+     0:00              \_ ps afx 2117
 1045 ?        Ssl    0:00 /usr/sbin/rsyslogd -n
 1046 ?        Ssl    0:01 /usr/bin/redis-server 127.0.0.1:6379
 1048 ?        Ssl    0:20 /usr/sbin/consul agent -config-dir=/etc/consul.d -ui-content-path=/consul/ui
 1049 ?        Ssl    0:03 /usr/sbin/traefik --consul --consul.endpoint=127.0.0.1:8500
 1096 ?        S      0:00 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
 1158 ?        Ss     0:00  \_ postgres: logger process
 1614 ?        Ss     0:00  \_ postgres: checkpointer process
 1615 ?        Ss     0:00  \_ postgres: writer process
 1616 ?        Ss     0:00  \_ postgres: wal writer process
 1617 ?        Ss     0:00  \_ postgres: autovacuum launcher process
 1618 ?        Ss     0:00  \_ postgres: stats collector process
 1370 ?        Ss     0:00 /usr/libexec/postfix/master -w
 1394 ?        S      0:00  \_ pickup -l -t unix -u
 1395 ?        S      0:00  \_ qmgr -l -t unix -u
 2063 ?        Sl     0:01 /usr/bin/jaeger-all-in-one --memory.max-traces=10000
 2069 ?        S      0:00  \_ /bin/bash -c exec &> >(tee -i -a /var/log/e/jaeger-collector-127.0.0.1-14267.log); exec /usr/bin/jaeger-all-in-one --memory.max-traces=10000
 2070 ?        S      0:00      \_ tee -i -a /var/log/e/jaeger-collector-127.0.0.1-14267.log

and going to power off => same behaviour - orphan on different job

``` 8449ccd-b112-fb6b-208d-0f6ce0163173 job_id=monitoring namespace=default results="Total changes: (place 0) (destructive 0) (inplace 0) (stop 0) Desired Changes for "monitoring": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 1) (canary 0)" 2020-09-16T15:29:29.563Z [DEBUG] worker.service_sched: setting eval status: eval_id=58449ccd-b112-fb6b-208d-0f6ce0163173 job_id=monitoring namespace=default status=complete 2020-09-16T15:29:29.565Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 modify_index=1980 2020-09-16T15:29:29.565Z [TRACE] client: updating alloc: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 index=1952 2020-09-16T15:29:29.567Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T15:29:29.567Z [DEBUG] worker: ack evaluation: eval_id=58449ccd-b112-fb6b-208d-0f6ce0163173 2020-09-16T15:29:29.567Z [DEBUG] worker: dequeued evaluation: eval_id=e5c31edc-c4aa-af0c-6483-7ecc51180fbc 2020-09-16T15:29:29.567Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=e5c31edc-c4aa-af0c-6483-7ecc51180fbc job_id=idispatcher namespace=default algorithm=binpack 2020-09-16T15:29:29.567Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=e5c31edc-c4aa-af0c-6483-7ecc51180fbc job_id=idispatcher namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 1) Desired Changes for "dispatcher": (place 1) (inplace 0) (destructive 0) (stop 1) (migrate 0) (ignore 1) (canary 0)" 2020-09-16T15:29:29.569Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 modify_index=1952 2020-09-16T15:29:29.569Z [TRACE] client: updating alloc: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe index=1960 2020-09-16T15:29:29.570Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker path=/usr/sbin/nomad args=[/usr/sbin/nomad, logmon] 2020-09-16T15:29:29.570Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker path=/usr/sbin/nomad pid=4014 2020-09-16T15:29:29.570Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker path=/usr/sbin/nomad 2020-09-16T15:29:29.571Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe modify_index=1960 2020-09-16T15:29:29.571Z [INFO] client.gc: garbage collecting allocation: alloc_id=585acbd3-0347-8626-f5c6-34679aa09f41 reason="new allocations and over max (50)" 2020-09-16T15:29:29.571Z [TRACE] client.alloc_runner.task_runner: Kill requested: alloc_id=585acbd3-0347-8626-f5c6-34679aa09f41 task=fast-service event_type=Killing event_reason= 2020-09-16T15:29:29.578Z [DEBUG] worker: submitted plan for evaluation: eval_id=e5c31edc-c4aa-af0c-6483-7ecc51180fbc 2020-09-16T15:29:29.578Z [DEBUG] worker.service_sched: setting eval status: eval_id=e5c31edc-c4aa-af0c-6483-7ecc51180fbc job_id=idispatcher namespace=default status=complete 2020-09-16T15:29:29.578Z [INFO] client.gc: marking allocation for GC: alloc_id=585acbd3-0347-8626-f5c6-34679aa09f41 2020-09-16T15:29:29.581Z [DEBUG] client.gc: alloc garbage collected: alloc_id=585acbd3-0347-8626-f5c6-34679aa09f41 2020-09-16T15:29:29.584Z [DEBUG] client: allocation updates applied: added=1 removed=0 updated=6 ignored=48 errors=0 2020-09-16T15:29:29.584Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T15:29:29.584Z [DEBUG] worker: ack evaluation: eval_id=e5c31edc-c4aa-af0c-6483-7ecc51180fbc 2020-09-16T15:29:29.584Z [DEBUG] worker: dequeued evaluation: eval_id=e577c473-ac4c-7254-8aed-d00fd36071f7 2020-09-16T15:29:29.584Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=e577c473-ac4c-7254-8aed-d00fd36071f7 job_id=web-webserver namespace=default algorithm=binpack 2020-09-16T15:29:29.584Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=e577c473-ac4c-7254-8aed-d00fd36071f7 job_id=web-webserver namespace=default results="Total changes: (place 0) (destructive 0) (inplace 0) (stop 0) Desired Changes for "web-webserver": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 1) (canary 0)" 2020-09-16T15:29:29.584Z [DEBUG] worker.service_sched: setting eval status: eval_id=e577c473-ac4c-7254-8aed-d00fd36071f7 job_id=web-webserver namespace=default status=complete 2020-09-16T15:29:29.585Z [DEBUG] client: updated allocations: index=1991 total=56 pulled=9 filtered=47 2020-09-16T15:29:29.585Z [TRACE] client: next heartbeat: period=12.694107367s 2020-09-16T15:29:29.585Z [DEBUG] client: allocation updates: added=1 removed=0 updated=7 ignored=48 2020-09-16T15:29:29.585Z [TRACE] client: updating alloc: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 index=1946 2020-09-16T15:29:29.587Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T15:29:29.587Z [DEBUG] worker: ack evaluation: eval_id=e577c473-ac4c-7254-8aed-d00fd36071f7 2020-09-16T15:29:29.588Z [DEBUG] worker: dequeued evaluation: eval_id=de580d4f-fe0f-8f1a-fd6e-68f0f1d3578b 2020-09-16T15:29:29.588Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=de580d4f-fe0f-8f1a-fd6e-68f0f1d3578b job_id=activemq namespace=default algorithm=binpack 2020-09-16T15:29:29.588Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=de580d4f-fe0f-8f1a-fd6e-68f0f1d3578b job_id=activemq namespace=default results="Total changes: (place 0) (destructive 0) (inplace 0) (stop 0) Desired Changes for "activemq": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 1) (canary 0)" 2020-09-16T15:29:29.588Z [DEBUG] worker.service_sched: setting eval status: eval_id=de580d4f-fe0f-8f1a-fd6e-68f0f1d3578b job_id=activemq namespace=default status=complete 2020-09-16T15:29:29.591Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker version=2 2020-09-16T15:29:29.591Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker @module=logmon address=/tmp/plugin617814454 network=unix timestamp=2020-09-16T15:29:29.590Z 2020-09-16T15:29:29.600Z [DEBUG] client.alloc_migrator: waiting for previous alloc to terminate: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c previous_alloc=3954c06e-8fbc-c467-e2f0-c15fb0863f5a 2020-09-16T15:29:29.600Z [DEBUG] client.alloc_migrator: waiting for previous alloc to terminate: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c previous_alloc=3954c06e-8fbc-c467-e2f0-c15fb0863f5a 2020-09-16T15:29:29.600Z [TRACE] client.alloc_runner.runner_hook.alloc_health_watcher: watching: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c deadline="2020-09-16 15:34:29.600611419 +0000 UTC m=+613.616424185" checks=true min_healthy_time=10s 2020-09-16T15:29:29.602Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 modify_index=1946 2020-09-16T15:29:29.602Z [TRACE] client: updating alloc: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe index=1960 2020-09-16T15:29:29.602Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker @module=logmon path=/var/lib/nomad/alloc/f074ee01-c03f-ff00-3037-ed4a68b216d1/alloc/logs/.web-hitlist-worker.stdout.fifo timestamp=2020-09-16T15:29:29.602Z 2020-09-16T15:29:29.603Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker @module=logmon path=/var/lib/nomad/alloc/f074ee01-c03f-ff00-3037-ed4a68b216d1/alloc/logs/.web-hitlist-worker.stderr.fifo timestamp=2020-09-16T15:29:29.603Z 2020-09-16T15:29:29.604Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T15:29:29.604Z [DEBUG] worker: ack evaluation: eval_id=de580d4f-fe0f-8f1a-fd6e-68f0f1d3578b 2020-09-16T15:29:29.604Z [DEBUG] worker: dequeued evaluation: eval_id=1795b407-f705-1b66-0683-b9bf9ca2116c 2020-09-16T15:29:29.604Z [TRACE] worker.service_sched.binpack: NewBinPackIterator created: eval_id=1795b407-f705-1b66-0683-b9bf9ca2116c job_id=matcher0 namespace=default algorithm=binpack 2020-09-16T15:29:29.604Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=1795b407-f705-1b66-0683-b9bf9ca2116c job_id=matcher0 namespace=default results="Total changes: (place 0) (destructive 0) (inplace 0) (stop 0) Desired Changes for "matcher": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 1) (canary 0)" 2020-09-16T15:29:29.604Z [DEBUG] worker.service_sched: setting eval status: eval_id=1795b407-f705-1b66-0683-b9bf9ca2116c job_id=matcher0 namespace=default status=complete 2020-09-16T15:29:29.605Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe modify_index=1960 2020-09-16T15:29:29.605Z [TRACE] client: updating alloc: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 index=1952 2020-09-16T15:29:29.607Z [DEBUG] worker: updated evaluation: eval="" 2020-09-16T15:29:29.607Z [DEBUG] worker: ack evaluation: eval_id=1795b407-f705-1b66-0683-b9bf9ca2116c 2020-09-16T15:29:29.610Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 modify_index=1952 2020-09-16T15:29:29.610Z [TRACE] client: updating alloc: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b index=1944 2020-09-16T15:29:29.612Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c done=false 2020-09-16T15:29:29.612Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c client_status=pending desired_status= 2020-09-16T15:29:29.618Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b modify_index=1944 2020-09-16T15:29:29.618Z [TRACE] client: updating alloc: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 index=1980 2020-09-16T15:29:29.620Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c task=jaeger-collector path=/usr/sbin/nomad args=[/usr/sbin/nomad, logmon] 2020-09-16T15:29:29.620Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c task=jaeger-collector path=/usr/sbin/nomad pid=4024 2020-09-16T15:29:29.620Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c task=jaeger-collector path=/usr/sbin/nomad 2020-09-16T15:29:29.632Z [DEBUG] client.alloc_runner: Discarding allocation update: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 skipped_modify_index=1836 new_modify_index=1980 2020-09-16T15:29:29.632Z [TRACE] client: updating alloc: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb index=1991 2020-09-16T15:29:29.641Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb modify_index=1991 2020-09-16T15:29:29.641Z [TRACE] client: updating alloc: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a index=1985 2020-09-16T15:29:29.656Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c task=jaeger-collector version=2 2020-09-16T15:29:29.660Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a modify_index=1985 2020-09-16T15:29:29.660Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c task=jaeger-collector address=/tmp/plugin124268286 network=unix @module=logmon timestamp=2020-09-16T15:29:29.656Z 2020-09-16T15:29:29.660Z [INFO] client.gc: garbage collecting allocation: alloc_id=29883c59-0cdd-02fd-08e5-df0c35292aa0 reason="new allocations and over max (50)" 2020-09-16T15:29:29.660Z [TRACE] client.alloc_runner.task_runner: Kill requested: alloc_id=29883c59-0cdd-02fd-08e5-df0c35292aa0 task=activemq event_type=Killing event_reason= 2020-09-16T15:29:29.662Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c task=jaeger-collector path=/var/lib/nomad/alloc/0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c/alloc/logs/.jaeger-collector.stdout.fifo @module=logmon timestamp=2020-09-16T15:29:29.662Z 2020-09-16T15:29:29.663Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c task=jaeger-collector @module=logmon path=/var/lib/nomad/alloc/0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c/alloc/logs/.jaeger-collector.stderr.fifo timestamp=2020-09-16T15:29:29.662Z 2020-09-16T15:29:29.664Z [INFO] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/web/hitlist_worker; exec &> >(tee -i -a /var/log/innrm/web_hitlist_worker-::1-3400.log); exec /bin/bash ./bin/production_run_webserver.sh]}" 2020-09-16T15:29:29.664Z [DEBUG] client.driver_mgr.raw_exec.executor: starting plugin: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 driver=raw_exec task_name=web-hitlist-worker path=/usr/sbin/nomad args=[/usr/sbin/nomad, executor, {"LogFile":"/var/lib/nomad/alloc/f074ee01-c03f-ff00-3037-ed4a68b216d1/web-hitlist-worker/executor.out","LogLevel":"debug","FSIsolation":false}] 2020-09-16T15:29:29.665Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin started: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 driver=raw_exec task_name=web-hitlist-worker path=/usr/sbin/nomad pid=4033 2020-09-16T15:29:29.665Z [DEBUG] client.driver_mgr.raw_exec.executor: waiting for RPC address: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 driver=raw_exec task_name=web-hitlist-worker path=/usr/sbin/nomad 2020-09-16T15:29:29.665Z [INFO] client.gc: marking allocation for GC: alloc_id=29883c59-0cdd-02fd-08e5-df0c35292aa0 2020-09-16T15:29:29.674Z [DEBUG] client.gc: alloc garbage collected: alloc_id=29883c59-0cdd-02fd-08e5-df0c35292aa0 2020-09-16T15:29:29.682Z [DEBUG] client: allocation updates applied: added=1 removed=0 updated=7 ignored=48 errors=0 2020-09-16T15:29:29.683Z [TRACE] client.alloc_migrator: previous allocation already terminated: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 previous_alloc=ca0da6da-6856-f79a-6219-662bb1c130fb 2020-09-16T15:29:29.683Z [TRACE] client.alloc_migrator: previous allocation already terminated: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 previous_alloc=ca0da6da-6856-f79a-6219-662bb1c130fb 2020-09-16T15:29:29.683Z [TRACE] client.alloc_runner.runner_hook.alloc_health_watcher: watching: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 deadline="2020-09-16 15:34:29.683722555 +0000 UTC m=+613.699535320" checks=true min_healthy_time=10s 2020-09-16T15:29:29.695Z [TRACE] client.alloc_runner: handling task state update: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 done=false 2020-09-16T15:29:29.703Z [INFO] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec &> >(tee -i -a /var/log/e/jaeger-collector-127.0.0.1-14267.log); exec /usr/bin/jaeger-all-in-one --memory.max-traces=10000]}" 2020-09-16T15:29:29.703Z [DEBUG] client.driver_mgr.raw_exec.executor: starting plugin: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c driver=raw_exec task_name=jaeger-collector path=/usr/sbin/nomad args=[/usr/sbin/nomad, executor, {"LogFile":"/var/lib/nomad/alloc/0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c/jaeger-collector/executor.out","LogLevel":"debug","FSIsolation":false}] 2020-09-16T15:29:29.707Z [DEBUG] client.driver_mgr.raw_exec.executor: using plugin: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 driver=raw_exec task_name=web-hitlist-worker version=2 2020-09-16T15:29:29.707Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin started: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c driver=raw_exec task_name=jaeger-collector path=/usr/sbin/nomad pid=4042 2020-09-16T15:29:29.707Z [DEBUG] client.driver_mgr.raw_exec.executor: waiting for RPC address: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c driver=raw_exec task_name=jaeger-collector path=/usr/sbin/nomad 2020-09-16T15:29:29.712Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 client_status=pending desired_status= 2020-09-16T15:29:29.712Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 task=dispatcher path=/usr/sbin/nomad args=[/usr/sbin/nomad, logmon] 2020-09-16T15:29:29.720Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 task=dispatcher path=/usr/sbin/nomad pid=4046 2020-09-16T15:29:29.721Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 task=dispatcher path=/usr/sbin/nomad 2020-09-16T15:29:29.739Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 task=dispatcher version=2 2020-09-16T15:29:29.739Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker state=running event=Started 2020-09-16T15:29:29.741Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 task=dispatcher address=/tmp/plugin139997309 network=unix @module=logmon timestamp=2020-09-16T15:29:29.739Z 2020-09-16T15:29:29.750Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 task=dispatcher @module=logmon path=/var/lib/nomad/alloc/b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2/alloc/logs/.dispatcher.stdout.fifo timestamp=2020-09-16T15:29:29.750Z 2020-09-16T15:29:29.750Z [DEBUG] client.driver_mgr.raw_exec.executor: using plugin: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c driver=raw_exec task_name=jaeger-collector version=2 2020-09-16T15:29:29.753Z [DEBUG] client: updated allocations: index=1996 total=56 pulled=7 filtered=49 2020-09-16T15:29:29.753Z [TRACE] client: next heartbeat: period=14.650393079s 2020-09-16T15:29:29.753Z [DEBUG] client: allocation updates: added=0 removed=0 updated=7 ignored=49 2020-09-16T15:29:29.753Z [TRACE] client: updating alloc: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 index=1952 2020-09-16T15:29:29.757Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 task=dispatcher @module=logmon path=/var/lib/nomad/alloc/b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2/alloc/logs/.dispatcher.stderr.fifo timestamp=2020-09-16T15:29:29.750Z 2020-09-16T15:29:29.769Z [TRACE] client.alloc_runner: handling task state update: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 done=false 2020-09-16T15:29:29.769Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 client_status=running desired_status= 2020-09-16T15:29:29.769Z [DEBUG] consul.health: watching check: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker check="web-hitlist-worker check" 2020-09-16T15:29:29.777Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 modify_index=1952 2020-09-16T15:29:29.777Z [TRACE] client: updating alloc: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb index=1991 2020-09-16T15:29:29.783Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb modify_index=1991 2020-09-16T15:29:29.783Z [TRACE] client: updating alloc: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe index=1960 2020-09-16T15:29:29.798Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c task=jaeger-collector state=running event=Started 2020-09-16T15:29:29.803Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe modify_index=1960 2020-09-16T15:29:29.803Z [TRACE] client: updating alloc: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b index=1944 2020-09-16T15:29:29.809Z [TRACE] client.alloc_runner: handling task state update: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c done=false 2020-09-16T15:29:29.809Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c client_status=running desired_status= 2020-09-16T15:29:29.824Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b modify_index=1944 2020-09-16T15:29:29.824Z [TRACE] client: updating alloc: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a index=1985 2020-09-16T15:29:29.828Z [DEBUG] consul.sync: sync complete: registered_services=1 deregistered_services=0 registered_checks=1 deregistered_checks=0 2020-09-16T15:29:29.838Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a modify_index=1985 2020-09-16T15:29:29.838Z [TRACE] client: updating alloc: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 index=1980 2020-09-16T15:29:29.843Z [DEBUG] client.alloc_runner: Discarding allocation update: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 skipped_modify_index=1980 new_modify_index=1980 2020-09-16T15:29:29.843Z [TRACE] client: updating alloc: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 index=1946 2020-09-16T15:29:29.855Z [INFO] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/e/natives; exec &> >(tee -i -a /var/log/e/dispatcher-::1-21233.log); exec ./IDispatcher -c /etc/conf.d/idispatcher.xml -d 1]}" 2020-09-16T15:29:29.855Z [DEBUG] client.driver_mgr.raw_exec.executor: starting plugin: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 driver=raw_exec task_name=dispatcher path=/usr/sbin/nomad args=[/usr/sbin/nomad, executor, {"LogFile":"/var/lib/nomad/alloc/b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2/dispatcher/executor.out","LogLevel":"debug","FSIsolation":false}] 2020-09-16T15:29:29.859Z [DEBUG] consul.sync: sync complete: registered_services=1 deregistered_services=0 registered_checks=1 deregistered_checks=0 2020-09-16T15:29:29.862Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin started: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 driver=raw_exec task_name=dispatcher path=/usr/sbin/nomad pid=4079 2020-09-16T15:29:29.862Z [DEBUG] client.driver_mgr.raw_exec.executor: waiting for RPC address: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 driver=raw_exec task_name=dispatcher path=/usr/sbin/nomad 2020-09-16T15:29:29.867Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 modify_index=1946 2020-09-16T15:29:29.867Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=7 ignored=49 errors=0 2020-09-16T15:29:29.918Z [DEBUG] client.driver_mgr.raw_exec.executor: using plugin: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 driver=raw_exec task_name=dispatcher version=2 2020-09-16T15:29:29.939Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 task=dispatcher state=running event=Started 2020-09-16T15:29:29.942Z [TRACE] client: next heartbeat: period=17.743000234s 2020-09-16T15:29:29.943Z [TRACE] client: next heartbeat: period=10.193051868s 2020-09-16T15:29:29.948Z [TRACE] client.alloc_runner: handling task state update: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 done=false 2020-09-16T15:29:29.948Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2 client_status=running desired_status= 2020-09-16T15:29:29.948Z [DEBUG] client: updated allocations: index=1997 total=56 pulled=7 filtered=49 2020-09-16T15:29:29.949Z [DEBUG] client: allocation updates: added=0 removed=0 updated=7 ignored=49 2020-09-16T15:29:29.949Z [TRACE] client: updating alloc: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 index=1980 2020-09-16T15:29:29.956Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 modify_index=1980 2020-09-16T15:29:29.956Z [TRACE] client: updating alloc: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a index=1985 2020-09-16T15:29:29.957Z [TRACE] client: next heartbeat: period=12.413362172s 2020-09-16T15:29:29.959Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a modify_index=1985 2020-09-16T15:29:29.959Z [TRACE] client: updating alloc: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe index=1960 2020-09-16T15:29:29.963Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe modify_index=1960 2020-09-16T15:29:29.963Z [TRACE] client: updating alloc: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 index=1952 2020-09-16T15:29:29.967Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 modify_index=1952 2020-09-16T15:29:29.967Z [TRACE] client: updating alloc: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b index=1944 2020-09-16T15:29:29.969Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b modify_index=1944 2020-09-16T15:29:29.969Z [TRACE] client: updating alloc: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 index=1946 2020-09-16T15:29:29.974Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 modify_index=1946 2020-09-16T15:29:29.974Z [TRACE] client: updating alloc: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb index=1991 2020-09-16T15:29:29.975Z [DEBUG] consul.sync: sync complete: registered_services=1 deregistered_services=0 registered_checks=1 deregistered_checks=0 2020-09-16T15:29:29.975Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/010bca37-809b-4db6-057a-223bd9e0769e/stats duration=153.514µs 2020-09-16T15:29:29.975Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/efbb3557-0148-9ddc-617b-8eb7eba053d6/stats duration=154.485µs 2020-09-16T15:29:29.976Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb modify_index=1991 2020-09-16T15:29:29.976Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=7 ignored=49 errors=0 2020-09-16T15:29:29.996Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/0027b62a-6483-5a64-d746-ceb8e8b13e86/stats duration=154.658µs 2020-09-16T15:29:30.001Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/da054000-3d76-7eb1-8b57-5e580849e3de/stats duration=263.875µs 2020-09-16T15:29:30.007Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/54c0d11a-06ea-46df-49c9-3487cd77452a/stats duration=205.313µs 2020-09-16T15:29:30.011Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/d9b7395c-9254-dd96-1f7b-011e5e305cd0/stats duration=151.159µs 2020-09-16T15:29:30.131Z [TRACE] client: next heartbeat: period=12.922619108s 2020-09-16T15:29:30.134Z [DEBUG] client: updated allocations: index=1998 total=56 pulled=7 filtered=49 2020-09-16T15:29:30.134Z [DEBUG] client: allocation updates: added=0 removed=0 updated=7 ignored=49 2020-09-16T15:29:30.134Z [TRACE] client: updating alloc: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a index=1985 2020-09-16T15:29:30.135Z [TRACE] client: next heartbeat: period=17.827454585s 2020-09-16T15:29:30.138Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a modify_index=1985 2020-09-16T15:29:30.138Z [TRACE] client: updating alloc: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b index=1944 2020-09-16T15:29:30.140Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b modify_index=1944 2020-09-16T15:29:30.140Z [TRACE] client: updating alloc: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 index=1980 2020-09-16T15:29:30.143Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 modify_index=1980 2020-09-16T15:29:30.143Z [TRACE] client: updating alloc: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe index=1960 2020-09-16T15:29:30.146Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe modify_index=1960 2020-09-16T15:29:30.147Z [TRACE] client: updating alloc: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb index=1991 2020-09-16T15:29:30.150Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb modify_index=1991 2020-09-16T15:29:30.150Z [TRACE] client: updating alloc: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 index=1952 2020-09-16T15:29:30.153Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 modify_index=1952 2020-09-16T15:29:30.153Z [TRACE] client: updating alloc: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 index=1946 2020-09-16T15:29:30.156Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 modify_index=1946 2020-09-16T15:29:30.156Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=7 ignored=49 errors=0 2020-09-16T15:29:30.275Z [DEBUG] http: request complete: method=GET path=/v1/node/9ee95205-0945-8494-15b6-74400d24799e/allocations?index=1980 duration=37.821361ms 2020-09-16T15:29:30.484Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2/stats duration=220.567µs 2020-09-16T15:29:30.486Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c/stats duration=139.7µs 2020-09-16T15:29:30.488Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/f074ee01-c03f-ff00-3037-ed4a68b216d1/stats duration=291.136µs 2020-09-16T15:29:30.856Z [DEBUG] http: request complete: method=GET path=/v1/client/stats?node_id=9ee95205-0945-8494-15b6-74400d24799e duration=155.651µs 2020-09-16T15:29:32.199Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=122.575µs 2020-09-16T15:29:32.905Z [DEBUG] http: request complete: method=GET path=/v1/client/stats?node_id=9ee95205-0945-8494-15b6-74400d24799e duration=227.944µs 2020-09-16T15:29:33.068Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/b1f82bab-7fba-b0f9-11a9-bc5a3c7f50d2/stats duration=168.86µs 2020-09-16T15:29:33.071Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c/stats duration=141.606µs 2020-09-16T15:29:33.075Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/f074ee01-c03f-ff00-3037-ed4a68b216d1/stats duration=151.237µs 2020-09-16T15:29:33.307Z [DEBUG] http: request complete: method=GET path=/v1/node/9ee95205-0945-8494-15b6-74400d24799e duration=199.157µs 2020-09-16T15:29:35.159Z [DEBUG] http: request complete: method=GET path=/v1/jobs duration=338.56µs 2020-09-16T15:29:35.609Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44210 2020-09-16T15:29:35.968Z [TRACE] client.alloc_runner: handling task state update: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 done=false 2020-09-16T15:29:35.968Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 client_status=running desired_status= 2020-09-16T15:29:35.971Z [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 driver=raw_exec task_name=web-hitlist-worker error="rpc error: code = Unavailable desc = transport is closing" 2020-09-16T15:29:35.971Z [ERROR] client.alloc_runner.task_runner.task_hook.stats_hook: failed to start stats collection for task: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker error="rpc error: code = Unavailable desc = connection error: desc = "transport: error while dialing: dial unix /tmp/plugin964886216: connect: no such file or directory"" 2020-09-16T15:29:35.974Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin process exited: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 driver=raw_exec task_name=web-hitlist-worker path=/usr/sbin/nomad pid=4033 2020-09-16T15:29:35.975Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin exited: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 driver=raw_exec task_name=web-hitlist-worker 2020-09-16T15:29:35.975Z [INFO] client.alloc_runner.task_runner: restarting task: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker reason="Restart within policy" delay=16.735502068s 2020-09-16T15:29:35.975Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker state=pending event=Restarting 2020-09-16T15:29:35.978Z [TRACE] client.alloc_runner: handling task state update: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 done=false 2020-09-16T15:29:35.978Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 client_status=pending desired_status= 2020-09-16T15:29:35.986Z [DEBUG] consul.sync: sync complete: registered_services=0 deregistered_services=1 registered_checks=0 deregistered_checks=1 2020-09-16T15:29:36.132Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1998 duration=952.30102ms 2020-09-16T15:29:36.142Z [TRACE] client: next heartbeat: period=17.172077827s 2020-09-16T15:29:36.147Z [DEBUG] client: updated allocations: index=1999 total=56 pulled=7 filtered=49 2020-09-16T15:29:36.147Z [DEBUG] client: allocation updates: added=0 removed=0 updated=7 ignored=49 2020-09-16T15:29:36.147Z [TRACE] client: updating alloc: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 index=1952 2020-09-16T15:29:36.148Z [TRACE] client: next heartbeat: period=15.019965004s 2020-09-16T15:29:36.151Z [DEBUG] http: request complete: method=GET path=/v1/node/9ee95205-0945-8494-15b6-74400d24799e/allocations?index=1998 duration=3.892538186s 2020-09-16T15:29:36.151Z [DEBUG] http: request complete: method=GET path=/v1/node/9ee95205-0945-8494-15b6-74400d24799e/allocations?index=1998 duration=2.808438611s 2020-09-16T15:29:36.151Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 modify_index=1952 2020-09-16T15:29:36.151Z [TRACE] client: updating alloc: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 index=1980 2020-09-16T15:29:36.153Z [DEBUG] client.alloc_runner: Discarding allocation update: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 skipped_modify_index=1980 new_modify_index=1980 2020-09-16T15:29:36.153Z [TRACE] client: updating alloc: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb index=1991 2020-09-16T15:29:36.155Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb modify_index=1991 2020-09-16T15:29:36.155Z [TRACE] client: updating alloc: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 index=1946 2020-09-16T15:29:36.158Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 modify_index=1946 2020-09-16T15:29:36.158Z [TRACE] client: updating alloc: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a index=1985 2020-09-16T15:29:36.162Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a modify_index=1985 2020-09-16T15:29:36.162Z [TRACE] client: updating alloc: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b index=1944 2020-09-16T15:29:36.164Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b modify_index=1944 2020-09-16T15:29:36.164Z [TRACE] client: updating alloc: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe index=1960 2020-09-16T15:29:36.167Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe modify_index=1960 2020-09-16T15:29:36.167Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=7 ignored=49 errors=0 2020-09-16T15:29:38.221Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=162.168µs 2020-09-16T15:29:41.609Z [TRACE] client.alloc_runner.runner_hook.alloc_health_watcher: health set: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c healthy=true 2020-09-16T15:29:41.612Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c client_status=running desired_status= 2020-09-16T15:29:41.732Z [TRACE] client: next heartbeat: period=17.310961579s 2020-09-16T15:29:41.733Z [DEBUG] client: updated allocations: index=2000 total=56 pulled=7 filtered=49 2020-09-16T15:29:41.734Z [DEBUG] client: allocation updates: added=0 removed=0 updated=7 ignored=49 2020-09-16T15:29:41.734Z [TRACE] client: updating alloc: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a index=1985 2020-09-16T15:29:41.734Z [TRACE] client: next heartbeat: period=10.178499675s 2020-09-16T15:29:41.735Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a modify_index=1985 2020-09-16T15:29:41.735Z [TRACE] client: updating alloc: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 index=1980 2020-09-16T15:29:41.737Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 modify_index=1980 2020-09-16T15:29:41.737Z [TRACE] client: updating alloc: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe index=1960 2020-09-16T15:29:41.739Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe modify_index=1960 2020-09-16T15:29:41.739Z [TRACE] client: updating alloc: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b index=1944 2020-09-16T15:29:41.741Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b modify_index=1944 2020-09-16T15:29:41.741Z [TRACE] client: updating alloc: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 index=1946 2020-09-16T15:29:41.742Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 modify_index=1946 2020-09-16T15:29:41.742Z [TRACE] client: updating alloc: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 index=1952 2020-09-16T15:29:41.743Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 modify_index=1952 2020-09-16T15:29:41.743Z [TRACE] client: updating alloc: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb index=1991 2020-09-16T15:29:41.745Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb modify_index=1991 2020-09-16T15:29:41.745Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=7 ignored=49 errors=0 2020-09-16T15:29:42.202Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=104.661µs 2020-09-16T15:29:45.079Z [DEBUG] http: request complete: method=GET path=/v1/job/web-hitlist-worker duration=178.567µs 2020-09-16T15:29:45.088Z [DEBUG] http: request complete: method=GET path=/v1/job/web-hitlist-worker/allocations duration=526.734µs 2020-09-16T15:29:45.092Z [DEBUG] http: request complete: method=GET path=/v1/job/web-hitlist-worker/evaluations duration=317.165µs 2020-09-16T15:29:45.434Z [DEBUG] http: request complete: method=GET path=/v1/job/web-hitlist-worker/deployment?index=1 duration=260.061µs 2020-09-16T15:29:45.439Z [DEBUG] http: request complete: method=GET path=/v1/job/web-hitlist-worker/summary?index=1 duration=209.453µs 2020-09-16T15:29:45.443Z [DEBUG] http: request complete: method=GET path=/v1/job/web-hitlist-worker/deployment duration=288.679µs 2020-09-16T15:29:45.446Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f074ee01-c03f-ff00-3037-ed4a68b216d1 duration=456.757µs 2020-09-16T15:29:45.450Z [DEBUG] http: request complete: method=GET path=/v1/allocation/efb90a4d-ba33-3595-3680-f081bd428b54 duration=350.164µs 2020-09-16T15:29:45.456Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0f029029-f7b0-77ee-c460-5999d1be1059 duration=360.438µs 2020-09-16T15:29:45.459Z [DEBUG] http: request complete: method=GET path=/v1/allocation/6395d722-eb92-6324-d32c-5d890ded629b duration=283.341µs 2020-09-16T15:29:45.463Z [DEBUG] http: request complete: method=GET path=/v1/allocation/50991da7-d351-dec7-eab9-e62917223a95 duration=276.151µs 2020-09-16T15:29:45.468Z [DEBUG] http: request complete: method=GET path=/v1/job/web-hitlist-worker/deployment duration=149.589µs 2020-09-16T15:29:45.471Z [DEBUG] http: request complete: method=GET path=/v1/job/web-hitlist-worker/versions?diffs=true duration=272.929µs 2020-09-16T15:29:45.516Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/0f029029-f7b0-77ee-c460-5999d1be1059/stats duration=144.334µs 2020-09-16T15:29:45.610Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44294 2020-09-16T15:29:48.224Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=156.151µs 2020-09-16T15:29:48.932Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/0f029029-f7b0-77ee-c460-5999d1be1059/stats duration=428.427µs 2020-09-16T15:29:51.914Z [TRACE] client: next heartbeat: period=11.689495322s 2020-09-16T15:29:52.203Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=78.866µs 2020-09-16T15:29:52.714Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker name=validate 2020-09-16T15:29:52.715Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker @module=logmon path=/var/lib/nomad/alloc/f074ee01-c03f-ff00-3037-ed4a68b216d1/alloc/logs/.web-hitlist-worker.stdout.fifo timestamp=2020-09-16T15:29:52.715Z 2020-09-16T15:29:52.716Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker @module=logmon path=/var/lib/nomad/alloc/f074ee01-c03f-ff00-3037-ed4a68b216d1/alloc/logs/.web-hitlist-worker.stderr.fifo timestamp=2020-09-16T15:29:52.715Z 2020-09-16T15:29:52.716Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker name=dispatch_payload 2020-09-16T15:29:52.716Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker name=artifacts 2020-09-16T15:29:52.716Z [TRACE] client.alloc_runner.task_runner: skipping done prestart hook: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker name=devices 2020-09-16T15:29:52.717Z [INFO] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/web/hitlist_worker; exec &> >(tee -i -a /var/log/innrm/web_hitlist_worker-::1-3400.log); exec /bin/bash ./bin/production_run_webserver.sh]}" 2020-09-16T15:29:52.717Z [DEBUG] client.driver_mgr.raw_exec.executor: starting plugin: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 driver=raw_exec task_name=web-hitlist-worker path=/usr/sbin/nomad args=[/usr/sbin/nomad, executor, {"LogFile":"/var/lib/nomad/alloc/f074ee01-c03f-ff00-3037-ed4a68b216d1/web-hitlist-worker/executor.out","LogLevel":"debug","FSIsolation":false}] 2020-09-16T15:29:52.718Z [DEBUG] client.driver_mgr.raw_exec.executor: plugin started: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 driver=raw_exec task_name=web-hitlist-worker path=/usr/sbin/nomad pid=4150 2020-09-16T15:29:52.718Z [DEBUG] client.driver_mgr.raw_exec.executor: waiting for RPC address: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 driver=raw_exec task_name=web-hitlist-worker path=/usr/sbin/nomad 2020-09-16T15:29:52.741Z [DEBUG] client.driver_mgr.raw_exec.executor: using plugin: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 driver=raw_exec task_name=web-hitlist-worker version=2 2020-09-16T15:29:52.748Z [TRACE] client.alloc_runner.task_runner: setting task state: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker state=running event=Started 2020-09-16T15:29:52.750Z [TRACE] client.alloc_runner: handling task state update: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 done=false 2020-09-16T15:29:52.750Z [TRACE] client.alloc_runner: sending updated alloc: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 client_status=running desired_status= 2020-09-16T15:29:52.750Z [DEBUG] consul.health: watching check: alloc_id=f074ee01-c03f-ff00-3037-ed4a68b216d1 task=web-hitlist-worker check="web-hitlist-worker check" 2020-09-16T15:29:52.774Z [DEBUG] consul.sync: sync complete: registered_services=1 deregistered_services=0 registered_checks=1 deregistered_checks=0 2020-09-16T15:29:52.933Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=1999 duration=15.754963704s 2020-09-16T15:29:52.933Z [DEBUG] http: request complete: method=GET path=/v1/job/web-hitlist-worker/allocations?index=2000 duration=7.801977522s 2020-09-16T15:29:52.934Z [DEBUG] http: request complete: method=GET path=/v1/job/web-hitlist-worker/summary?index=1999 duration=5.799927717s 2020-09-16T15:29:52.936Z [TRACE] client: next heartbeat: period=17.616279796s 2020-09-16T15:29:52.943Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/0f029029-f7b0-77ee-c460-5999d1be1059/stats duration=162.659µs 2020-09-16T15:29:52.952Z [TRACE] client: next heartbeat: period=16.963475213s 2020-09-16T15:29:52.953Z [DEBUG] client: updated allocations: index=2001 total=56 pulled=7 filtered=49 2020-09-16T15:29:52.954Z [DEBUG] client: allocation updates: added=0 removed=0 updated=7 ignored=49 2020-09-16T15:29:52.954Z [TRACE] client: updating alloc: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b index=1944 2020-09-16T15:29:52.955Z [TRACE] client: next heartbeat: period=10.123733118s 2020-09-16T15:29:52.959Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=b33dc2b6-4cdd-9c1a-09ca-9124ff41e52b modify_index=1944 2020-09-16T15:29:52.959Z [TRACE] client: updating alloc: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb index=1991 2020-09-16T15:29:52.963Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=ca0da6da-6856-f79a-6219-662bb1c130fb modify_index=1991 2020-09-16T15:29:52.963Z [TRACE] client: updating alloc: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 index=1946 2020-09-16T15:29:52.970Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=60c5e684-7314-5859-6ac1-2e66b4d2b943 modify_index=1946 2020-09-16T15:29:52.970Z [TRACE] client: updating alloc: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 index=1980 2020-09-16T15:29:52.972Z [DEBUG] client.alloc_runner: Discarding allocation update: alloc_id=efb90a4d-ba33-3595-3680-f081bd428b54 skipped_modify_index=1980 new_modify_index=1980 2020-09-16T15:29:52.972Z [TRACE] client: updating alloc: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe index=1960 2020-09-16T15:29:52.977Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=19368a06-1e30-432b-3b25-4f8a0609b8fe modify_index=1960 2020-09-16T15:29:52.977Z [TRACE] client: updating alloc: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 index=1952 2020-09-16T15:29:52.978Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=650c389a-686c-497c-f62b-80bd26c6a642 modify_index=1952 2020-09-16T15:29:52.978Z [TRACE] client: updating alloc: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a index=1985 2020-09-16T15:29:52.981Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=3954c06e-8fbc-c467-e2f0-c15fb0863f5a modify_index=1985 2020-09-16T15:29:52.981Z [DEBUG] client: allocation updates applied: added=0 ```

Application log

Puma starting in single mode...
* Version 3.12.0 (ruby 2.6.3-p62), codename: Llamas in Pajamas
* Min threads: 0, max threads: 16
* Environment: production
Warning: External plugins not supported in local idkit library
web_vars.yml: config/web_vars.yml
database.yml: config/database.yml
* Listening on tcp://0.0.0.0:3400
/opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/binder.rb:272:in `initialize': Address already in use - bind(2) for "0.0.0.0" port 3400 (Errno::EADDRINUSE)
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/binder.rb:272:in `new'
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/binder.rb:272:in `add_tcp_listener'
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/binder.rb:105:in `block in parse'
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/binder.rb:88:in `each'
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/binder.rb:88:in `parse'
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/runner.rb:151:in `load_and_bind'
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/single.rb:96:in `run'
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/launcher.rb:184:in `run'
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/lib/puma/cli.rb:78:in `run'
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/gems/puma-3.12.0/bin/puma:10:in `<top (required)>'
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/bin/puma:23:in `load'
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/bin/puma:23:in `<main>'
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/bin/ruby_executable_hooks:24:in `eval'
    from /opt/web/hitlist_worker/vendor/bundle/ruby/2.6.0/bin/ruby_executable_hooks:24:in `<main>'
jozef-slezak commented 4 years ago

Raw starts processes using exec like this. Not sure if it is ok for Nomad.

"Tasks": [
        {
          "Name": "dispatcher",
          "Driver": "raw_exec",
          "User": "e",
          "Config": {
            "command": "/bin/bash",
            "args": [
              "-c",
              "cd /opt/e/natives;  exec &> >(tee -i -a /var/log/e/dispatcher-${NOMAD_IP_DISPATCHER}-${NOMAD_PORT_DISPATCHER}.log); exec ./IDispatcher -c /etc//conf.d/idispatcher.xml -d 1"
            ]
          },
exec bundle exec rake production_run:delayed_job[${WEB_PORT}]
jozef-slezak commented 4 years ago

Today I was able to reproduce this behavior with jaeger-controller HCL (raw-exec). I am intentionally not using any shell script (like above).

{
  "Stop": false,
  "Region": "global",
  "Namespace": "default",
  "ID": "jaeger-collector",
  "ParentID": "",
  "Name": "jaeger-collector",
  "Type": "service",
  "Priority": 50,
  "AllAtOnce": false,
  "Datacenters": [
    "dc1"
  ],
  "Constraints": null,
  "Affinities": null,
  "Spreads": null,
  "TaskGroups": [
    {
      "Name": "jaeger-collector",
      "Count": 1,
      "Update": {
        "Stagger": 30000000000,
        "MaxParallel": 1,
        "HealthCheck": "checks",
        "MinHealthyTime": 10000000000,
        "HealthyDeadline": 300000000000,
        "ProgressDeadline": 600000000000,
        "AutoRevert": false,
        "AutoPromote": false,
        "Canary": 0
      },
      "Migrate": {
        "MaxParallel": 1,
        "HealthCheck": "checks",
        "MinHealthyTime": 10000000000,
        "HealthyDeadline": 300000000000
      },
      "Constraints": null,
      "Scaling": null,
      "RestartPolicy": {
        "Attempts": 2,
        "Interval": 1800000000000,
        "Delay": 15000000000,
        "Mode": "fail"
      },
      "Tasks": [
        {
          "Name": "jaeger-collector",
          "Driver": "raw_exec",
          "User": "jaeger",
          "Config": {
            "command": "/usr/bin/jaeger-all-in-one",
            "args": [
              "--memory.max-traces=10000"
            ]
          },
          "Env": {
            "SPAN_STORAGE_TYPE": "memory"
          },
          "Services": [
            {
              "Name": "jaeger-collector",
              "TaskName": "",
              "PortLabel": "JAEGER_COLLECTOR",
              "AddressMode": "auto",
              "EnableTagOverride": false,
              "Tags": null,
              "CanaryTags": null,
              "Checks": [
                {
                  "Name": "Jaeger Collector Check",
                  "Type": "tcp",
                  "Command": "",
                  "Args": null,
                  "Path": "",
                  "Protocol": "",
                  "PortLabel": "",
                  "Expose": false,
                  "AddressMode": "",
                  "Interval": 5000000000,
                  "Timeout": 3000000000,
                  "InitialStatus": "",
                  "TLSSkipVerify": false,
                  "Method": "",
                  "Header": null,
                  "CheckRestart": null,
                  "GRPCService": "",
                  "GRPCUseTLS": false,
                  "TaskName": "",
                  "SuccessBeforePassing": 0,
                  "FailuresBeforeCritical": 0
                }
              ],
              "Connect": null,
              "Meta": null,
              "CanaryMeta": null
            }
          ],
          "Vault": null,
          "Templates": null,
          "Constraints": null,
          "Affinities": null,
          "Resources": {
            "CPU": 100,
            "MemoryMB": 100,
            "DiskMB": 0,
            "IOPS": 0,
            "Networks": [
              {
                "Mode": "",
                "Device": "",
                "CIDR": "",
                "IP": "",
                "MBits": 10,
                "DNS": null,
                "ReservedPorts": [
                  {
                    "Label": "JAEGER_COLLECTOR",
                    "Value": 14267,
                    "To": 0,
                    "HostNetwork": "default"
                  }
                ],
                "DynamicPorts": null
              }
            ],
            "Devices": null
          },
          "RestartPolicy": {
            "Attempts": 2,
            "Interval": 1800000000000,
            "Delay": 15000000000,
            "Mode": "fail"
          },
          "DispatchPayload": null,
          "Lifecycle": null,
          "Meta": null,
          "KillTimeout": 30000000000,
          "LogConfig": {
            "MaxFiles": 10,
            "MaxFileSizeMB": 10
          },
          "Artifacts": null,
          "Leader": false,
          "ShutdownDelay": 0,
          "VolumeMounts": null,
          "KillSignal": "",
          "Kind": "",
          "CSIPluginConfig": null
        }
      ],
      "EphemeralDisk": {
        "Sticky": false,
        "SizeMB": 150,
        "Migrate": false
      },
      "Meta": null,
      "ReschedulePolicy": {
        "Attempts": 0,
        "Interval": 0,
        "Delay": 30000000000,
        "DelayFunction": "exponential",
        "MaxDelay": 3600000000000,
        "Unlimited": true
      },
      "Affinities": null,
      "Spreads": null,
      "Networks": null,
      "Services": null,
      "Volumes": null,
      "ShutdownDelay": null,
      "StopAfterClientDisconnect": null
    }
  ],
  "Update": {
    "Stagger": 30000000000,
    "MaxParallel": 1,
    "HealthCheck": "",
    "MinHealthyTime": 0,
    "HealthyDeadline": 0,
    "ProgressDeadline": 0,
    "AutoRevert": false,
    "AutoPromote": false,
    "Canary": 0
  },
  "Multiregion": null,
  "Periodic": null,
  "ParameterizedJob": null,
  "Dispatched": false,
  "Payload": null,
  "Meta": null,
  "ConsulToken": "",
  "VaultToken": "",
  "VaultNamespace": "",
  "NomadTokenID": "",
  "Status": "dead",
  "StatusDescription": "",
  "Stable": true,
  "Version": 1,
  "SubmitTime": 1600334945802876700,
  "CreateIndex": 73,
  "ModifyIndex": 3225,
  "JobModifyIndex": 2767
}

Logs:

``` od=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=295.748µs 2020-09-17T09:56:08.292Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=312.289µs 2020-09-17T09:56:08.297Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=313.111µs 2020-09-17T09:56:08.297Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54078 2020-09-17T09:56:08.304Z [DEBUG] http: request complete: method=GET path=/v1/allocation/8c8fc559-d8b1-4a8b-3c74-3f5ad784acfa duration=360.99µs 2020-09-17T09:56:08.318Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=122.365µs 2020-09-17T09:56:08.318Z [DEBUG] http: request complete: method=GET path=/v1/allocation/c5a26035-6550-1373-f49a-820a1dff7f4e duration=309.276µs 2020-09-17T09:56:08.344Z [DEBUG] http: request complete: method=GET path=/v1/allocation/595f6bcb-0e3d-f2ca-120c-73798be3237f duration=436.691µs 2020-09-17T09:56:08.348Z [DEBUG] http: request complete: method=GET path=/v1/allocation/318280e8-cb41-d137-610a-92f32470f452 duration=291.632µs 2020-09-17T09:56:08.349Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=95.41µs 2020-09-17T09:56:08.352Z [DEBUG] http: request complete: method=GET path=/v1/allocation/2afc095c-f312-f7fc-178a-45cc5b5508cf duration=300.015µs 2020-09-17T09:56:08.356Z [DEBUG] http: request complete: method=GET path=/v1/allocation/22e43108-de83-5052-1372-e3e4f75fa87b duration=270.324µs 2020-09-17T09:56:08.361Z [DEBUG] http: request complete: method=GET path=/v1/allocation/b4135fa4-3c60-44e4-1a94-8185dfa1ba3f duration=289.653µs 2020-09-17T09:56:08.366Z [DEBUG] http: request complete: method=GET path=/v1/allocation/1313c9f7-d5c3-4908-eb55-f86caa3cc37a duration=290.706µs 2020-09-17T09:56:08.371Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0577ad80-3128-acb4-184c-bf49e1645673 duration=417.446µs 2020-09-17T09:56:08.375Z [DEBUG] http: request complete: method=GET path=/v1/allocation/4701d1bc-fcd1-137d-2c68-3109e816abbd duration=300.437µs 2020-09-17T09:56:08.379Z [DEBUG] http: request complete: method=GET path=/v1/allocation/6282fe01-2535-3f04-ff33-901e813e6380 duration=298.289µs 2020-09-17T09:56:08.383Z [DEBUG] http: request complete: method=GET path=/v1/allocation/d73907ea-7ae6-5295-ca17-693f3c2c2eb9 duration=311.685µs 2020-09-17T09:56:08.387Z [DEBUG] http: request complete: method=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=313.628µs 2020-09-17T09:56:08.390Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=310.184µs 2020-09-17T09:56:08.394Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=295.659µs 2020-09-17T09:56:08.399Z [DEBUG] http: request complete: method=GET path=/v1/allocation/c5a26035-6550-1373-f49a-820a1dff7f4e duration=884.439µs 2020-09-17T09:56:08.404Z [DEBUG] http: request complete: method=GET path=/v1/allocation/595f6bcb-0e3d-f2ca-120c-73798be3237f duration=357.125µs 2020-09-17T09:56:08.409Z [DEBUG] http: request complete: method=GET path=/v1/allocation/318280e8-cb41-d137-610a-92f32470f452 duration=273.761µs 2020-09-17T09:56:08.413Z [DEBUG] http: request complete: method=GET path=/v1/allocation/2afc095c-f312-f7fc-178a-45cc5b5508cf duration=314.055µs 2020-09-17T09:56:08.419Z [DEBUG] http: request complete: method=GET path=/v1/allocation/22e43108-de83-5052-1372-e3e4f75fa87b duration=349.217µs 2020-09-17T09:56:08.423Z [DEBUG] http: request complete: method=GET path=/v1/allocation/b4135fa4-3c60-44e4-1a94-8185dfa1ba3f duration=331.564µs 2020-09-17T09:56:08.427Z [DEBUG] http: request complete: method=GET path=/v1/allocation/1313c9f7-d5c3-4908-eb55-f86caa3cc37a duration=324.093µs 2020-09-17T09:56:08.431Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0577ad80-3128-acb4-184c-bf49e1645673 duration=340.444µs 2020-09-17T09:56:08.435Z [DEBUG] http: request complete: method=GET path=/v1/allocation/4701d1bc-fcd1-137d-2c68-3109e816abbd duration=338.597µs 2020-09-17T09:56:08.441Z [DEBUG] http: request complete: method=GET path=/v1/allocation/6282fe01-2535-3f04-ff33-901e813e6380 duration=379.556µs 2020-09-17T09:56:08.446Z [DEBUG] http: request complete: method=GET path=/v1/allocation/d73907ea-7ae6-5295-ca17-693f3c2c2eb9 duration=329.901µs 2020-09-17T09:56:08.451Z [DEBUG] http: request complete: method=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=399.226µs 2020-09-17T09:56:08.455Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=314.907µs 2020-09-17T09:56:08.459Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=394.227µs 2020-09-17T09:56:08.462Z [DEBUG] http: request complete: method=GET path=/v1/allocation/595f6bcb-0e3d-f2ca-120c-73798be3237f duration=304.633µs 2020-09-17T09:56:08.466Z [DEBUG] http: request complete: method=GET path=/v1/allocation/318280e8-cb41-d137-610a-92f32470f452 duration=373.997µs 2020-09-17T09:56:08.470Z [DEBUG] http: request complete: method=GET path=/v1/allocation/2afc095c-f312-f7fc-178a-45cc5b5508cf duration=275.237µs 2020-09-17T09:56:08.476Z [DEBUG] http: request complete: method=GET path=/v1/allocation/22e43108-de83-5052-1372-e3e4f75fa87b duration=275.814µs 2020-09-17T09:56:08.480Z [DEBUG] http: request complete: method=GET path=/v1/allocation/b4135fa4-3c60-44e4-1a94-8185dfa1ba3f duration=430.253µs 2020-09-17T09:56:08.484Z [DEBUG] http: request complete: method=GET path=/v1/allocation/1313c9f7-d5c3-4908-eb55-f86caa3cc37a duration=312.969µs 2020-09-17T09:56:08.488Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0577ad80-3128-acb4-184c-bf49e1645673 duration=452.747µs 2020-09-17T09:56:08.492Z [DEBUG] http: request complete: method=GET path=/v1/allocation/4701d1bc-fcd1-137d-2c68-3109e816abbd duration=325.384µs 2020-09-17T09:56:08.498Z [DEBUG] http: request complete: method=GET path=/v1/allocation/6282fe01-2535-3f04-ff33-901e813e6380 duration=341.363µs 2020-09-17T09:56:08.502Z [DEBUG] http: request complete: method=GET path=/v1/allocation/d73907ea-7ae6-5295-ca17-693f3c2c2eb9 duration=284.543µs 2020-09-17T09:56:08.506Z [DEBUG] http: request complete: method=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=259.058µs 2020-09-17T09:56:08.510Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=298.433µs 2020-09-17T09:56:08.514Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=270.48µs 2020-09-17T09:56:08.518Z [DEBUG] http: request complete: method=GET path=/v1/allocation/318280e8-cb41-d137-610a-92f32470f452 duration=297.371µs 2020-09-17T09:56:08.521Z [DEBUG] http: request complete: method=GET path=/v1/allocation/2afc095c-f312-f7fc-178a-45cc5b5508cf duration=369.415µs 2020-09-17T09:56:08.526Z [DEBUG] http: request complete: method=GET path=/v1/allocation/22e43108-de83-5052-1372-e3e4f75fa87b duration=387.379µs 2020-09-17T09:56:08.530Z [DEBUG] http: request complete: method=GET path=/v1/allocation/b4135fa4-3c60-44e4-1a94-8185dfa1ba3f duration=302.274µs 2020-09-17T09:56:08.534Z [DEBUG] http: request complete: method=GET path=/v1/allocation/1313c9f7-d5c3-4908-eb55-f86caa3cc37a duration=327.975µs 2020-09-17T09:56:08.537Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0577ad80-3128-acb4-184c-bf49e1645673 duration=310.241µs 2020-09-17T09:56:08.542Z [DEBUG] http: request complete: method=GET path=/v1/allocation/4701d1bc-fcd1-137d-2c68-3109e816abbd duration=314.674µs 2020-09-17T09:56:08.546Z [DEBUG] http: request complete: method=GET path=/v1/allocation/6282fe01-2535-3f04-ff33-901e813e6380 duration=299.887µs 2020-09-17T09:56:08.552Z [DEBUG] http: request complete: method=GET path=/v1/allocation/d73907ea-7ae6-5295-ca17-693f3c2c2eb9 duration=349.053µs 2020-09-17T09:56:08.556Z [DEBUG] http: request complete: method=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=374.031µs 2020-09-17T09:56:08.559Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=301.628µs 2020-09-17T09:56:08.564Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=361.372µs 2020-09-17T09:56:08.570Z [DEBUG] http: request complete: method=GET path=/v1/allocation/2afc095c-f312-f7fc-178a-45cc5b5508cf duration=341.968µs 2020-09-17T09:56:08.577Z [DEBUG] http: request complete: method=GET path=/v1/allocation/22e43108-de83-5052-1372-e3e4f75fa87b duration=400.017µs 2020-09-17T09:56:08.581Z [DEBUG] http: request complete: method=GET path=/v1/allocation/b4135fa4-3c60-44e4-1a94-8185dfa1ba3f duration=298.869µs 2020-09-17T09:56:08.585Z [DEBUG] http: request complete: method=GET path=/v1/allocation/1313c9f7-d5c3-4908-eb55-f86caa3cc37a duration=412.161µs 2020-09-17T09:56:08.590Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0577ad80-3128-acb4-184c-bf49e1645673 duration=409.529µs 2020-09-17T09:56:08.594Z [DEBUG] http: request complete: method=GET path=/v1/allocation/4701d1bc-fcd1-137d-2c68-3109e816abbd duration=404.649µs 2020-09-17T09:56:08.598Z [DEBUG] http: request complete: method=GET path=/v1/allocation/6282fe01-2535-3f04-ff33-901e813e6380 duration=520.492µs 2020-09-17T09:56:08.604Z [DEBUG] http: request complete: method=GET path=/v1/allocation/d73907ea-7ae6-5295-ca17-693f3c2c2eb9 duration=468.339µs 2020-09-17T09:56:08.608Z [DEBUG] http: request complete: method=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=402.858µs 2020-09-17T09:56:08.612Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=346.195µs 2020-09-17T09:56:08.616Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=286.096µs 2020-09-17T09:56:08.639Z [DEBUG] http: request complete: method=GET path=/v1/allocation/22e43108-de83-5052-1372-e3e4f75fa87b duration=304.668µs 2020-09-17T09:56:08.645Z [DEBUG] http: request complete: method=GET path=/v1/allocation/b4135fa4-3c60-44e4-1a94-8185dfa1ba3f duration=282.739µs 2020-09-17T09:56:08.649Z [DEBUG] http: request complete: method=GET path=/v1/allocation/1313c9f7-d5c3-4908-eb55-f86caa3cc37a duration=455.34µs 2020-09-17T09:56:08.653Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0577ad80-3128-acb4-184c-bf49e1645673 duration=291.648µs 2020-09-17T09:56:08.657Z [DEBUG] http: request complete: method=GET path=/v1/allocation/4701d1bc-fcd1-137d-2c68-3109e816abbd duration=282.261µs 2020-09-17T09:56:08.661Z [DEBUG] http: request complete: method=GET path=/v1/allocation/6282fe01-2535-3f04-ff33-901e813e6380 duration=284.351µs 2020-09-17T09:56:08.666Z [DEBUG] http: request complete: method=GET path=/v1/allocation/d73907ea-7ae6-5295-ca17-693f3c2c2eb9 duration=282.207µs 2020-09-17T09:56:08.670Z [DEBUG] http: request complete: method=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=313.187µs 2020-09-17T09:56:08.674Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=284.592µs 2020-09-17T09:56:08.677Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=259.24µs 2020-09-17T09:56:08.685Z [DEBUG] http: request complete: method=GET path=/v1/allocation/b4135fa4-3c60-44e4-1a94-8185dfa1ba3f duration=415.522µs 2020-09-17T09:56:08.692Z [DEBUG] http: request complete: method=GET path=/v1/allocation/1313c9f7-d5c3-4908-eb55-f86caa3cc37a duration=296.883µs 2020-09-17T09:56:08.699Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0577ad80-3128-acb4-184c-bf49e1645673 duration=272.747µs 2020-09-17T09:56:08.705Z [DEBUG] http: request complete: method=GET path=/v1/allocation/4701d1bc-fcd1-137d-2c68-3109e816abbd duration=340.133µs 2020-09-17T09:56:08.709Z [DEBUG] http: request complete: method=GET path=/v1/allocation/6282fe01-2535-3f04-ff33-901e813e6380 duration=318.495µs 2020-09-17T09:56:08.713Z [DEBUG] http: request complete: method=GET path=/v1/allocation/d73907ea-7ae6-5295-ca17-693f3c2c2eb9 duration=260.898µs 2020-09-17T09:56:08.718Z [DEBUG] http: request complete: method=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=251.254µs 2020-09-17T09:56:08.722Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=353.54µs 2020-09-17T09:56:08.726Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=270.561µs 2020-09-17T09:56:08.743Z [DEBUG] http: request complete: method=GET path=/v1/allocation/1313c9f7-d5c3-4908-eb55-f86caa3cc37a duration=363.659µs 2020-09-17T09:56:08.748Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0577ad80-3128-acb4-184c-bf49e1645673 duration=279.528µs 2020-09-17T09:56:08.754Z [DEBUG] http: request complete: method=GET path=/v1/allocation/4701d1bc-fcd1-137d-2c68-3109e816abbd duration=351.647µs 2020-09-17T09:56:08.757Z [DEBUG] http: request complete: method=GET path=/v1/allocation/6282fe01-2535-3f04-ff33-901e813e6380 duration=296.78µs 2020-09-17T09:56:08.762Z [DEBUG] http: request complete: method=GET path=/v1/allocation/d73907ea-7ae6-5295-ca17-693f3c2c2eb9 duration=438.851µs 2020-09-17T09:56:08.766Z [DEBUG] http: request complete: method=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=279.752µs 2020-09-17T09:56:08.771Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=297.424µs 2020-09-17T09:56:08.776Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=358.276µs 2020-09-17T09:56:08.808Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0577ad80-3128-acb4-184c-bf49e1645673 duration=360.819µs 2020-09-17T09:56:08.813Z [DEBUG] http: request complete: method=GET path=/v1/allocation/4701d1bc-fcd1-137d-2c68-3109e816abbd duration=312.879µs 2020-09-17T09:56:08.817Z [DEBUG] http: request complete: method=GET path=/v1/allocation/6282fe01-2535-3f04-ff33-901e813e6380 duration=338.805µs 2020-09-17T09:56:08.821Z [DEBUG] http: request complete: method=GET path=/v1/allocation/d73907ea-7ae6-5295-ca17-693f3c2c2eb9 duration=363.817µs 2020-09-17T09:56:08.825Z [DEBUG] http: request complete: method=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=324.523µs 2020-09-17T09:56:08.830Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=297.577µs 2020-09-17T09:56:08.835Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=349.67µs 2020-09-17T09:56:08.842Z [DEBUG] http: request complete: method=GET path=/v1/allocation/4701d1bc-fcd1-137d-2c68-3109e816abbd duration=305.396µs 2020-09-17T09:56:08.847Z [DEBUG] http: request complete: method=GET path=/v1/allocation/6282fe01-2535-3f04-ff33-901e813e6380 duration=321.112µs 2020-09-17T09:56:08.851Z [DEBUG] http: request complete: method=GET path=/v1/allocation/d73907ea-7ae6-5295-ca17-693f3c2c2eb9 duration=362.421µs 2020-09-17T09:56:08.855Z [DEBUG] http: request complete: method=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=317.445µs 2020-09-17T09:56:08.859Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=307.68µs 2020-09-17T09:56:08.863Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=479.769µs 2020-09-17T09:56:08.924Z [DEBUG] http: request complete: method=GET path=/v1/allocation/6282fe01-2535-3f04-ff33-901e813e6380 duration=265.792µs 2020-09-17T09:56:08.928Z [DEBUG] http: request complete: method=GET path=/v1/allocation/d73907ea-7ae6-5295-ca17-693f3c2c2eb9 duration=304.51µs 2020-09-17T09:56:08.932Z [DEBUG] http: request complete: method=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=493.635µs 2020-09-17T09:56:08.935Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=322.259µs 2020-09-17T09:56:08.939Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=292.253µs 2020-09-17T09:56:08.952Z [DEBUG] http: request complete: method=GET path=/v1/allocation/d73907ea-7ae6-5295-ca17-693f3c2c2eb9 duration=353.684µs 2020-09-17T09:56:08.956Z [DEBUG] http: request complete: method=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=311.09µs 2020-09-17T09:56:08.959Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=266.856µs 2020-09-17T09:56:08.963Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=370.557µs 2020-09-17T09:56:08.972Z [DEBUG] http: request complete: method=GET path=/v1/allocation/cabc6570-da2e-a842-d6de-e84874e5c6a4 duration=352.792µs 2020-09-17T09:56:08.976Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=351.181µs 2020-09-17T09:56:08.982Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=254.392µs 2020-09-17T09:56:09.003Z [DEBUG] http: request complete: method=GET path=/v1/allocation/f7ae483c-f802-9b10-31bd-7442f96412a4 duration=1.002485ms 2020-09-17T09:56:09.009Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=340.077µs 2020-09-17T09:56:09.031Z [DEBUG] http: request complete: method=GET path=/v1/allocation/88d9cf3f-1470-cdbf-354e-c7245363188e duration=380.618µs 2020-09-17T09:56:10.748Z [DEBUG] http: request complete: method=GET path=/v1/allocation/2f6f16a1-9a2d-92a2-5489-4277f21ca794 duration=460.221µs 2020-09-17T09:56:11.536Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=169.695µs 2020-09-17T09:56:11.539Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=184.823µs 2020-09-17T09:56:12.120Z [TRACE] client: next heartbeat: period=11.270698448s 2020-09-17T09:56:13.576Z [DEBUG] http: request failed: method=GET path=/v1/client/fs/logs/d49da96c-d730-6ad5-a0ce-35b80c2a9397?follow=true&offset=50000&origin=end&task=matcher-numa0&type=stdout error="task "matcher-numa0" not started yet. No logs available" code=404 2020-09-17T09:56:13.576Z [DEBUG] http: request complete: method=GET path=/v1/client/fs/logs/d49da96c-d730-6ad5-a0ce-35b80c2a9397?follow=true&offset=50000&origin=end&task=matcher-numa0&type=stdout duration=373.696µs 2020-09-17T09:56:15.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=206.002µs 2020-09-17T09:56:15.537Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=273.213µs 2020-09-17T09:56:15.649Z [DEBUG] http: request failed: method=GET path=/v1/client/fs/logs/d49da96c-d730-6ad5-a0ce-35b80c2a9397?follow=true&offset=50000&origin=end&task=matcher-numa0&type=stderr error="task "matcher-numa0" not started yet. No logs available" code=404 2020-09-17T09:56:15.649Z [DEBUG] http: request complete: method=GET path=/v1/client/fs/logs/d49da96c-d730-6ad5-a0ce-35b80c2a9397?follow=true&offset=50000&origin=end&task=matcher-numa0&type=stderr duration=527.416µs 2020-09-17T09:56:16.410Z [DEBUG] http: request failed: method=GET path=/v1/client/fs/logs/d49da96c-d730-6ad5-a0ce-35b80c2a9397?follow=true&offset=50000&origin=end&task=matcher-numa0&type=stdout error="task "matcher-numa0" not started yet. No logs available" code=404 2020-09-17T09:56:16.410Z [DEBUG] http: request complete: method=GET path=/v1/client/fs/logs/d49da96c-d730-6ad5-a0ce-35b80c2a9397?follow=true&offset=50000&origin=end&task=matcher-numa0&type=stdout duration=1.338248ms 2020-09-17T09:56:17.572Z [DEBUG] http: request failed: method=GET path=/v1/client/fs/logs/d49da96c-d730-6ad5-a0ce-35b80c2a9397?follow=true&offset=50000&origin=end&task=matcher-numa0&type=stderr error="task "matcher-numa0" not started yet. No logs available" code=404 2020-09-17T09:56:17.572Z [DEBUG] http: request complete: method=GET path=/v1/client/fs/logs/d49da96c-d730-6ad5-a0ce-35b80c2a9397?follow=true&offset=50000&origin=end&task=matcher-numa0&type=stderr duration=417.487µs 2020-09-17T09:56:18.280Z [DEBUG] http: request failed: method=GET path=/v1/client/fs/logs/d49da96c-d730-6ad5-a0ce-35b80c2a9397?follow=true&offset=50000&origin=end&task=matcher-numa0&type=stdout error="task "matcher-numa0" not started yet. No logs available" code=404 2020-09-17T09:56:18.280Z [DEBUG] http: request complete: method=GET path=/v1/client/fs/logs/d49da96c-d730-6ad5-a0ce-35b80c2a9397?follow=true&offset=50000&origin=end&task=matcher-numa0&type=stdout duration=370.323µs 2020-09-17T09:56:18.299Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54130 2020-09-17T09:56:18.319Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=145.624µs 2020-09-17T09:56:18.351Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=115.486µs 2020-09-17T09:56:19.531Z [DEBUG] http: request complete: method=GET path=/v1/jobs duration=552.165µs 2020-09-17T09:56:19.538Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=163.603µs 2020-09-17T09:56:19.540Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=185.432µs 2020-09-17T09:56:19.576Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0 duration=286.532µs 2020-09-17T09:56:23.392Z [TRACE] client: next heartbeat: period=13.097219958s 2020-09-17T09:56:28.300Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54180 2020-09-17T09:56:28.321Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=106.396µs 2020-09-17T09:56:28.352Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=110.322µs 2020-09-17T09:56:36.490Z [TRACE] client: next heartbeat: period=12.500687125s 2020-09-17T09:56:38.300Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54228 2020-09-17T09:56:38.322Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=115.361µs 2020-09-17T09:56:38.355Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=147.638µs 2020-09-17T09:56:39.106Z [DEBUG] http: request complete: method=GET path=/v1/jobs duration=183.395µs 2020-09-17T09:56:48.302Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54282 2020-09-17T09:56:48.325Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=118.852µs 2020-09-17T09:56:48.360Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=117.402µs 2020-09-17T09:56:48.992Z [TRACE] client: next heartbeat: period=11.675004156s 2020-09-17T09:56:58.302Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54332 2020-09-17T09:56:58.328Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=173.313µs 2020-09-17T09:56:58.362Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=113.034µs 2020-09-17T09:57:00.668Z [TRACE] client: next heartbeat: period=12.653407555s 2020-09-17T09:57:08.304Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54384 2020-09-17T09:57:08.330Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=154.965µs 2020-09-17T09:57:08.364Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=164.199µs 2020-09-17T09:57:08.386Z [DEBUG] worker: dequeued evaluation: eval_id=5e348c28-6684-2bb2-f0d3-ea2c836372e8 2020-09-17T09:57:08.386Z [DEBUG] core.sched: eval GC scanning before cutoff index: index=0 eval_gc_threshold=1h0m0s 2020-09-17T09:57:08.386Z [DEBUG] worker: ack evaluation: eval_id=5e348c28-6684-2bb2-f0d3-ea2c836372e8 2020-09-17T09:57:08.386Z [DEBUG] worker: dequeued evaluation: eval_id=96ab1e94-e38e-ee88-2de9-9c15dd114b7f 2020-09-17T09:57:08.386Z [TRACE] core.sched: garbage collecting unclaimed CSI volume claims: eval.JobID=csi-volume-claim-gc 2020-09-17T09:57:08.386Z [DEBUG] core.sched: CSI volume claim GC scanning before cutoff index: index=4 csi_volume_claim_gc_threshold=5m0s 2020-09-17T09:57:08.386Z [DEBUG] worker: ack evaluation: eval_id=96ab1e94-e38e-ee88-2de9-9c15dd114b7f 2020-09-17T09:57:08.386Z [DEBUG] worker: dequeued evaluation: eval_id=4493a789-2e37-8ea0-d9ed-eff8a7215608 2020-09-17T09:57:08.386Z [DEBUG] core.sched: node GC scanning before cutoff index: index=0 node_gc_threshold=24h0m0s 2020-09-17T09:57:08.386Z [DEBUG] worker: ack evaluation: eval_id=4493a789-2e37-8ea0-d9ed-eff8a7215608 2020-09-17T09:57:08.386Z [DEBUG] worker: dequeued evaluation: eval_id=0353a3fb-61d8-8e46-a0a5-f1d3df6d7d94 2020-09-17T09:57:08.386Z [DEBUG] core.sched: deployment GC scanning before cutoff index: index=0 deployment_gc_threshold=1h0m0s 2020-09-17T09:57:08.386Z [DEBUG] worker: ack evaluation: eval_id=0353a3fb-61d8-8e46-a0a5-f1d3df6d7d94 2020-09-17T09:57:08.386Z [DEBUG] worker: dequeued evaluation: eval_id=cca49dac-41cd-e305-1e66-cf3d4bb38e22 2020-09-17T09:57:08.386Z [DEBUG] core.sched: CSI plugin GC scanning before cutoff index: index=0 csi_plugin_gc_threshold=1h0m0s 2020-09-17T09:57:08.386Z [DEBUG] worker: ack evaluation: eval_id=cca49dac-41cd-e305-1e66-cf3d4bb38e22 2020-09-17T09:57:08.386Z [DEBUG] worker: dequeued evaluation: eval_id=d4f00cee-5c87-37d6-cf50-09d4ddf02659 2020-09-17T09:57:08.386Z [DEBUG] core.sched: job GC scanning before cutoff index: index=0 job_gc_threshold=4h0m0s 2020-09-17T09:57:08.386Z [DEBUG] worker: ack evaluation: eval_id=d4f00cee-5c87-37d6-cf50-09d4ddf02659 2020-09-17T09:57:08.750Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=2.637802ms 2020-09-17T09:57:08.764Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=185.901µs 2020-09-17T09:57:12.535Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=332.233µs 2020-09-17T09:57:12.541Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=268.442µs 2020-09-17T09:57:13.323Z [TRACE] client: next heartbeat: period=18.494462001s 2020-09-17T09:57:16.535Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=255.734µs 2020-09-17T09:57:16.535Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=158.101µs 2020-09-17T09:57:18.305Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54434 2020-09-17T09:57:18.332Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=104.997µs 2020-09-17T09:57:18.365Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=204.846µs 2020-09-17T09:57:20.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=168.911µs 2020-09-17T09:57:20.536Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=167.003µs 2020-09-17T09:57:24.533Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=198.384µs 2020-09-17T09:57:24.537Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=229.671µs 2020-09-17T09:57:28.306Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54484 2020-09-17T09:57:28.334Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=341.999µs 2020-09-17T09:57:28.366Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=109.221µs 2020-09-17T09:57:28.536Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=264.976µs 2020-09-17T09:57:28.537Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=272.831µs 2020-09-17T09:57:28.732Z [DEBUG] http: request complete: method=GET path=/v1/job/pre-extractor/deployment?index=3085 duration=5m10.070533455s 2020-09-17T09:57:31.819Z [TRACE] client: next heartbeat: period=14.321829704s 2020-09-17T09:57:32.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=207.091µs 2020-09-17T09:57:32.536Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=202.48µs 2020-09-17T09:57:34.306Z [DEBUG] http: request complete: method=GET path=/v1/regions duration=87.447µs 2020-09-17T09:57:34.318Z [DEBUG] http: request failed: method=GET path=/v1/acl/token/self error="RPC Error:: 400,ACL support disabled" code=400 2020-09-17T09:57:34.319Z [DEBUG] http: request complete: method=GET path=/v1/acl/token/self duration=409.588µs 2020-09-17T09:57:34.321Z [DEBUG] http: request complete: method=GET path=/v1/agent/members duration=148.307µs 2020-09-17T09:57:34.360Z [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501 2020-09-17T09:57:34.360Z [DEBUG] http: request complete: method=GET path=/v1/namespaces duration=1.655907ms 2020-09-17T09:57:34.503Z [DEBUG] http: request complete: method=GET path=/v1/jobs duration=250.447µs 2020-09-17T09:57:36.536Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=357.909µs 2020-09-17T09:57:36.536Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=272.481µs 2020-09-17T09:57:38.307Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54534 2020-09-17T09:57:38.336Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=175.293µs 2020-09-17T09:57:38.367Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=182.997µs 2020-09-17T09:57:40.535Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=194.074µs 2020-09-17T09:57:40.537Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=162.577µs 2020-09-17T09:57:44.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=209.524µs 2020-09-17T09:57:44.537Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=253.929µs 2020-09-17T09:57:46.141Z [TRACE] client: next heartbeat: period=11.060413007s 2020-09-17T09:57:48.307Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54586 2020-09-17T09:57:48.337Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=123.669µs 2020-09-17T09:57:48.370Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=142.401µs 2020-09-17T09:57:48.535Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=217.193µs 2020-09-17T09:57:48.540Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=476.443µs 2020-09-17T09:57:52.536Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=168.194µs 2020-09-17T09:57:52.540Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=159.909µs 2020-09-17T09:57:56.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=326.328µs 2020-09-17T09:57:56.538Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=362.504µs 2020-09-17T09:57:57.202Z [TRACE] client: next heartbeat: period=15.42130762s 2020-09-17T09:57:58.308Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54634 2020-09-17T09:57:58.339Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=86.37µs 2020-09-17T09:57:58.374Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=144.025µs 2020-09-17T09:57:58.778Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/evaluations?index=3272 duration=5m0.304589912s 2020-09-17T09:58:00.537Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=164.385µs 2020-09-17T09:58:00.539Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=171.848µs 2020-09-17T09:58:00.893Z [DEBUG] http: request complete: method=GET path=/v1/job/jaeger-collector duration=240.39µs 2020-09-17T09:58:00.935Z [DEBUG] http: request complete: method=GET path=/v1/job/jaeger-collector/allocations duration=562.752µs 2020-09-17T09:58:00.942Z [DEBUG] http: request complete: method=GET path=/v1/job/jaeger-collector/evaluations duration=4.638467ms 2020-09-17T09:58:01.409Z [DEBUG] http: request complete: method=GET path=/v1/job/jaeger-collector/deployment?index=1 duration=612.692µs 2020-09-17T09:58:01.418Z [DEBUG] http: request complete: method=GET path=/v1/job/jaeger-collector/summary?index=1 duration=198.109µs 2020-09-17T09:58:01.420Z [DEBUG] http: request complete: method=GET path=/v1/job/jaeger-collector/deployment duration=234.318µs 2020-09-17T09:58:01.426Z [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501 2020-09-17T09:58:01.426Z [DEBUG] http: request complete: method=GET path=/v1/namespace/default duration=330.781µs 2020-09-17T09:58:01.433Z [DEBUG] http: request complete: method=GET path=/v1/node/9ee95205-0945-8494-15b6-74400d24799e duration=389.745µs 2020-09-17T09:58:01.435Z [DEBUG] http: request complete: method=GET path=/v1/allocation/d68f2d4d-95bf-0d05-d567-e453220b01d0 duration=385.084µs 2020-09-17T09:58:01.442Z [DEBUG] http: request complete: method=GET path=/v1/allocation/0d3c90d5-b81d-a6f7-0716-0fd1155e0e0c duration=366.393µs 2020-09-17T09:58:01.445Z [DEBUG] http: request complete: method=GET path=/v1/allocation/59e4d686-4fec-e210-14bc-c8a8eed20cd2 duration=369.988µs 2020-09-17T09:58:01.449Z [DEBUG] http: request complete: method=GET path=/v1/allocation/98e0b801-c484-68df-1dc6-bff0855ceb13 duration=403.791µs 2020-09-17T09:58:01.451Z [DEBUG] http: request complete: method=GET path=/v1/allocation/3954c06e-8fbc-c467-e2f0-c15fb0863f5a duration=332.405µs 2020-09-17T09:58:01.456Z [DEBUG] http: request complete: method=GET path=/v1/job/jaeger-collector/deployment duration=160.835µs 2020-09-17T09:58:01.461Z [DEBUG] http: request complete: method=GET path=/v1/job/jaeger-collector/versions?diffs=true duration=621.718µs 2020-09-17T09:58:02.596Z [DEBUG] http: request complete: method=GET path=/v1/job/jaeger-collector duration=236.295µs 2020-09-17T09:58:03.738Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0?index=3205 duration=5m5.168627332s 2020-09-17T09:58:04.535Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=161.473µs 2020-09-17T09:58:04.536Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=152.65µs 2020-09-17T09:58:05.830Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/summary?index=3289 duration=5m3.28895152s 2020-09-17T09:58:07.555Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=159.398µs 2020-09-17T09:58:07.556Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=158.083µs 2020-09-17T09:58:08.310Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54686 2020-09-17T09:58:08.340Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=112.787µs 2020-09-17T09:58:08.376Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=166.79µs 2020-09-17T09:58:11.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=151.211µs 2020-09-17T09:58:11.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=132.049µs 2020-09-17T09:58:12.625Z [TRACE] client: next heartbeat: period=11.661854141s 2020-09-17T09:58:13.473Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/allocations?index=3299 duration=5m8.972840668s 2020-09-17T09:58:14.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=164.278µs 2020-09-17T09:58:15.533Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=134.661µs 2020-09-17T09:58:17.306Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/deployment?index=3263 duration=5m14.889097032s 2020-09-17T09:58:18.312Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54738 2020-09-17T09:58:18.341Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=117.828µs 2020-09-17T09:58:18.384Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=209.702µs 2020-09-17T09:58:18.535Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=206.828µs 2020-09-17T09:58:18.984Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/allocations?index=3301 duration=5m2.704957401s 2020-09-17T09:58:19.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=205.073µs 2020-09-17T09:58:20.262Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/deployment?index=3263 duration=5m5.029261384s 2020-09-17T09:58:21.482Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/evaluations?index=3272 duration=5m6.619173328s 2020-09-17T09:58:22.199Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0?index=3205 duration=5m7.330966991s 2020-09-17T09:58:22.533Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=169.875µs 2020-09-17T09:58:23.537Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=265.272µs 2020-09-17T09:58:24.288Z [TRACE] client: next heartbeat: period=15.590831794s 2020-09-17T09:58:26.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=227.676µs 2020-09-17T09:58:27.549Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=186.073µs 2020-09-17T09:58:28.313Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54790 2020-09-17T09:58:28.345Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=138.896µs 2020-09-17T09:58:28.386Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=116.303µs 2020-09-17T09:58:30.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=136.231µs 2020-09-17T09:58:31.535Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=170.977µs 2020-09-17T09:58:33.728Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/summary?index=3289 duration=5m18.495385524s 2020-09-17T09:58:34.533Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=286.995µs 2020-09-17T09:58:35.533Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=148.93µs 2020-09-17T09:58:38.314Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54840 2020-09-17T09:58:38.346Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=105.422µs 2020-09-17T09:58:38.389Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=109.104µs 2020-09-17T09:58:38.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=185.896µs 2020-09-17T09:58:39.533Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=148.489µs 2020-09-17T09:58:39.881Z [TRACE] client: next heartbeat: period=12.156120246s 2020-09-17T09:58:41.157Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/deployment?index=3263 duration=5m0.204839331s 2020-09-17T09:58:42.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=148.592µs 2020-09-17T09:58:43.024Z [TRACE] client: next heartbeat: period=16.8995296s 2020-09-17T09:58:43.027Z [DEBUG] client: updated allocations: index=3307 total=121 pulled=8 filtered=113 2020-09-17T09:58:43.028Z [DEBUG] client: allocation updates: added=0 removed=0 updated=8 ignored=113 2020-09-17T09:58:43.028Z [TRACE] client: updating alloc: alloc_id=4cd5bda2-8452-64e6-9ec2-83b25a0da06b index=3196 2020-09-17T09:58:43.028Z [TRACE] client: next heartbeat: period=10.696441425s 2020-09-17T09:58:43.031Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=4cd5bda2-8452-64e6-9ec2-83b25a0da06b modify_index=3196 2020-09-17T09:58:43.031Z [TRACE] client: updating alloc: alloc_id=0fc95388-68f7-2fdf-7ce9-8004d7753ad4 index=2823 2020-09-17T09:58:43.033Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=0fc95388-68f7-2fdf-7ce9-8004d7753ad4 modify_index=2823 2020-09-17T09:58:43.033Z [TRACE] client: updating alloc: alloc_id=3d8caa25-2284-b005-0a3a-7a1d3a48c6d4 index=3191 2020-09-17T09:58:43.035Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=3d8caa25-2284-b005-0a3a-7a1d3a48c6d4 modify_index=3191 2020-09-17T09:58:43.035Z [TRACE] client: updating alloc: alloc_id=809c4c3d-dc86-c172-1f6f-a2db7672a901 index=2825 2020-09-17T09:58:43.037Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=809c4c3d-dc86-c172-1f6f-a2db7672a901 modify_index=2825 2020-09-17T09:58:43.037Z [TRACE] client: updating alloc: alloc_id=d9b7395c-9254-dd96-1f7b-011e5e305cd0 index=2305 2020-09-17T09:58:43.039Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=d9b7395c-9254-dd96-1f7b-011e5e305cd0 modify_index=2305 2020-09-17T09:58:43.039Z [TRACE] client: updating alloc: alloc_id=5c36fee7-3cee-faa2-9b40-9f53fa4b057d index=2307 2020-09-17T09:58:43.040Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=5c36fee7-3cee-faa2-9b40-9f53fa4b057d modify_index=2307 2020-09-17T09:58:43.040Z [TRACE] client: updating alloc: alloc_id=35776b49-1c42-0938-fe44-c9b4b9e42a68 index=3203 2020-09-17T09:58:43.043Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=35776b49-1c42-0938-fe44-c9b4b9e42a68 modify_index=3203 2020-09-17T09:58:43.043Z [TRACE] client: updating alloc: alloc_id=efbb3557-0148-9ddc-617b-8eb7eba053d6 index=2835 2020-09-17T09:58:43.044Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=efbb3557-0148-9ddc-617b-8eb7eba053d6 modify_index=2835 2020-09-17T09:58:43.044Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=8 ignored=113 errors=0 2020-09-17T09:58:43.384Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/summary?index=3289 duration=5m2.432883173s 2020-09-17T09:58:43.541Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=152.218µs 2020-09-17T09:58:45.794Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/allocations?index=3307 duration=5m3.152824919s 2020-09-17T09:58:46.237Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0/evaluations?index=3272 duration=5m5.284940175s 2020-09-17T09:58:46.533Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=213.985µs 2020-09-17T09:58:47.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=264.85µs 2020-09-17T09:58:48.315Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54888 2020-09-17T09:58:48.348Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=98.821µs 2020-09-17T09:58:48.391Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=181.016µs 2020-09-17T09:58:49.536Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=240.05µs 2020-09-17T09:58:51.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=241.114µs 2020-09-17T09:58:52.599Z [DEBUG] http: request complete: method=GET path=/v1/job/matcher0?index=3205 duration=5m11.651025768s 2020-09-17T09:58:53.533Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=176.544µs 2020-09-17T09:58:53.725Z [TRACE] client: next heartbeat: period=14.815367114s 2020-09-17T09:58:55.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=191.645µs 2020-09-17T09:58:56.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=186.678µs 2020-09-17T09:58:58.317Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54938 2020-09-17T09:58:58.353Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=106.325µs 2020-09-17T09:58:58.393Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=116.056µs 2020-09-17T09:58:59.533Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=210.482µs 2020-09-17T09:59:00.533Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=189.54µs 2020-09-17T09:59:02.218Z [DEBUG] http: request complete: method=GET path=/v1/jobs?index=3299 duration=5m8.748644063s 2020-09-17T09:59:03.535Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=188.112µs 2020-09-17T09:59:03.846Z [DEBUG] http: request complete: method=GET path=/v1/job/jaeger-collector duration=171.693µs 2020-09-17T09:59:08.319Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54992 2020-09-17T09:59:08.354Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=213.739µs 2020-09-17T09:59:08.396Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=234.989µs 2020-09-17T09:59:08.541Z [TRACE] client: next heartbeat: period=19.64691651s 2020-09-17T09:59:13.990Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=204.298µs 2020-09-17T09:59:13.990Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=219.092µs 2020-09-17T09:59:17.535Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=232.49µs 2020-09-17T09:59:17.538Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=238.359µs 2020-09-17T09:59:18.319Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:55040 2020-09-17T09:59:18.356Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=126.065µs 2020-09-17T09:59:18.398Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=100.687µs 2020-09-17T09:59:21.534Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/4c6467a4-1a1d-2ecb-2de1-8a9e5d8b9998/stats duration=155.424µs 2020-09-17T09:59:21.535Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/36569f70-bf83-5d87-60b5-6dbc8b88a66b/stats duration=381.04µs 2020-09-17T09:59:21.773Z [DEBUG] http: request complete: method=GET path=/v1/job/pre-extractor?index=3235 duration=5m4.674600955s ```

After few minutes I needed to force evaluation:

curl -XPOST http://localhost:4646/v1/node/d130775b-36ae-57e4-000c-7c5761ed589e/evaluate
node not found[demo@demo-inno-77 ~]$  curl -XPOST http://localhost:4646/v1/node/9ee95205-0945-8494-15b6-74400d24799e/evaluate
{"HeartbeatTTL":0,"EvalIDs":["d927e570-94ef-ffeb-68f0-593ee70df3d1","69c7b976-febf-0ea5-1d80-55743579b1df","80673e08-ef75-d300-157d-6fb67dd7fa2c","c2678871-ebd6-d9d6-2612-ee742b92ec42","d97aa742-c2c1-b41e-b83d-81b3eaa78488","657a3073-ea0e-2bfa-317c-6a6555e3b599","759ab869-8cdc-7625-823e-8cd044df5bef","8f021acc-80ee-b362-c841-9a0027823ae8","481771d2-3f5e-46da-dbd6-4ad87ed04e8c","9c864f9c-5a81-44f3-6af4-231bb84df7a3","378d915b-1b64-2947-125f-4ce419a3f79a","f6d8209a-83c1-98bd-7b82-abc53f23f337","b7f52c37-87ba-cf70-5ba4-0ae847d92ed3","4e3a909c-fe90-2d4f-97be-d9a242c8c4ac","573365a8-d601-91cf-80cc-5221accf49c6"],"EvalCreateIndex":3316,"NodeModifyIndex":0,"Features":0,"LeaderRPCAddr":"127.0.0.1:4647","NumNodes":1,"Servers":[{"RPCAdvertiseAddr":"127.0.0.1:4647","RPCMajorVersion":1,"RPCMinorVersion":1,"Datacenter":"dc1"}],"Index":3316,"LastContact":0,"KnownLeader":false}

I am going to reproduce it using exec driver (instead raw_exec).

jozef-slezak commented 4 years ago

Could high CPU load at startup cause problem to Nomad?

tgross commented 4 years ago

@jozef-slezak I've wrapped your logs in a <details> block again for legibility. You've been trying to change a bunch of different things with the task drivers, but I want to narrow down the behavior rather than changing it.

The behavior you should be seeing on a poweroff is as follows:

The behavior it seems you're seeing is all the same except that when Nomad reschedules the task, the task can't run because somehow the previous task still somehow has a resource locked up after the poweroff. This shouldn't be possible unless the process has on-disk artifacts like a PID file.

So what we need to do here is:

tgross commented 4 years ago

Here's an example of what we'd expect to see in this scenario:

$ nomad job run ./example.nomad
==> Monitoring evaluation "b2a9ee29"
    Evaluation triggered by job "example"
    Evaluation within deployment: "e26b15de"
    Allocation "5d7bd025" created: node "97386544", group "cache"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "b2a9ee29" finished with status "complete"

$ nomad alloc status 5d7
...
Recent Events:
Time                       Type        Description
2020-09-17T09:54:31-04:00  Started     Task started by client
2020-09-17T09:54:31-04:00  Task Setup  Building Task Directory
2020-09-17T09:54:31-04:00  Received    Task received by client

Then the host is powered off (not gracefully shut down), and restarted:

$ nomad alloc status 5d7
...
Recent Events:
Time                       Type        Description
2020-09-17T09:55:46-04:00  Started     Task started by client
2020-09-17T09:55:29-04:00  Restarting  Task restarting in 16.680143874s
2020-09-17T09:55:29-04:00  Terminated  Exit Code: 255, Exit Message: "Docker container exited with non-zero exit code: 255"
2020-09-17T09:54:31-04:00  Started     Task started by client
2020-09-17T09:54:31-04:00  Task Setup  Building Task Directory
2020-09-17T09:54:31-04:00  Received    Task received by client
logs starting from reboot ``` Started Nomad. ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation. ==> Loaded configuration from /etc/nomad.d/standalone.hcl ==> Starting Nomad agent... ==> Nomad agent configuration: Advertise Addrs: HTTP: 10.0.2.15:4646; RPC: 10.0.2.15:4647; Serf: 10.0.2.15:4648 Bind Addrs: HTTP: 0.0.0.0:4646; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648 Client: true Log Level: DEBUG Region: global (DC: dc1) Server: true Version: 0.12.4-dev ==> Nomad agent started! Log data will stream in below: ... (skipping all the plugin loading and fingerprinting bits) ... 2020-09-17T13:55:29.454Z [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=driver 2020-09-17T13:55:29.459Z [WARN] client.driver_mgr.docker: failed to reattach to docker logger process: driver=docker error="failed to reattach to docker logger process: Reattachment process not found" 2020-09-17T13:55:29.459Z [DEBUG] client.driver_mgr.docker.docker_logger: starting plugin: driver=docker path=/opt/gopath/bin/nomad args=[/opt/gopath/bin/nomad, docker_logger] 2020-09-17T13:55:29.474Z [DEBUG] client: updated allocations: index=15 total=1 pulled=0 filtered=1 2020-09-17T13:55:29.459Z [DEBUG] client.driver_mgr.docker.docker_logger: plugin started: driver=docker path=/opt/gopath/bin/nomad pid=1746 2020-09-17T13:55:29.459Z [DEBUG] client.driver_mgr.docker.docker_logger: waiting for RPC address: driver=docker path=/opt/gopath/bin/nomad 2020-09-17T13:55:29.459Z [INFO] client: node registration complete 2020-09-17T13:55:29.465Z [DEBUG] client.driver_mgr.docker.docker_logger: using plugin: driver=docker version=2 2020-09-17T13:55:29.465Z [DEBUG] client.driver_mgr.docker.docker_logger.nomad: plugin address: driver=docker @module=docker_logger address=/tmp/plugin947237445 network=unix timestamp=2020-09-17T13:55:29.464Z 2020-09-17T13:55:29.468Z [DEBUG] client.driver_mgr.docker.docker_logger.nomad: using client connection initialized from environment: driver=docker @module=docker_logger timestamp=2020-09-17T13:55:29.468Z 2020-09-17T13:55:29.468Z [INFO] client: started client: node_id=97386544-dd75-cc6b-0a55-b0065c1a41a3 2020-09-17T13:55:29.469Z [WARN] http: enable_debug is set to true. This is insecure and should not be enabled in production 2020-09-17T13:55:29.475Z [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1 2020-09-17T13:55:29.475Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0 2020-09-17T13:55:29.479Z [WARN] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=5d7bd025-9ce1-2b59-9cbc-9f6939518ff2 task=redis error="Reattachment process not found" 2020-09-17T13:55:29.479Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=5d7bd025-9ce1-2b59-9cbc-9f6939518ff2 task=redis path=/opt/gopath/bin/nomad args=[/opt/gopath/bin/nomad, logmon] 2020-09-17T13:55:29.480Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=5d7bd025-9ce1-2b59-9cbc-9f6939518ff2 task=redis path=/opt/gopath/bin/nomad pid=1755 2020-09-17T13:55:29.480Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=5d7bd025-9ce1-2b59-9cbc-9f6939518ff2 task=redis path=/opt/gopath/bin/nomad 2020-09-17T13:55:29.481Z [DEBUG] client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/opt/gopath/bin/nomad pid=1746 2020-09-17T13:55:29.481Z [DEBUG] client.driver_mgr.docker.docker_logger: plugin exited: driver=docker 2020-09-17T13:55:29.483Z [DEBUG] worker: dequeued evaluation: eval_id=5d687819-6343-034e-bcbc-d4e56da1cd5c 2020-09-17T13:55:29.483Z [DEBUG] client: evaluations triggered by node update: num_evals=1 2020-09-17T13:55:29.483Z [DEBUG] client: state updated: node_status=ready 2020-09-17T13:55:29.483Z [DEBUG] client.server_mgr: new server list: new_servers=[10.0.2.15:4647] old_servers=[0.0.0.0:4647, 10.0.2.15:4647] 2020-09-17T13:55:29.484Z [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=5d687819-6343-034e-bcbc-d4e56da1cd5c job_id=example namespace=default results="Total changes: (place 0) (destructive 0) (inplace 0) (stop 0) Desired Changes for "cache": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 1) (canary 0)" 2020-09-17T13:55:29.484Z [DEBUG] worker.service_sched: setting eval status: eval_id=5d687819-6343-034e-bcbc-d4e56da1cd5c job_id=example namespace=default status=complete 2020-09-17T13:55:29.485Z [DEBUG] worker: updated evaluation: eval="" 2020-09-17T13:55:29.486Z [DEBUG] worker: ack evaluation: eval_id=5d687819-6343-034e-bcbc-d4e56da1cd5c 2020-09-17T13:55:29.487Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=5d7bd025-9ce1-2b59-9cbc-9f6939518ff2 task=redis version=2 2020-09-17T13:55:29.487Z [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=5d7bd025-9ce1-2b59-9cbc-9f6939518ff2 task=redis @module=logmon address=/tmp/plugin812215936 network=unix timestamp=2020-09-17T13:55:29.487Z 2020-09-17T13:55:29.489Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=5d7bd025-9ce1-2b59-9cbc-9f6939518ff2 task=redis @module=logmon path=/var/nomad/alloc/5d7bd025-9ce1-2b59-9cbc-9f6939518ff2/alloc/logs/.redis.stdout.fifo timestamp=2020-09-17T13:55:29.489Z 2020-09-17T13:55:29.489Z [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=5d7bd025-9ce1-2b59-9cbc-9f6939518ff2 task=redis path=/var/nomad/alloc/5d7bd025-9ce1-2b59-9cbc-9f6939518ff2/alloc/logs/.redis.stderr.fifo @module=logmon timestamp=2020-09-17T13:55:29.489Z 2020-09-17T13:55:29.492Z [DEBUG] client.alloc_runner.task_runner.task_hook.stats_hook: failed to start stats collection for task with unrecoverable error: alloc_id=5d7bd025-9ce1-2b59-9cbc-9f6939518ff2 task=redis error="container stopped" 2020-09-17T13:55:29.502Z [WARN] client.driver_mgr.docker: RemoveImage on non-referenced counted image id: driver=docker image_id=sha256:87856cc39862cec77541d68382e4867d7ccb29a85a17221446c857ddaebca916 2020-09-17T13:55:29.502Z [INFO] client.alloc_runner.task_runner: restarting task: alloc_id=5d7bd025-9ce1-2b59-9cbc-9f6939518ff2 task=redis reason="Restart within policy" delay=16.680143874s 2020-09-17T13:55:29.728Z [DEBUG] client: updated allocations: index=26 total=1 pulled=0 filtered=1 2020-09-17T13:55:29.728Z [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1 2020-09-17T13:55:29.728Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0 ==> Newer Nomad version available: 0.12.4 (currently running: 0.12.4-dev) 2020-09-17T13:55:35.780Z [DEBUG] client: state changed, updating node and re-registering 2020-09-17T13:55:35.787Z [INFO] client: node registration complete 2020-09-17T13:55:36.091Z [DEBUG] http: request complete: method=GET path=/v1/allocations?prefix=5d duration=1.308001ms 2020-09-17T13:55:36.093Z [DEBUG] http: request complete: method=GET path=/v1/allocation/5d7bd025-9ce1-2b59-9cbc-9f6939518ff2?namespace=default duration=355.705µs 2020-09-17T13:55:36.104Z [DEBUG] http: request complete: method=GET path=/v1/client/allocation/5d7bd025-9ce1-2b59-9cbc-9f6939518ff2/stats duration=325.676µs 2020-09-17T13:55:46.190Z [DEBUG] client.driver_mgr.docker: image reference count incremented: driver=docker image_name=redis:3.2 image_id=sha256:87856cc39862cec77541d68382e4867d7ccb29a85a17221446c857ddaebca916 references=1 2020-09-17T13:55:46.190Z [DEBUG] client.driver_mgr.docker: configured resources: driver=docker task_name=redis memory=268435456 memory_reservation=0 cpu_shares=500 cpu_quota=0 cpu_period=0 2020-09-17T13:55:46.190Z [DEBUG] client.driver_mgr.docker: binding directories: driver=docker task_name=redis binds="[]string{"/var/nomad/alloc/5d7bd025-9ce1-2b59-9cbc-9f6939518ff2/alloc:/alloc", "/var/nomad/alloc/5d7bd025-9ce1-2b59-9cbc-9f6939518ff2/redis/local:/local", "/var/nomad/alloc/5d7bd025-9ce1-2b59-9cbc-9f6939518ff2/redis/secrets:/secrets"}" 2020-09-17T13:55:46.190Z [DEBUG] client.driver_mgr.docker: networking mode not specified; using default: driver=docker task_name=redis 2020-09-17T13:55:46.190Z [DEBUG] client.driver_mgr.docker: allocated static port: driver=docker task_name=redis ip=10.0.2.15 port=21929 2020-09-17T13:55:46.190Z [DEBUG] client.driver_mgr.docker: exposed port: driver=docker task_name=redis port=21929 2020-09-17T13:55:46.190Z [DEBUG] client.driver_mgr.docker: applied labels on the container: driver=docker task_name=redis labels=map[com.hashicorp.nomad.alloc_id:5d7bd025-9ce1-2b59-9cbc-9f6939518ff2] 2020-09-17T13:55:46.190Z [DEBUG] client.driver_mgr.docker: setting container name: driver=docker task_name=redis container_name=redis-5d7bd025-9ce1-2b59-9cbc-9f6939518ff2 2020-09-17T13:55:46.279Z [INFO] client.driver_mgr.docker: created container: driver=docker container_id=a8aa993ff461c9511e79554bf16c1d879982b16557eea19256d4ada13a469387 2020-09-17T13:55:46.863Z [INFO] client.driver_mgr.docker: started container: driver=docker container_id=a8aa993ff461c9511e79554bf16c1d879982b16557eea19256d4ada13a469387 2020-09-17T13:55:46.863Z [DEBUG] client.driver_mgr.docker.docker_logger: starting plugin: driver=docker path=/opt/gopath/bin/nomad args=[/opt/gopath/bin/nomad, docker_logger] 2020-09-17T13:55:46.864Z [DEBUG] client.driver_mgr.docker.docker_logger: plugin started: driver=docker path=/opt/gopath/bin/nomad pid=2029 2020-09-17T13:55:46.864Z [DEBUG] client.driver_mgr.docker.docker_logger: waiting for RPC address: driver=docker path=/opt/gopath/bin/nomad 2020-09-17T13:55:46.873Z [DEBUG] client.driver_mgr.docker.docker_logger: using plugin: driver=docker version=2 2020-09-17T13:55:46.873Z [DEBUG] client.driver_mgr.docker.docker_logger.nomad: plugin address: driver=docker @module=docker_logger address=/tmp/plugin853086176 network=unix timestamp=2020-09-17T13:55:46.873Z 2020-09-17T13:55:46.874Z [DEBUG] client.driver_mgr.docker.docker_logger.nomad: using client connection initialized from environment: driver=docker @module=docker_logger timestamp=2020-09-17T13:55:46.874Z ```

Note from the logs the line around "Reattachment process not found": this is where the task is being marked as lost so it can be rescheduled, and what we'd expect to see in your case. You'll want to provide the logs around the restoration process so we can debug this further.

tgross commented 3 years ago

It looks like we never got a chance to follow-up here. I'm going to close this issue for now but please feel free to re-open if you can reproduce with the requested diagnostics.

github-actions[bot] commented 1 year ago

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.