nomad stuck on (re)start

[MorphBonehunter]

Nomad version

Nomad v0.9.0 (18dd59056ee1d7b2df51256fe900a98460d3d6b9)

Operating system and Environment details

ArchLinux

Issue

After upgrading my nomad environment i discover nomad got stuck if i restart it while some docker containers are running if the . So i try to reproduce the steps with an minimal config.

Reproduction steps

• minimal nomad config:

  data_dir  = "/var/lib/nomad"
  log_level = "debug"
  server {
  enabled = true
  bootstrap_expect = 1
  }
  client {
  enabled = true
  }

• minimal docker job example:

  job "test" {
  datacenters = ["dc1"]
  type = "service"
  group "test" {
      task "test" {
          driver = "docker" 
          config {
              image = "redis:3.2"
          }
      }
  }
  }

• start with an clean nomad directory structure: rm -r /var/lib/nomad/* && mkdir /var/lib/nomad/plugins

• start nomad:

  
  nomad agent -config=nomad.hcl.example 
  ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
  ==> Loaded configuration from nomad.hcl.example
  ==> Starting Nomad agent...
  ==> Nomad agent configuration:
  
     Advertise Addrs: HTTP: 192.168.200.16:4646; RPC: 192.168.200.16:4647; Serf: 192.168.200.16:4648
          Bind Addrs: HTTP: 0.0.0.0:4646; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
              Client: true
           Log Level: debug
              Region: global (DC: dc1)
              Server: true
             Version: 0.9.0

==> Nomad agent started! Log data will stream in below:

2019-04-14T12:03:32.805+0200 [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/plugins
2019-04-14T12:03:32.805+0200 [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/plugins
2019-04-14T12:03:32.805+0200 [INFO ] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
2019-04-14T12:03:32.805+0200 [INFO ] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
2019-04-14T12:03:32.805+0200 [INFO ] agent: detected plugin: name=java type=driver plugin_version=0.1.0
2019-04-14T12:03:32.805+0200 [INFO ] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
2019-04-14T12:03:32.805+0200 [INFO ] agent: detected plugin: name=rkt type=driver plugin_version=0.1.0
2019-04-14T12:03:32.805+0200 [INFO ] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
2019-04-14T12:03:32.805+0200 [INFO ] agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
2019-04-14T12:03:32.816+0200 [INFO ] nomad: raft: Initial configuration (index=1): [{Suffrage:Voter ID:192.168.200.16:4647 Address:192.168.200.16:4647}]
2019-04-14T12:03:32.817+0200 [INFO ] nomad: raft: Node at 192.168.200.16:4647 [Follower] entering Follower state (Leader: "")
2019-04-14T12:03:32.818+0200 [INFO ] nomad: serf: EventMemberJoin: cassandra.global 192.168.200.16
2019-04-14T12:03:32.818+0200 [INFO ] nomad: starting scheduling worker(s): num_workers=6 schedulers="[service batch system _core]"
2019-04-14T12:03:32.818+0200 [INFO ] client: using state directory: state_dir=/var/lib/nomad/client
2019-04-14T12:03:32.819+0200 [INFO ] nomad: adding server: server="cassandra.global (Addr: 192.168.200.16:4647) (DC: dc1)"
2019-04-14T12:03:32.819+0200 [DEBUG] nomad: lost contact with Nomad quorum, falling back to Consul for server list
2019-04-14T12:03:32.820+0200 [ERROR] nomad: error looking up Nomad servers in Consul: error="server.nomad: unable to query Consul datacenters: Get http://127.0.0.1:8500/v1/catalog/datacenters: dial tcp 127.0.0.1:8500: connect: connection refused"
2019-04-14T12:03:32.823+0200 [INFO ] client: using alloc directory: alloc_dir=/var/lib/nomad/alloc
2019-04-14T12:03:32.825+0200 [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters="[arch cgroup consul cpu host memory network nomad signal storage vault env_aws env_gce]"
2019-04-14T12:03:32.825+0200 [INFO ] client.fingerprint_mgr.cgroup: cgroups are available
2019-04-14T12:03:32.825+0200 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=cgroup period=15s
2019-04-14T12:03:32.825+0200 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul period=15s
2019-04-14T12:03:32.826+0200 [DEBUG] client.fingerprint_mgr.cpu: detected cpu frequency: MHz=2207
2019-04-14T12:03:32.826+0200 [DEBUG] client.fingerprint_mgr.cpu: detected core count: cores=6
2019-04-14T12:03:32.829+0200 [WARN ] client.fingerprint_mgr.network: unable to parse speed: path=/usr/bin/ethtool device=lan0
2019-04-14T12:03:32.829+0200 [DEBUG] client.fingerprint_mgr.network: unable to parse link speed: path=/sys/class/net/lan0/speed
2019-04-14T12:03:32.829+0200 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected and no speed specified by user, falling back to default speed: mbits=1000
2019-04-14T12:03:32.829+0200 [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=lan0 IP=192.168.200.16
2019-04-14T12:03:32.833+0200 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault period=15s
2019-04-14T12:03:34.290+0200 [WARN ] nomad: raft: Heartbeat timeout from "" reached, starting election
2019-04-14T12:03:34.290+0200 [INFO ] nomad: raft: Node at 192.168.200.16:4647 [Candidate] entering Candidate state in term 2
2019-04-14T12:03:34.304+0200 [DEBUG] nomad: raft: Votes needed: 1
2019-04-14T12:03:34.304+0200 [DEBUG] nomad: raft: Vote granted from 192.168.200.16:4647 in term 2. Tally: 1
2019-04-14T12:03:34.304+0200 [INFO ] nomad: raft: Election won. Tally: 1
2019-04-14T12:03:34.304+0200 [INFO ] nomad: raft: Node at 192.168.200.16:4647 [Leader] entering Leader state
2019-04-14T12:03:34.304+0200 [INFO ] nomad: cluster leadership acquired
2019-04-14T12:03:34.322+0200 [DEBUG] nomad: leader reconciling job summaries: index=0
2019-04-14T12:03:34.834+0200 [DEBUG] client.fingerprint_mgr.env_aws: error querying AWS Metadata URL, skipping
2019-04-14T12:03:36.835+0200 [DEBUG] client.fingerprint_mgr.env_gce: could not read value for attribute: attribute=machine-type error="Get http://169.254.169.254/computeMetadata/v1/instance/machine-type: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
2019-04-14T12:03:36.835+0200 [DEBUG] client.fingerprint_mgr.env_gce: error querying GCE Metadata URL, skipping
2019-04-14T12:03:36.835+0200 [DEBUG] client.fingerprint_mgr: detected fingerprints: node_attrs="[arch cgroup cpu host network nomad signal storage]"
2019-04-14T12:03:36.835+0200 [INFO ] client.plugin: starting plugin manager: plugin-type=driver
2019-04-14T12:03:36.835+0200 [INFO ] client.plugin: starting plugin manager: plugin-type=device
2019-04-14T12:03:36.837+0200 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=driver
2019-04-14T12:03:36.837+0200 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=device
2019-04-14T12:03:36.837+0200 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=device
2019-04-14T12:03:36.838+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=qemu health=undetected description=
2019-04-14T12:03:36.839+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=raw_exec health=undetected description=disabled
2019-04-14T12:03:36.842+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=rkt health=undetected description="Failed to execute rkt version: exec: "rkt": executable file not found in $PATH"
2019-04-14T12:03:36.843+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=exec health=healthy description=Healthy
2019-04-14T12:03:36.844+0200 [INFO ] client: started client: node_id=cad4d630-9775-9c83-7218-b78913955043
2019-04-14T12:03:36.849+0200 [DEBUG] client: updated allocations: index=1 total=0 pulled=0 filtered=0
2019-04-14T12:03:36.849+0200 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=0
2019-04-14T12:03:36.852+0200 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=0 errors=0
2019-04-14T12:03:36.870+0200 [INFO ] client: node registration complete
2019-04-14T12:03:36.888+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=docker health=healthy description=Healthy
2019-04-14T12:03:36.933+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=java health=healthy description=Healthy
2019-04-14T12:03:36.933+0200 [DEBUG] client.driver_mgr: detected drivers: drivers="[exec docker java]"
2019-04-14T12:03:36.933+0200 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=driver
2019-04-14T12:03:43.075+0200 [DEBUG] client: state changed, updating node and re-registering
2019-04-14T12:03:43.089+0200 [INFO ] client: node registration complete

* deploy job:

nomad run example.job ==> Monitoring evaluation "d8b93abb" Evaluation triggered by job "test" Allocation "bfb2fea6" created: node "cad4d630", group "test" Evaluation status changed: "pending" -> "complete" ==> Evaluation "d8b93abb" finished with status "complete"

2019-04-14T12:04:01.653+0200 [DEBUG] worker: dequeued evaluation: eval_id=d8b93abb-1229-2459-7dfb-7b939bbe57aa
2019-04-14T12:04:01.653+0200 [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=d8b93abb-1229-2459-7dfb-7b939bbe57aa job_id=test namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 0)

Desired Changes for "test": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)" 2019-04-14T12:04:01.655+0200 [DEBUG] http: request complete: method=PUT path=/v1/jobs duration=21.415825ms 2019-04-14T12:04:01.663+0200 [DEBUG] http: request complete: method=GET path=/v1/evaluation/d8b93abb-1229-2459-7dfb-7b939bbe57aa duration=1.245792ms 2019-04-14T12:04:01.663+0200 [DEBUG] worker: submitted plan for evaluation: eval_id=d8b93abb-1229-2459-7dfb-7b939bbe57aa 2019-04-14T12:04:01.663+0200 [DEBUG] worker.service_sched: setting eval status: eval_id=d8b93abb-1229-2459-7dfb-7b939bbe57aa job_id=test namespace=default status=complete 2019-04-14T12:04:01.667+0200 [DEBUG] http: request complete: method=GET path=/v1/evaluation/d8b93abb-1229-2459-7dfb-7b939bbe57aa/allocations duration=1.133694ms 2019-04-14T12:04:01.670+0200 [DEBUG] worker: updated evaluation: eval="<Eval "d8b93abb-1229-2459-7dfb-7b939bbe57aa" JobID: "test" Namespace: "default">" 2019-04-14T12:04:01.670+0200 [DEBUG] worker: ack evaluation: eval_id=d8b93abb-1229-2459-7dfb-7b939bbe57aa 2019-04-14T12:04:01.670+0200 [DEBUG] client: updated allocations: index=11 total=1 pulled=1 filtered=0 2019-04-14T12:04:01.670+0200 [DEBUG] client: allocation updates: added=1 removed=0 updated=0 ignored=0 2019-04-14T12:04:01.682+0200 [DEBUG] client: allocation updates applied: added=1 removed=0 updated=0 ignored=0 errors=0 2019-04-14T12:04:01.708+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 task=test path=/usr/bin/nomad args="[/usr/bin/nomad logmon]" 2019-04-14T12:04:01.708+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 task=test path=/usr/bin/nomad pid=9074 2019-04-14T12:04:01.708+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 task=test path=/usr/bin/nomad 2019-04-14T12:04:01.907+0200 [DEBUG] client: updated allocations: index=13 total=1 pulled=0 filtered=1 2019-04-14T12:04:01.907+0200 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1 2019-04-14T12:04:01.907+0200 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0 2019-04-14T12:04:02.425+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 task=test version=2 2019-04-14T12:04:02.425+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 task=test address=/tmp/plugin692541686 network=unix @module=logmon timestamp=2019-04-14T12:04:02.425+0200 2019-04-14T12:04:02.428+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 task=test path=/var/lib/nomad/alloc/bfb2fea6-c9e3-d8e8-8818-a497df8c8b29/alloc/logs/.test.stdout.fifo @module=logmon timestamp=2019-04-14T12:04:02.428+0200 2019-04-14T12:04:02.428+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 task=test path=/var/lib/nomad/alloc/bfb2fea6-c9e3-d8e8-8818-a497df8c8b29/alloc/logs/.test.stderr.fifo @module=logmon timestamp=2019-04-14T12:04:02.428+0200 2019-04-14T12:04:02.446+0200 [DEBUG] client.driver_mgr.docker: image reference count incremented: driver=docker image_name=redis:3.2 image_id=sha256:87856cc39862cec77541d68382e4867d7ccb29a85a17221446c857ddaebca916 references=1 2019-04-14T12:04:02.446+0200 [DEBUG] client.driver_mgr.docker: configured resources: driver=docker task_name=test memory=314572800 cpu_shares=100 cpu_quota=0 cpu_period=0 2019-04-14T12:04:02.446+0200 [DEBUG] client.driver_mgr.docker: binding directories: driver=docker task_name=test binds="[]string{"/var/lib/nomad/alloc/bfb2fea6-c9e3-d8e8-8818-a497df8c8b29/alloc:/alloc", "/var/lib/nomad/alloc/bfb2fea6-c9e3-d8e8-8818-a497df8c8b29/test/local:/local", "/var/lib/nomad/alloc/bfb2fea6-c9e3-d8e8-8818-a497df8c8b29/test/secrets:/secrets"}" 2019-04-14T12:04:02.446+0200 [DEBUG] client.driver_mgr.docker: networking mode not specified; using default: driver=docker task_name=test network_mode=bridge 2019-04-14T12:04:02.446+0200 [DEBUG] client.driver_mgr.docker: no network interfaces are available: driver=docker task_name=test 2019-04-14T12:04:02.446+0200 [DEBUG] client.driver_mgr.docker: setting container name: driver=docker task_name=test container_name=test-bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 2019-04-14T12:04:02.493+0200 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=35eede160c63deff0c813f6030599cd4cd301d5066c77f50e4ef5942d05e32b8 2019-04-14T12:04:02.670+0200 [DEBUG] http: request complete: method=GET path=/v1/evaluation/d8b93abb-1229-2459-7dfb-7b939bbe57aa duration=163.048µs 2019-04-14T12:04:02.671+0200 [DEBUG] http: request complete: method=GET path=/v1/evaluation/d8b93abb-1229-2459-7dfb-7b939bbe57aa/allocations duration=219.381µs 2019-04-14T12:04:03.085+0200 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=35eede160c63deff0c813f6030599cd4cd301d5066c77f50e4ef5942d05e32b8 2019-04-14T12:04:03.085+0200 [DEBUG] client.driver_mgr.docker.docker_logger: starting plugin: driver=docker path=/usr/bin/nomad args="[/usr/bin/nomad docker_logger]" 2019-04-14T12:04:03.086+0200 [DEBUG] client.driver_mgr.docker.docker_logger: plugin started: driver=docker path=/usr/bin/nomad pid=9172 2019-04-14T12:04:03.086+0200 [DEBUG] client.driver_mgr.docker.docker_logger: waiting for RPC address: driver=docker path=/usr/bin/nomad 2019-04-14T12:04:03.169+0200 [DEBUG] client.driver_mgr.docker.docker_logger: using plugin: driver=docker version=2 2019-04-14T12:04:03.169+0200 [DEBUG] client.driver_mgr.docker.docker_logger.nomad: plugin address: driver=docker @module=docker_logger address=/tmp/plugin855862962 network=unix timestamp=2019-04-14T12:04:03.169+0200 2019-04-14T12:04:03.170+0200 [DEBUG] client.driver_mgr.docker.docker_logger.nomad: using client connection initialized from environment: driver=docker @module=docker_logger timestamp=2019-04-14T12:04:03.170+0200 2019-04-14T12:04:03.313+0200 [DEBUG] client: updated allocations: index=14 total=1 pulled=0 filtered=1 2019-04-14T12:04:03.313+0200 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1 2019-04-14T12:04:03.313+0200 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0 2019-04-14T12:04:13.305+0200 [DEBUG] client: updated allocations: index=15 total=1 pulled=0 filtered=1 2019-04-14T12:04:13.306+0200 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1 2019-04-14T12:04:13.306+0200 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0

* check docker and processlist:

docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 35eede160c63 redis:3.2 "docker-entrypoint.s…" 53 seconds ago Up 52 seconds 6379/tcp test-bfb2fea6-c9e3-d8e8-8818-a497df8c8b29

pgrep -a nomad 8967 nomad agent -config=nomad.hcl.example 9074 /usr/bin/nomad logmon 9172 /usr/bin/nomad docker_logger

* restart nomad:

^C==> Caught signal: interrupt 2019-04-14T12:05:55.530+0200 [DEBUG] http: shutting down http server 2019-04-14T12:05:55.530+0200 [INFO ] agent: requesting shutdown 2019-04-14T12:05:55.530+0200 [INFO ] client: shutting down 2019-04-14T12:05:55.531+0200 [DEBUG] client.driver_mgr.docker.docker_logger.nomad: plugin received interrupt signal, ignoring: driver=docker count=1 @module=docker_logger timestamp=2019-04-14T12:05:55.530+0200 2019-04-14T12:05:55.531+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin received interrupt signal, ignoring: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 task=test @module=logmon count=1 timestamp=2019-04-14T12:05:55.530+0200 2019-04-14T12:05:55.531+0200 [DEBUG] client.driver_mgr.docker: error collecting stats from container: container_id=35eede160c63deff0c813f6030599cd4cd301d5066c77f50e4ef5942d05e32b8 driver=docker error="context canceled" 2019-04-14T12:05:55.539+0200 [INFO ] client.plugin: shutting down plugin manager: plugin-type=device 2019-04-14T12:05:55.544+0200 [INFO ] client.plugin: plugin manager finished: plugin-type=device 2019-04-14T12:05:55.544+0200 [INFO ] client.plugin: shutting down plugin manager: plugin-type=driver 2019-04-14T12:05:55.579+0200 [INFO ] client.plugin: plugin manager finished: plugin-type=driver 2019-04-14T12:05:55.582+0200 [INFO ] nomad: shutting down server 2019-04-14T12:05:55.582+0200 [WARN ] nomad: serf: Shutdown without a Leave 2019-04-14T12:05:55.583+0200 [INFO ] agent: shutdown complete

nomad agent -config=nomad.hcl.example ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation. ==> Loaded configuration from nomad.hcl.example ==> Starting Nomad agent... ==> Nomad agent configuration:

   Advertise Addrs: HTTP: 192.168.200.16:4646; RPC: 192.168.200.16:4647; Serf: 192.168.200.16:4648
        Bind Addrs: HTTP: 0.0.0.0:4646; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
            Client: true
         Log Level: debug
            Region: global (DC: dc1)
            Server: true
           Version: 0.9.0

==> Nomad agent started! Log data will stream in below:

2019-04-14T12:06:10.867+0200 [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/plugins
2019-04-14T12:06:10.867+0200 [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/plugins
2019-04-14T12:06:10.867+0200 [INFO ] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
2019-04-14T12:06:10.867+0200 [INFO ] agent: detected plugin: name=rkt type=driver plugin_version=0.1.0
2019-04-14T12:06:10.867+0200 [INFO ] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
2019-04-14T12:06:10.867+0200 [INFO ] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
2019-04-14T12:06:10.867+0200 [INFO ] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
2019-04-14T12:06:10.867+0200 [INFO ] agent: detected plugin: name=java type=driver plugin_version=0.1.0
2019-04-14T12:06:10.867+0200 [INFO ] agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
2019-04-14T12:06:10.874+0200 [INFO ] nomad: raft: Initial configuration (index=1): [{Suffrage:Voter ID:192.168.200.16:4647 Address:192.168.200.16:4647}]
2019-04-14T12:06:10.874+0200 [INFO ] nomad: raft: Node at 192.168.200.16:4647 [Follower] entering Follower state (Leader: "")
2019-04-14T12:06:10.875+0200 [INFO ] nomad: serf: EventMemberJoin: cassandra.global 192.168.200.16
2019-04-14T12:06:10.875+0200 [INFO ] nomad: starting scheduling worker(s): num_workers=6 schedulers="[service batch system _core]"
2019-04-14T12:06:10.875+0200 [INFO ] client: using state directory: state_dir=/var/lib/nomad/client
2019-04-14T12:06:10.876+0200 [INFO ] client: using alloc directory: alloc_dir=/var/lib/nomad/alloc
2019-04-14T12:06:10.877+0200 [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters="[arch cgroup consul cpu host memory network nomad signal storage vault env_aws env_gce]"
2019-04-14T12:06:10.877+0200 [INFO ] client.fingerprint_mgr.cgroup: cgroups are available
2019-04-14T12:06:10.877+0200 [WARN ] nomad: serf: Failed to re-join any previously known node
2019-04-14T12:06:10.877+0200 [INFO ] nomad: adding server: server="cassandra.global (Addr: 192.168.200.16:4647) (DC: dc1)"
2019-04-14T12:06:10.877+0200 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=cgroup period=15s
2019-04-14T12:06:10.877+0200 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul period=15s
2019-04-14T12:06:10.877+0200 [DEBUG] nomad: lost contact with Nomad quorum, falling back to Consul for server list
2019-04-14T12:06:10.877+0200 [ERROR] nomad: error looking up Nomad servers in Consul: error="server.nomad: unable to query Consul datacenters: Get http://127.0.0.1:8500/v1/catalog/datacenters: dial tcp 127.0.0.1:8500: connect: connection refused"
2019-04-14T12:06:10.878+0200 [DEBUG] client.fingerprint_mgr.cpu: detected cpu frequency: MHz=2207
2019-04-14T12:06:10.878+0200 [DEBUG] client.fingerprint_mgr.cpu: detected core count: cores=6
2019-04-14T12:06:10.882+0200 [WARN ] client.fingerprint_mgr.network: unable to parse speed: path=/usr/bin/ethtool device=lan0
2019-04-14T12:06:10.882+0200 [DEBUG] client.fingerprint_mgr.network: unable to parse link speed: path=/sys/class/net/lan0/speed
2019-04-14T12:06:10.882+0200 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected and no speed specified by user, falling back to default speed: mbits=1000
2019-04-14T12:06:10.882+0200 [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=lan0 IP=192.168.200.16
2019-04-14T12:06:10.887+0200 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault period=15s
2019-04-14T12:06:12.323+0200 [WARN ] nomad: raft: Heartbeat timeout from "" reached, starting election
2019-04-14T12:06:12.323+0200 [INFO ] nomad: raft: Node at 192.168.200.16:4647 [Candidate] entering Candidate state in term 3
2019-04-14T12:06:12.336+0200 [DEBUG] nomad: raft: Votes needed: 1
2019-04-14T12:06:12.336+0200 [DEBUG] nomad: raft: Vote granted from 192.168.200.16:4647 in term 3. Tally: 1
2019-04-14T12:06:12.336+0200 [INFO ] nomad: raft: Election won. Tally: 1
2019-04-14T12:06:12.336+0200 [INFO ] nomad: raft: Node at 192.168.200.16:4647 [Leader] entering Leader state
2019-04-14T12:06:12.336+0200 [INFO ] nomad: cluster leadership acquired
2019-04-14T12:06:12.353+0200 [DEBUG] nomad: leader reconciling job summaries: index=15
2019-04-14T12:06:12.359+0200 [DEBUG] nomad.fsm.service_sched: reconciled current state with desired state: eval_id=f3ad22ef-3f57-b24b-d1e9-189fa285dd7a job_id=test namespace=default results="Total changes: (place 0) (destructive 0) (inplace 0) (stop 0)

Desired Changes for "test": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 1) (canary 0)" 2019-04-14T12:06:12.359+0200 [DEBUG] nomad.fsm.service_sched: setting eval status: eval_id=f3ad22ef-3f57-b24b-d1e9-189fa285dd7a job_id=test namespace=default status=complete 2019-04-14T12:06:12.888+0200 [DEBUG] client.fingerprint_mgr.env_aws: error querying AWS Metadata URL, skipping 2019-04-14T12:06:14.889+0200 [DEBUG] client.fingerprint_mgr.env_gce: could not read value for attribute: attribute=machine-type error="Get http://169.254.169.254/computeMetadata/v1/instance/machine-type: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" 2019-04-14T12:06:14.889+0200 [DEBUG] client.fingerprint_mgr.env_gce: error querying GCE Metadata URL, skipping 2019-04-14T12:06:14.889+0200 [DEBUG] client.fingerprint_mgr: detected fingerprints: node_attrs="[arch cgroup cpu host network nomad signal storage]" 2019-04-14T12:06:14.889+0200 [INFO ] client.plugin: starting plugin manager: plugin-type=driver 2019-04-14T12:06:14.889+0200 [INFO ] client.plugin: starting plugin manager: plugin-type=device 2019-04-14T12:06:14.890+0200 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=driver 2019-04-14T12:06:14.890+0200 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=device 2019-04-14T12:06:14.890+0200 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=device 2019-04-14T12:06:14.893+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=rkt health=undetected description="Failed to execute rkt version: exec: "rkt": executable file not found in $PATH" 2019-04-14T12:06:14.893+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=qemu health=undetected description= 2019-04-14T12:06:14.894+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=raw_exec health=undetected description=disabled 2019-04-14T12:06:14.894+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=exec health=healthy description=Healthy 2019-04-14T12:06:14.910+0200 [INFO ] client: started client: node_id=cad4d630-9775-9c83-7218-b78913955043 2019-04-14T12:06:14.913+0200 [DEBUG] client: updated allocations: index=15 total=1 pulled=0 filtered=1 2019-04-14T12:06:14.913+0200 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1 2019-04-14T12:06:14.914+0200 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0 2019-04-14T12:06:14.933+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=docker health=healthy description=Healthy 2019-04-14T12:06:14.934+0200 [INFO ] client: node registration complete 2019-04-14T12:06:14.986+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=java health=healthy description=Healthy 2019-04-14T12:06:14.986+0200 [DEBUG] client.driver_mgr: detected drivers: drivers="[exec docker java]" 2019-04-14T12:06:14.986+0200 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=driver 2019-04-14T12:06:15.715+0200 [DEBUG] client: evaluations triggered by node update: num_evals=1 2019-04-14T12:06:15.715+0200 [DEBUG] client: state updated: node_status=ready 2019-04-14T12:06:15.716+0200 [DEBUG] worker: dequeued evaluation: eval_id=aa076d89-7605-1aa4-2644-44638484301b 2019-04-14T12:06:15.716+0200 [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=aa076d89-7605-1aa4-2644-44638484301b job_id=test namespace=default results="Total changes: (place 0) (destructive 0) (inplace 0) (stop 0) Desired Changes for "test": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 1) (canary 0)" 2019-04-14T12:06:15.716+0200 [DEBUG] worker.service_sched: setting eval status: eval_id=aa076d89-7605-1aa4-2644-44638484301b job_id=test namespace=default status=complete 2019-04-14T12:06:15.721+0200 [DEBUG] worker: updated evaluation: eval="<Eval "aa076d89-7605-1aa4-2644-44638484301b" JobID: "test" Namespace: "default">" 2019-04-14T12:06:15.722+0200 [DEBUG] worker: ack evaluation: eval_id=aa076d89-7605-1aa4-2644-44638484301b 2019-04-14T12:06:23.117+0200 [DEBUG] client: state changed, updating node and re-registering 2019-04-14T12:06:23.121+0200 [INFO ] client: node registration complete

* check docker and processlist again:

docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 35eede160c63 redis:3.2 "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 6379/tcp test-bfb2fea6-c9e3-d8e8-8818-a497df8c8b29

ps -ef |grep nomad root 9074 1 0 12:04 pts/0 00:00:00 /usr/bin/nomad logmon root 9172 1 0 12:04 pts/0 00:00:00 /usr/bin/nomad docker_logger root 9501 5578 1 12:06 pts/0 00:00:01 nomad agent -config=nomad.hcl.example

* so here is the problem in my eyes, this testcase doesn't take into account that normaly nomad is startet with systemd and in this case the `logmon` and `docker_logger` got also killed and aren't running when nomad restarts. so i have to mimic the systemd behavior...
* so stop nomad again:

^C==> Caught signal: interrupt 2019-04-14T12:12:09.120+0200 [DEBUG] http: shutting down http server 2019-04-14T12:12:09.120+0200 [INFO ] agent: requesting shutdown 2019-04-14T12:12:09.120+0200 [INFO ] client: shutting down 2019-04-14T12:12:09.121+0200 [DEBUG] client.driver_mgr.docker: error collecting stats from container: container_id=35eede160c63deff0c813f6030599cd4cd301d5066c77f50e4ef5942d05e32b8 driver=docker error="context canceled" 2019-04-14T12:12:09.129+0200 [INFO ] client.plugin: shutting down plugin manager: plugin-type=device 2019-04-14T12:12:09.133+0200 [INFO ] client.plugin: plugin manager finished: plugin-type=device 2019-04-14T12:12:09.133+0200 [INFO ] client.plugin: shutting down plugin manager: plugin-type=driver 2019-04-14T12:12:09.164+0200 [INFO ] client.plugin: plugin manager finished: plugin-type=driver 2019-04-14T12:12:09.166+0200 [INFO ] nomad: shutting down server 2019-04-14T12:12:09.166+0200 [WARN ] nomad: serf: Shutdown without a Leave 2019-04-14T12:12:09.167+0200 [INFO ] agent: shutdown complete

* kill the remaining nomad processes `pkill nomad`
* restart nomad:

nomad agent -config=nomad.hcl.example ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation. ==> Loaded configuration from nomad.hcl.example ==> Starting Nomad agent...

From now on, nomad doesn't start up again, so i checked the docker and processlist again:

docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 35eede160c63 redis:3.2 "docker-entrypoint.s…" 12 minutes ago Up 12 minutes 6379/tcp test-bfb2fea6-c9e3-d8e8-8818-a497df8c8b29

pgrep -a nomad 10705 nomad agent -config=nomad.hcl.example 10737 /usr/bin/nomad docker_logger

So there isn't an new `logmon` process, just an `docker_logger` and i can not recover nomad.
So i kill nomad with `kill -3 10705` to produce an dump:

SIGQUIT: quit PC=0x463213 m=0 sigcode=0

goroutine 86 [syscall]: runtime.notetsleepg(0x3763e00, 0x6fc225ce9, 0x1) /usr/local/go/src/runtime/lock_futex.go:227 +0x37 fp=0xc0002c7758 sp=0xc0002c7728 pc=0x40f8d7 runtime.timerproc(0x3763de0) /usr/local/go/src/runtime/time.go:288 +0x30e fp=0xc0002c77d8 sp=0xc0002c7758 pc=0x4506ee runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0002c77e0 sp=0xc0002c77d8 pc=0x4613d1 created by runtime.(*timersBucket).addtimerLocked /usr/local/go/src/runtime/time.go:170 +0x114

goroutine 1 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport.(Stream).waitOnHeader(0xc00038ea00, 0xc0004a26a0, 0x1) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport/transport.go:242 +0xcc github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport.(Stream).RecvCompress(0xc00038ea00, 0xc00004e500, 0xaada71) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport/transport.go:253 +0x2b github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(csAttempt).recvMsg(0xc000236630, 0x1dc2300, 0xc000448620, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:741 +0x5c0 github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(clientStream).RecvMsg.func1(0xc000236630, 0xf2, 0xf2) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:648 +0x3d github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(clientStream).withRetry(0xc000330360, 0xc0007a6d30, 0xc0007a6d20, 0xab3750, 0xc000330360) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:534 +0x7f github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(clientStream).RecvMsg(0xc000330360, 0x1dc2300, 0xc000448620, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:647 +0x8d github.com/hashicorp/nomad/vendor/google.golang.org/grpc.invoke(0x223f0c0, 0xc0000b8060, 0x2006cf0, 0x40, 0x1ea3fe0, 0xc0004187e0, 0x1dc2300, 0xc000448620, 0xc000348000, 0x0, ...) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/call.go:73 +0x141 github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(ClientConn).Invoke(0xc000348000, 0x223f0c0, 0xc0000b8060, 0x2006cf0, 0x40, 0x1ea3fe0, 0xc0004187e0, 0x1dc2300, 0xc000448620, 0x0, ...) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/call.go:37 +0x1b3 github.com/hashicorp/nomad/drivers/docker/docklog/proto.(dockerLoggerClient).Start(0xc00000e178, 0x223f0c0, 0xc0000b8060, 0xc0004187e0, 0x0, 0x0, 0x0, 0x1e49e40, 0x1fa6101, 0xc00038c000) /opt/gopath/src/github.com/hashicorp/nomad/drivers/docker/docklog/proto/docker_logger.pb.go:235 +0xd2 github.com/hashicorp/nomad/drivers/docker/docklog.(dockerLoggerClient).Start(0xc0003161a0, 0xc00038c000, 0x7f986c07aad8, 0xc0003161a0) /opt/gopath/src/github.com/hashicorp/nomad/drivers/docker/docklog/client.go:26 +0x144 github.com/hashicorp/nomad/drivers/docker.(Driver).setupNewDockerLogger(0xc00035fd60, 0xc0003dc800, 0xc0004c80f0, 0xbf24dfd05046531e, 0xf70ccf36, 0x375e500, 0xc0004a2100, 0xc0000551a0, 0xc0005b30c0, 0xedea0b, ...) /opt/gopath/src/github.com/hashicorp/nomad/drivers/docker/driver.go:142 +0x2c1 github.com/hashicorp/nomad/drivers/docker.(Driver).RecoverTask(0xc00035fd60, 0xc001204180, 0xc0005b31d8, 0xc0005d8840) /opt/gopath/src/github.com/hashicorp/nomad/drivers/docker/driver.go:201 +0x701 github.com/hashicorp/nomad/client/allocrunner/taskrunner.(TaskRunner).restoreHandle(0xc00047a000, 0xc001204180, 0xc00058bb20, 0xc0000b8cc0) /opt/gopath/src/github.com/hashicorp/nomad/client/allocrunner/taskrunner/task_runner.go:884 +0x67 github.com/hashicorp/nomad/client/allocrunner/taskrunner.(TaskRunner).Restore(0xc00047a000, 0xc0012150b0, 0xc0005b32f0) /opt/gopath/src/github.com/hashicorp/nomad/client/allocrunner/taskrunner/task_runner.go:870 +0xfa github.com/hashicorp/nomad/client/allocrunner.(allocRunner).Restore(0xc00123e000, 0xc00123e000, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:333 +0x145 github.com/hashicorp/nomad/client.(Client).restoreState(0xc000460fc0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/client/client.go:972 +0x641 github.com/hashicorp/nomad/client.NewClient(0xc0002d2680, 0x222a780, 0xc0001e8228, 0x223f3c0, 0xc0004860d0, 0x0, 0x1b6ef00, 0xc000481ad0) /opt/gopath/src/github.com/hashicorp/nomad/client/client.go:420 +0x1199 github.com/hashicorp/nomad/command/agent.(Agent).setupClient(0xc000236580, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:733 +0x129 github.com/hashicorp/nomad/command/agent.NewAgent(0xc00008fe00, 0x22533a0, 0xc000383da0, 0x22250c0, 0xc000449600, 0xc0003a42d0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:116 +0x21d github.com/hashicorp/nomad/command/agent.(Command).setupAgent(0xc0001710a0, 0xc00008fe00, 0x22533a0, 0xc000383da0, 0x22250c0, 0xc000449600, 0xc0003a42d0, 0x3, 0xc00004c070) /opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:409 +0xc3 github.com/hashicorp/nomad/command/agent.(Command).Run(0xc0001710a0, 0xc0000a6170, 0x1, 0x1, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:588 +0x492 github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc00043e500, 0xc00043e500, 0xc000366c80, 0x37) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli/cli.go:255 +0x207 main.RunCustom(0xc0000a6160, 0x2, 0x2, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/main.go:131 +0x48e main.Run(0xc0000a6160, 0x2, 0x2, 0xc00009a058) /opt/gopath/src/github.com/hashicorp/nomad/main.go:76 +0x3f main.main() /opt/gopath/src/github.com/hashicorp/nomad/main.go:72 +0x62

goroutine 19 [syscall, 5 minutes]: os/signal.signal_recv(0x0) /usr/local/go/src/runtime/sigqueue.go:139 +0x9c os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x22 created by os/signal.init.0 /usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 67 [select, 1 minutes]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc000370a80) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 262 [semacquire, 5 minutes]: sync.runtime_Semacquire(0xc00049ed08) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(WaitGroup).Wait(0xc00049ed00) /usr/local/go/src/sync/waitgroup.go:130 +0x64 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run(0xc00020e900) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:174 +0x1ca created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.newInstanceManager /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:132 +0x271

goroutine 264 [select, 5 minutes]: github.com/hashicorp/nomad/client/devicemanager.(manager).fingerprint(0xc0011d8a00) /opt/gopath/src/github.com/hashicorp/nomad/client/devicemanager/manager.go:171 +0x30b created by github.com/hashicorp/nomad/client/devicemanager.(manager).Run /opt/gopath/src/github.com/hashicorp/nomad/client/devicemanager/manager.go:165 +0x330

goroutine 82 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.(*InmemSignal).run(0xc0005120c0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/inmem_signal.go:64 +0xb3 created by github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.NewInmemSignal /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/inmem_signal.go:38 +0x13e

goroutine 83 [sleep]: time.Sleep(0x3b9aca00) /usr/local/go/src/runtime/time.go:105 +0x14f github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.(*Metrics).collectStats(0xc000236370) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/metrics.go:168 +0x2f created by github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.New /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/start.go:75 +0x119

goroutine 84 [select, 5 minutes]: github.com/hashicorp/nomad/command/agent/consul.(ServiceClient).Run(0xc0004860d0) /opt/gopath/src/github.com/hashicorp/nomad/command/agent/consul/client.go:301 +0x229 created by github.com/hashicorp/nomad/command/agent.(Agent).setupConsul /opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:1033 +0x170

goroutine 85 [select]: github.com/hashicorp/nomad/command/agent/consul.checkConsulTLSSkipVerify(0x223f080, 0xc000380200, 0x22533a0, 0xc001148120, 0x224cb20, 0xc0011e4020, 0xc00009a300) /opt/gopath/src/github.com/hashicorp/nomad/command/agent/consul/version_checker.go:42 +0x180 created by github.com/hashicorp/nomad/command/agent/consul.(*ServiceClient).Run /opt/gopath/src/github.com/hashicorp/nomad/command/agent/consul/client.go:295 +0x13e

goroutine 55 [select, 1 minutes]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc000506120) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 56 [select, 1 minutes]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc0005061e0) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 57 [select, 1 minutes]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc0005062a0) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 58 [select, 1 minutes]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc000506360) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 133 [select, 5 minutes]: github.com/hashicorp/nomad/nomad.(*EvaluatePool).run(0xc00119c000, 0xc000498060) /opt/gopath/src/github.com/hashicorp/nomad/nomad/plan_apply_pool.go:104 +0x1af created by github.com/hashicorp/nomad/nomad.NewEvaluatePool /opt/gopath/src/github.com/hashicorp/nomad/nomad/plan_apply_pool.go:49 +0x12e

goroutine 59 [select, 1 minutes]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc000506420) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 60 [select, 1 minutes]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc0005064e0) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 134 [select, 5 minutes]: github.com/hashicorp/nomad/nomad.(*EvaluatePool).run(0xc00119c000, 0xc0004980c0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/plan_apply_pool.go:104 +0x1af created by github.com/hashicorp/nomad/nomad.NewEvaluatePool /opt/gopath/src/github.com/hashicorp/nomad/nomad/plan_apply_pool.go:49 +0x12e

goroutine 90 [select, 1 minutes]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc001148ba0) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 91 [select, 1 minutes]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc0011496e0) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 92 [select, 1 minutes]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc001149920) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 68 [select, 1 minutes]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc000371560) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 69 [select, 1 minutes]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc000315560) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 70 [select]: github.com/hashicorp/nomad/helper/pool.(*ConnPool).reap(0xc00036a500) /opt/gopath/src/github.com/hashicorp/nomad/helper/pool/pool.go:442 +0x305 created by github.com/hashicorp/nomad/helper/pool.NewPool /opt/gopath/src/github.com/hashicorp/nomad/helper/pool/pool.go:172 +0x13b

goroutine 71 [select, 5 minutes]: github.com/hashicorp/nomad/nomad.(RaftLayer).Accept(0xc00036a960, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/raft_rpc.go:63 +0xb2 github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(NetworkTransport).listen(0xc0002ce160) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/net_transport.go:470 +0x53 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.NewNetworkTransportWithConfig /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/net_transport.go:166 +0x16b

goroutine 72 [select]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(Raft).leaderLoop(0xc0000ca2c0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/raft.go:501 +0x300 github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(Raft).runLeader(0xc0000ca2c0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/raft.go:420 +0x34b github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(Raft).run(0xc0000ca2c0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/raft.go:140 +0x68 github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(Raft).run-fm() /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/api.go:505 +0x2a github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(raftState).goFunc.func1(0xc0000ca2c0, 0xc0003d73c0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/state.go:146 +0x53 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(raftState).goFunc /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/state.go:144 +0x66

goroutine 73 [select]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(Raft).runFSM(0xc0000ca2c0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/fsm.go:116 +0x1df github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(Raft).runFSM-fm() /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/api.go:506 +0x2a github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(raftState).goFunc.func1(0xc0000ca2c0, 0xc0003d73d0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/state.go:146 +0x53 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(raftState).goFunc /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/state.go:144 +0x66

goroutine 74 [select, 2 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(Raft).runSnapshots(0xc0000ca2c0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/snapshot.go:71 +0x115 github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(Raft).runSnapshots-fm() /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/api.go:507 +0x2a github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(raftState).goFunc.func1(0xc0000ca2c0, 0xc0003d73e0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/state.go:146 +0x53 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(raftState).goFunc /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/state.go:144 +0x66

goroutine 75 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.(*serfQueries).stream(0xc00052e8a0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/internal_query.go:80 +0xb4 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.newSerfQueries /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/internal_query.go:73 +0xab

goroutine 76 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.(*Snapshotter).teeStream(0xc00040f380) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/snapshot.go:223 +0xfa created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.NewSnapshotter /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/snapshot.go:152 +0x545

goroutine 77 [select]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.(*Snapshotter).stream(0xc00040f380) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/snapshot.go:266 +0x15b created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.NewSnapshotter /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/snapshot.go:153 +0x567

goroutine 78 [IO wait, 5 minutes]: internal/poll.runtime_pollWait(0x7f986c0f9e30, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x66 internal/poll.(pollDesc).wait(0xc000136918, 0x72, 0xc000086500, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a internal/poll.(pollDesc).waitRead(0xc000136918, 0xffffffffffffff00, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Accept(0xc000136900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1a0 net.(netFD).accept(0xc000136900, 0x0, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(TCPListener).accept(0xc0001e9270, 0x42fd92, 0x8, 0xc000086080) /usr/local/go/src/net/tcpsock_posix.go:139 +0x2e net.(TCPListener).AcceptTCP(0xc0001e9270, 0x2035f40, 0xc00044f0c0, 0x0) /usr/local/go/src/net/tcpsock.go:247 +0x47 github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.(*NetTransport).tcpListen(0xc00044f0a0, 0xc0001e9270) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/net_transport.go:235 +0x66 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.NewNetTransport /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/net_transport.go:109 +0x219

goroutine 79 [IO wait, 5 minutes]: internal/poll.runtime_pollWait(0x7f986c0f9d60, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x66 internal/poll.(pollDesc).wait(0xc000136998, 0x72, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a internal/poll.(pollDesc).waitRead(0xc000136998, 0xc00018a000, 0x10000, 0x10000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).ReadFrom(0xc000136980, 0xc00018a000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:219 +0x168 net.(netFD).readFrom(0xc000136980, 0xc00018a000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x4108af, 0xc000590e70) /usr/local/go/src/net/fd_unix.go:208 +0x5b net.(UDPConn).readFrom(0xc0001e9280, 0xc00018a000, 0x10000, 0x10000, 0x101, 0xc000590e48, 0xc000590ec0, 0x448127) /usr/local/go/src/net/udpsock_posix.go:47 +0x6a net.(UDPConn).ReadFrom(0xc0001e9280, 0xc00018a000, 0x10000, 0x10000, 0x10000, 0x10000, 0x0, 0x0, 0x0) /usr/local/go/src/net/udpsock.go:121 +0x6d github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.(*NetTransport).udpListen(0xc00044f0a0, 0xc0001e9280) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/net_transport.go:270 +0xc5 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.NewNetTransport /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/net_transport.go:110 +0x194

goroutine 80 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.(*Memberlist).streamListen(0xc000393440) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/net.go:198 +0x105 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.newMemberlist /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/memberlist.go:186 +0x42f

goroutine 81 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.(*Memberlist).packetListen(0xc000393440) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/net.go:301 +0x133 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.newMemberlist /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/memberlist.go:187 +0x454

goroutine 98 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.(*Memberlist).packetHandler(0xc000393440) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/net.go:420 +0x332 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.newMemberlist /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/memberlist.go:188 +0x479

goroutine 100 [select]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.(Memberlist).triggerFunc(0xc000393440, 0x12a05f200, 0xc000507020, 0xc000202780, 0xc000317eb0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/state.go:133 +0x164 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.(Memberlist).schedule /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/state.go:99 +0x365

goroutine 101 [select, 1 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.(Memberlist).pushPullTrigger(0xc000393440, 0xc000202780) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/state.go:160 +0x1a8 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.(Memberlist).schedule /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/state.go:105 +0x2c0

goroutine 102 [select]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.(Memberlist).triggerFunc(0xc000393440, 0x1dcd6500, 0xc0005070e0, 0xc000202780, 0xc000317ee0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/state.go:133 +0x164 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist.(Memberlist).schedule /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/memberlist/state.go:111 +0x1ba

goroutine 103 [select]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.(*Serf).handleReap(0xc0002c5380) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/serf.go:1412 +0x11d created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.Create /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/serf.go:406 +0x8a8

goroutine 104 [select, 1 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.(*Serf).handleReconnect(0xc0002c5380) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/serf.go:1430 +0xd0 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.Create /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/serf.go:407 +0x8cd

goroutine 105 [select, 1 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.(*Serf).checkQueueDepth(0xc0002c5380, 0x1f994fd, 0x6, 0xc00052e9f0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/serf.go:1545 +0xd5 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.Create /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/serf.go:408 +0x910

goroutine 106 [select, 1 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.(*Serf).checkQueueDepth(0xc0002c5380, 0x1f97d9f, 0x5, 0xc00052ea20) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/serf.go:1545 +0xd5 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.Create /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/serf.go:409 +0x956

goroutine 107 [select, 1 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.(*Serf).checkQueueDepth(0xc0002c5380, 0x1f97fbb, 0x5, 0xc00052ea50) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/serf.go:1545 +0xd5 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf.Create /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/serf/serf/serf.go:410 +0x99c

goroutine 147 [select]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/consul/agent/consul/autopilot.(Autopilot).serverHealthLoop(0xc00035ff40) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/consul/agent/consul/autopilot/autopilot.go:333 +0x121 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/consul/agent/consul/autopilot.(Autopilot).Start /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/consul/agent/consul/autopilot/autopilot.go:81 +0x140

goroutine 109 [sync.Cond.Wait, 5 minutes]: sync.runtime_notifyListWait(0xc000086790, 0x0) /usr/local/go/src/runtime/sema.go:510 +0xeb sync.(Cond).Wait(0xc000086780) /usr/local/go/src/sync/cond.go:56 +0x92 github.com/hashicorp/nomad/nomad.(Worker).checkPaused(0xc000508770) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:100 +0x45 github.com/hashicorp/nomad/nomad.(Worker).dequeueEvaluation(0xc000508770, 0x1dcd6500, 0x0, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:157 +0x13d github.com/hashicorp/nomad/nomad.(Worker).run(0xc000508770) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:109 +0x49 created by github.com/hashicorp/nomad/nomad.NewWorker /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:82 +0x14f

goroutine 110 [sync.Cond.Wait, 5 minutes]: sync.runtime_notifyListWait(0xc0000868d0, 0x0) /usr/local/go/src/runtime/sema.go:510 +0xeb sync.(Cond).Wait(0xc0000868c0) /usr/local/go/src/sync/cond.go:56 +0x92 github.com/hashicorp/nomad/nomad.(Worker).checkPaused(0xc0005087e0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:100 +0x45 github.com/hashicorp/nomad/nomad.(Worker).dequeueEvaluation(0xc0005087e0, 0x1dcd6500, 0x0, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:157 +0x13d github.com/hashicorp/nomad/nomad.(Worker).run(0xc0005087e0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:109 +0x49 created by github.com/hashicorp/nomad/nomad.NewWorker /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:82 +0x14f

goroutine 111 [sync.Cond.Wait, 5 minutes]: sync.runtime_notifyListWait(0xc000086a90, 0x0) /usr/local/go/src/runtime/sema.go:510 +0xeb sync.(Cond).Wait(0xc000086a80) /usr/local/go/src/sync/cond.go:56 +0x92 github.com/hashicorp/nomad/nomad.(Worker).checkPaused(0xc000508850) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:100 +0x45 github.com/hashicorp/nomad/nomad.(Worker).dequeueEvaluation(0xc000508850, 0x1dcd6500, 0x0, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:157 +0x13d github.com/hashicorp/nomad/nomad.(Worker).run(0xc000508850) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:109 +0x49 created by github.com/hashicorp/nomad/nomad.NewWorker /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:82 +0x14f

goroutine 112 [sync.Cond.Wait, 5 minutes]: sync.runtime_notifyListWait(0xc000086ad0, 0x0) /usr/local/go/src/runtime/sema.go:510 +0xeb sync.(Cond).Wait(0xc000086ac0) /usr/local/go/src/sync/cond.go:56 +0x92 github.com/hashicorp/nomad/nomad.(Worker).checkPaused(0xc0005088c0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:100 +0x45 github.com/hashicorp/nomad/nomad.(Worker).dequeueEvaluation(0xc0005088c0, 0x1dcd6500, 0x0, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:157 +0x13d github.com/hashicorp/nomad/nomad.(Worker).run(0xc0005088c0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:109 +0x49 created by github.com/hashicorp/nomad/nomad.NewWorker /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:82 +0x14f

goroutine 113 [select]: github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers(0xc0003a7360, 0xc000315bc0, 0x4, 0x6, 0xc001207980, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:483 +0x21d github.com/hashicorp/nomad/nomad.(EvalBroker).Dequeue(0xc0003a7360, 0xc000315bc0, 0x4, 0x6, 0x1dcd6500, 0xbf24e01e2db267f7, 0x497d844a20, 0x375e500, 0xc0011755c0, 0xc00052ce00) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:348 +0x106 github.com/hashicorp/nomad/nomad.(Eval).Dequeue(0xc000366300, 0xc001207920, 0xc0011755c0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_endpoint.go:100 +0x1ef reflect.Value.call(0xc00052cea0, 0xc0001e88d8, 0x13, 0x1f96bd1, 0x4, 0xc001169ca8, 0x3, 0x3, 0x1d81d00, 0x19, ...) /usr/local/go/src/reflect/value.go:447 +0x454 reflect.Value.Call(0xc00052cea0, 0xc0001e88d8, 0x13, 0xc001169ca8, 0x3, 0x3, 0x16, 0x16, 0x16) /usr/local/go/src/reflect/value.go:308 +0xa4 net/rpc.(service).call(0xc000381440, 0xc00036a820, 0xc00016d4a0, 0x0, 0xc0003d5680, 0xc00047e1c0, 0x1cfe3e0, 0xc001207920, 0x16, 0x1b91e00, ...) /usr/local/go/src/net/rpc/server.go:384 +0x14e net/rpc.(Server).ServeRequest(0xc00036a820, 0x223f400, 0xc001175580, 0x5cb308f8, 0xc02db204ed) /usr/local/go/src/net/rpc/server.go:505 +0x242 github.com/hashicorp/nomad/nomad.(Server).RPC(0xc000102a00, 0x1fa3a8d, 0xc, 0x1cfe3e0, 0xc000702840, 0x1b91e00, 0xc0005308c0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/server.go:1408 +0xb9 github.com/hashicorp/nomad/nomad.(Worker).dequeueEvaluation(0xc000508930, 0x1dcd6500, 0x24, 0xc000482330, 0x24, 0x1, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:161 +0x1ba github.com/hashicorp/nomad/nomad.(Worker).run(0xc000508930) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:109 +0x49 created by github.com/hashicorp/nomad/nomad.NewWorker /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:82 +0x14f

goroutine 114 [select]: github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers(0xc0003a7360, 0xc000315bc0, 0x4, 0x6, 0xc0012119e0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:483 +0x21d github.com/hashicorp/nomad/nomad.(EvalBroker).Dequeue(0xc0003a7360, 0xc000315bc0, 0x4, 0x6, 0x1dcd6500, 0xbf24e01e1faf78e3, 0x496f815b2f, 0x375e500, 0xc00113fc40, 0xc00052ce00) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:348 +0x106 github.com/hashicorp/nomad/nomad.(Eval).Dequeue(0xc000366300, 0xc001211980, 0xc00113fc40, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_endpoint.go:100 +0x1ef reflect.Value.call(0xc00052cea0, 0xc0001e88d8, 0x13, 0x1f96bd1, 0x4, 0xc00074dca8, 0x3, 0x3, 0x1d81d00, 0x19, ...) /usr/local/go/src/reflect/value.go:447 +0x454 reflect.Value.Call(0xc00052cea0, 0xc0001e88d8, 0x13, 0xc00074dca8, 0x3, 0x3, 0x16, 0x16, 0x16) /usr/local/go/src/reflect/value.go:308 +0xa4 net/rpc.(service).call(0xc000381440, 0xc00036a820, 0xc000375560, 0x0, 0xc0003d5680, 0xc00050e080, 0x1cfe3e0, 0xc001211980, 0x16, 0x1b91e00, ...) /usr/local/go/src/net/rpc/server.go:384 +0x14e net/rpc.(Server).ServeRequest(0xc00036a820, 0x223f400, 0xc00113ef00, 0x5cb308f8, 0xc01faf1993) /usr/local/go/src/net/rpc/server.go:505 +0x242 github.com/hashicorp/nomad/nomad.(Server).RPC(0xc000102a00, 0x1fa3a8d, 0xc, 0x1cfe3e0, 0xc00071eba0, 0x1b91e00, 0xc00118ef80, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/server.go:1408 +0xb9 github.com/hashicorp/nomad/nomad.(Worker).dequeueEvaluation(0xc0005089a0, 0x1dcd6500, 0x24, 0xc001244960, 0x24, 0x1, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:161 +0x1ba github.com/hashicorp/nomad/nomad.(Worker).run(0xc0005089a0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:109 +0x49 created by github.com/hashicorp/nomad/nomad.NewWorker /opt/gopath/src/github.com/hashicorp/nomad/nomad/worker.go:82 +0x14f

goroutine 115 [select]: github.com/hashicorp/nomad/nomad.(Server).setupBootstrapHandler.func3(0xc00052f6e0, 0xc000102a00) /opt/gopath/src/github.com/hashicorp/nomad/nomad/server.go:867 +0xeb created by github.com/hashicorp/nomad/nomad.(Server).setupBootstrapHandler /opt/gopath/src/github.com/hashicorp/nomad/nomad/server.go:863 +0xfa

goroutine 116 [select, 5 minutes]: github.com/hashicorp/nomad/nomad.(*Server).monitorLeadership(0xc000102a00) /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:61 +0xd6 created by github.com/hashicorp/nomad/nomad.NewServer /opt/gopath/src/github.com/hashicorp/nomad/nomad/server.go:385 +0x1354

goroutine 117 [select, 5 minutes]: github.com/hashicorp/nomad/nomad.(*Server).serfEventHandler(0xc000102a00) /opt/gopath/src/github.com/hashicorp/nomad/nomad/serf.go:30 +0xd8 created by github.com/hashicorp/nomad/nomad.NewServer /opt/gopath/src/github.com/hashicorp/nomad/nomad/server.go:388 +0x1379

goroutine 118 [IO wait, 5 minutes]: internal/poll.runtime_pollWait(0x7f986c0f9f00, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x66 internal/poll.(pollDesc).wait(0xc000136418, 0x72, 0xc0001ee000, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a internal/poll.(pollDesc).waitRead(0xc000136418, 0xffffffffffffff00, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Accept(0xc000136400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1a0 net.(netFD).accept(0xc000136400, 0x1, 0x1, 0xc00055af48) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(TCPListener).accept(0xc0001e91d0, 0x40ab3a, 0xc00045c000, 0x0) /usr/local/go/src/net/tcpsock_posix.go:139 +0x2e net.(TCPListener).Accept(0xc0001e91d0, 0xc00045c000, 0x0, 0x3f800000, 0xc000086d00) /usr/local/go/src/net/tcpsock.go:260 +0x47 github.com/hashicorp/nomad/nomad.(rpcHandler).listen(0xc000366200, 0x223f080, 0xc000086d00) /opt/gopath/src/github.com/hashicorp/nomad/nomad/rpc.go:98 +0xa9 created by github.com/hashicorp/nomad/nomad.(Server).startRPCListener /opt/gopath/src/github.com/hashicorp/nomad/nomad/server.go:419 +0x91

goroutine 119 [select]: github.com/hashicorp/nomad/nomad.(*EvalBroker).EmitStats(0xc0003a7360, 0x3b9aca00, 0xc000202660) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:820 +0x2d2 created by github.com/hashicorp/nomad/nomad.NewServer /opt/gopath/src/github.com/hashicorp/nomad/nomad/server.go:394 +0x13cc

goroutine 120 [select]: github.com/hashicorp/nomad/nomad.(*PlanQueue).EmitStats(0xc00036a8c0, 0x3b9aca00, 0xc000202660) /opt/gopath/src/github.com/hashicorp/nomad/nomad/plan_queue.go:200 +0xe1 created by github.com/hashicorp/nomad/nomad.NewServer /opt/gopath/src/github.com/hashicorp/nomad/nomad/server.go:397 +0x1411

goroutine 121 [select]: github.com/hashicorp/nomad/nomad.(*BlockedEvals).EmitStats(0xc0003a7400, 0x3b9aca00, 0xc000202660) /opt/gopath/src/github.com/hashicorp/nomad/nomad/blocked_evals.go:671 +0xe4 created by github.com/hashicorp/nomad/nomad.NewServer /opt/gopath/src/github.com/hashicorp/nomad/nomad/server.go:400 +0x1452

goroutine 122 [select]: github.com/hashicorp/nomad/nomad.(*vaultClient).EmitStats(0xc000486750, 0x3b9aca00, 0xc000202660) /opt/gopath/src/github.com/hashicorp/nomad/nomad/vault.go:1295 +0x103 created by github.com/hashicorp/nomad/nomad.NewServer /opt/gopath/src/github.com/hashicorp/nomad/nomad/server.go:403 +0x1499

goroutine 123 [select]: github.com/hashicorp/nomad/nomad.(*nodeHeartbeater).heartbeatStats(0xc001246630) /opt/gopath/src/github.com/hashicorp/nomad/nomad/heartbeat.go:201 +0xef created by github.com/hashicorp/nomad/nomad.NewServer /opt/gopath/src/github.com/hashicorp/nomad/nomad/server.go:406 +0x14c5

goroutine 124 [select]: github.com/hashicorp/nomad/helper/pool.(*ConnPool).reap(0xc0000d0dc0) /opt/gopath/src/github.com/hashicorp/nomad/helper/pool/pool.go:442 +0x305 created by github.com/hashicorp/nomad/helper/pool.NewPool /opt/gopath/src/github.com/hashicorp/nomad/helper/pool/pool.go:172 +0x13b

goroutine 125 [select, 5 minutes]: github.com/hashicorp/nomad/client.(Client).rpcConnListener(0xc000460fc0) /opt/gopath/src/github.com/hashicorp/nomad/client/rpc.go:248 +0xf4 created by github.com/hashicorp/nomad/client.(Client).setupClientRpc /opt/gopath/src/github.com/hashicorp/nomad/client/rpc.go:228 +0x10e

goroutine 148 [select]: github.com/hashicorp/nomad/nomad.(PlanQueue).Dequeue(0xc00036a8c0, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/plan_queue.go:155 +0x112 github.com/hashicorp/nomad/nomad.(planner).planApply(0xc000366260) /opt/gopath/src/github.com/hashicorp/nomad/nomad/plan_apply.go:86 +0xc1 created by github.com/hashicorp/nomad/nomad.(*Server).establishLeadership /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:206 +0x224

goroutine 149 [select, 5 minutes]: github.com/hashicorp/nomad/nomad.(EvalBroker).runDelayedEvalsWatcher(0xc0003a7360, 0x223f080, 0xc000796400, 0xc0002024e0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:761 +0x198 created by github.com/hashicorp/nomad/nomad.(EvalBroker).SetEnabled /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:170 +0xf5

goroutine 126 [select]: github.com/hashicorp/nomad/client.(FingerprintManager).runFingerprint(0xc0000d1090, 0x2229680, 0xc000518a20, 0x37e11d600, 0x1f99b5d, 0x6) /opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:147 +0x22d created by github.com/hashicorp/nomad/client.(FingerprintManager).setupFingerprinters /opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:131 +0x360

goroutine 135 [select]: github.com/hashicorp/nomad/nomad.(*EvaluatePool).run(0xc00119c000, 0xc000498120) /opt/gopath/src/github.com/hashicorp/nomad/nomad/plan_apply_pool.go:104 +0x1af created by github.com/hashicorp/nomad/nomad.NewEvaluatePool /opt/gopath/src/github.com/hashicorp/nomad/nomad/plan_apply_pool.go:49 +0x12e

goroutine 130 [select]: github.com/hashicorp/nomad/client.(FingerprintManager).runFingerprint(0xc0000d1090, 0x2229700, 0xc0005191d0, 0x37e11d600, 0x1f99bab, 0x6) /opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:147 +0x22d created by github.com/hashicorp/nomad/client.(FingerprintManager).setupFingerprinters /opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:131 +0x360

goroutine 146 [select]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/consul/agent/consul/autopilot.(Autopilot).run(0xc00035ff40) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/consul/agent/consul/autopilot/autopilot.go:108 +0x152 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/consul/agent/consul/autopilot.(Autopilot).Start /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/consul/agent/consul/autopilot/autopilot.go:80 +0x11e

goroutine 33 [select]: github.com/hashicorp/nomad/nomad.(Server).leaderLoop(0xc000102a00, 0xc00045e180) /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:161 +0x1d1 github.com/hashicorp/nomad/nomad.(Server).monitorLeadership.func1(0xc000504130, 0xc000102a00, 0xc00045e180) /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:74 +0x5b created by github.com/hashicorp/nomad/nomad.(*Server).monitorLeadership /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:72 +0x1b9

goroutine 48 [select]: github.com/hashicorp/nomad/client.(FingerprintManager).runFingerprint(0xc0000d1090, 0x2229940, 0xc000480ff0, 0x37e11d600, 0x1f98ebb, 0x5) /opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:147 +0x22d created by github.com/hashicorp/nomad/client.(FingerprintManager).setupFingerprinters /opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:131 +0x360

goroutine 150 [select, 5 minutes]: github.com/hashicorp/nomad/nomad.(BlockedEvals).watchCapacity(0xc0003a7400, 0xc000202600, 0xc000315e60) /opt/gopath/src/github.com/hashicorp/nomad/nomad/blocked_evals.go:480 +0xf7 created by github.com/hashicorp/nomad/nomad.(BlockedEvals).SetEnabled /opt/gopath/src/github.com/hashicorp/nomad/nomad/blocked_evals.go:141 +0x8a

goroutine 151 [select]: github.com/hashicorp/nomad/nomad.(BlockedEvals).prune(0xc0003a7400, 0xc000202600) /opt/gopath/src/github.com/hashicorp/nomad/nomad/blocked_evals.go:689 +0xf5 created by github.com/hashicorp/nomad/nomad.(BlockedEvals).SetEnabled /opt/gopath/src/github.com/hashicorp/nomad/nomad/blocked_evals.go:142 +0xb8

goroutine 152 [select, 5 minutes]: github.com/hashicorp/nomad/nomad/deploymentwatcher.(*AllocUpdateBatcher).batcher(0xc000794a80) /opt/gopath/src/github.com/hashicorp/nomad/nomad/deploymentwatcher/batcher.go:67 +0x2e4 created by github.com/hashicorp/nomad/nomad/deploymentwatcher.NewAllocUpdateBatcher /opt/gopath/src/github.com/hashicorp/nomad/nomad/deploymentwatcher/batcher.go:37 +0xc6

goroutine 153 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-memdb.watchFew(0x223f080, 0xc000796440, 0xc000541d10, 0x20, 0x20, 0x1205813, 0xc000542040) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-memdb/watch_few.go:16 +0x70a github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-memdb.WatchSet.WatchCtx(0xc000536030, 0x223f080, 0xc000796440, 0x3782560, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-memdb/watch.go:82 +0x126 github.com/hashicorp/nomad/nomad/state.(StateStore).BlockingQuery(0xc0003fd8c0, 0xc001193ef8, 0x1, 0x223f080, 0xc000796440, 0x0, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/state/state_store.go:161 +0x16e github.com/hashicorp/nomad/nomad/deploymentwatcher.(Watcher).getDeploys(0xc0003d4300, 0x223f080, 0xc000796440, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/deploymentwatcher/deployments_watcher.go:180 +0x7a github.com/hashicorp/nomad/nomad/deploymentwatcher.(Watcher).watchDeployments(0xc0003d4300, 0x223f080, 0xc000796440) /opt/gopath/src/github.com/hashicorp/nomad/nomad/deploymentwatcher/deployments_watcher.go:152 +0x29b created by github.com/hashicorp/nomad/nomad/deploymentwatcher.(Watcher).SetEnabled /opt/gopath/src/github.com/hashicorp/nomad/nomad/deploymentwatcher/deployments_watcher.go:125 +0xf0

goroutine 154 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-memdb.watchFew(0x223f080, 0xc0007964c0, 0xc001197908, 0x20, 0x20, 0x6, 0xf) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-memdb/watch_few.go:16 +0x70a github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-memdb.WatchSet.WatchCtx(0xc0003fdef0, 0x223f080, 0xc0007964c0, 0x0, 0xf) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-memdb/watch.go:82 +0x126 github.com/hashicorp/nomad/nomad/state.(StateStore).BlockingQuery(0xc0003fd8c0, 0xc001197af0, 0xf, 0x223f080, 0xc0007964c0, 0xc0001a5ae0, 0xc0001a5b00, 0x125fd15, 0xc000514960, 0x2035f30) /opt/gopath/src/github.com/hashicorp/nomad/nomad/state/state_store.go:161 +0x16e github.com/hashicorp/nomad/nomad/drainer.(drainingJobWatcher).getJobAllocs(0xc000514900, 0x223f080, 0xc0007964c0, 0xf, 0x2, 0x2, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/drainer/watch_jobs.go:424 +0xb6 github.com/hashicorp/nomad/nomad/drainer.(*drainingJobWatcher).watch(0xc000514900) /opt/gopath/src/github.com/hashicorp/nomad/nomad/drainer/watch_jobs.go:146 +0x10cc created by github.com/hashicorp/nomad/nomad/drainer.NewDrainingJobWatcher /opt/gopath/src/github.com/hashicorp/nomad/nomad/drainer/watch_jobs.go:89 +0x1e3

goroutine 155 [select]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-memdb.watchFew(0x223f080, 0xc000796480, 0xc0011e1b68, 0x20, 0x20, 0x1a9b1c0, 0xc00114a060) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-memdb/watch_few.go:16 +0x70a github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-memdb.WatchSet.WatchCtx(0xc001214300, 0x223f080, 0xc000796480, 0xc001214420, 0x27) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-memdb/watch.go:82 +0x126 github.com/hashicorp/nomad/nomad/state.(StateStore).BlockingQuery(0xc0003fd8c0, 0xc0011e1d50, 0x27, 0x223f080, 0xc000796480, 0xc001144540, 0xc000000001, 0x1fba013, 0x16, 0xc001152b80) /opt/gopath/src/github.com/hashicorp/nomad/nomad/state/state_store.go:161 +0x16e github.com/hashicorp/nomad/nomad/drainer.(nodeDrainWatcher).getNodes(0xc000796500, 0x27, 0x16, 0xc001152b80, 0x2, 0x2) /opt/gopath/src/github.com/hashicorp/nomad/nomad/drainer/watch_nodes.go:217 +0xa7 github.com/hashicorp/nomad/nomad/drainer.(*nodeDrainWatcher).watch(0xc000796500) /opt/gopath/src/github.com/hashicorp/nomad/nomad/drainer/watch_nodes.go:154 +0xa8f created by github.com/hashicorp/nomad/nomad/drainer.NewNodeDrainWatcher /opt/gopath/src/github.com/hashicorp/nomad/nomad/drainer/watch_nodes.go:145 +0x105

goroutine 156 [select, 5 minutes]: github.com/hashicorp/nomad/nomad/drainer.(*deadlineHeap).watch(0xc000796540) /opt/gopath/src/github.com/hashicorp/nomad/nomad/drainer/drain_heap.go:61 +0x18f created by github.com/hashicorp/nomad/nomad/drainer.NewDeadlineHeap /opt/gopath/src/github.com/hashicorp/nomad/nomad/drainer/drain_heap.go:46 +0x115

goroutine 157 [select, 5 minutes]: github.com/hashicorp/nomad/nomad/drainer.(NodeDrainer).run(0xc00039e2d0, 0x223f080, 0xc000796480) /opt/gopath/src/github.com/hashicorp/nomad/nomad/drainer/drainer.go:227 +0x1b4 created by github.com/hashicorp/nomad/nomad/drainer.(NodeDrainer).SetEnabled /opt/gopath/src/github.com/hashicorp/nomad/nomad/drainer/drainer.go:198 +0xe7

goroutine 158 [select, 5 minutes]: github.com/hashicorp/nomad/nomad.(PeriodicDispatch).run(0xc00052c060, 0x223f080, 0xc000796600, 0xc000202720) /opt/gopath/src/github.com/hashicorp/nomad/nomad/periodic.go:335 +0x1b1 created by github.com/hashicorp/nomad/nomad.(PeriodicDispatch).SetEnabled /opt/gopath/src/github.com/hashicorp/nomad/nomad/periodic.go:176 +0x100

goroutine 159 [select]: github.com/hashicorp/nomad/nomad.(Server).schedulePeriodic(0xc000102a00, 0xc00045e180) /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:467 +0x24f created by github.com/hashicorp/nomad/nomad.(Server).establishLeadership /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:241 +0x3d3

goroutine 160 [select]: github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers(0xc0003a7360, 0xc00077df50, 0x1, 0x1, 0xc001207800, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:483 +0x21d github.com/hashicorp/nomad/nomad.(EvalBroker).Dequeue(0xc0003a7360, 0xc00077df50, 0x1, 0x1, 0x3b9aca00, 0x0, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:348 +0x106 github.com/hashicorp/nomad/nomad.(Server).reapFailedEvaluations(0xc000102a00, 0xc00045e180) /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:514 +0xb8 created by github.com/hashicorp/nomad/nomad.(Server).establishLeadership /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:244 +0x405

goroutine 161 [select]: github.com/hashicorp/nomad/nomad.(BlockedEvals).GetDuplicates(0xc0003a7400, 0x3b9aca00, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/nomad/blocked_evals.go:623 +0x177 github.com/hashicorp/nomad/nomad.(Server).reapDupBlockedEvaluations(0xc000102a00, 0xc00045e180) /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:560 +0x86 created by github.com/hashicorp/nomad/nomad.(*Server).establishLeadership /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:247 +0x437

goroutine 162 [select]: github.com/hashicorp/nomad/nomad.(Server).periodicUnblockFailedEvals(0xc000102a00, 0xc00045e180) /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:591 +0xfc created by github.com/hashicorp/nomad/nomad.(Server).establishLeadership /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:250 +0x469

goroutine 163 [select]: github.com/hashicorp/nomad/nomad.(Server).publishJobSummaryMetrics(0xc000102a00, 0xc00045e180) /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:607 +0x110 created by github.com/hashicorp/nomad/nomad.(Server).establishLeadership /opt/gopath/src/github.com/hashicorp/nomad/nomad/leader.go:253 +0x49b

goroutine 293 [select]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).fingerprint(0xc00020e840) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:295 +0x1f9 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run.func1(0xc00020e840, 0xc00016d370) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:162 +0x2b created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:161 +0x179

goroutine 238 [select]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).fingerprint(0xc00020e600) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:295 +0x1f9 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run.func1(0xc00020e600, 0xc00016d2b0) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:162 +0x2b created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:161 +0x179

goroutine 266 [select]: github.com/hashicorp/nomad/client.(*AllocGarbageCollector).Run(0xc000233bc0) /opt/gopath/src/github.com/hashicorp/nomad/client/gc.go:92 +0xf6 created by github.com/hashicorp/nomad/client.NewClient /opt/gopath/src/github.com/hashicorp/nomad/client/client.go:394 +0x1069

goroutine 245 [select]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc001162960) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 261 [semacquire, 5 minutes]: sync.runtime_Semacquire(0xc00016d378) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(WaitGroup).Wait(0xc00016d370) /usr/local/go/src/sync/waitgroup.go:130 +0x64 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run(0xc00020e840) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:174 +0x1ca created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.newInstanceManager /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:132 +0x271

goroutine 224 [IO wait, 5 minutes]: internal/poll.runtime_pollWait(0x7f986c0f9540, 0x72, 0xc0001a14d8) /usr/local/go/src/runtime/netpoll.go:173 +0x66 internal/poll.(pollDesc).wait(0xc000371458, 0x72, 0xffffffffffffff01, 0x22276a0, 0x365f100) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a internal/poll.(pollDesc).waitRead(0xc000371458, 0xc000738001, 0xfdc, 0xfdc) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc000371440, 0xc000738024, 0xfdc, 0xfdc, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x179 os.(File).read(0xc0003d2208, 0xc000738024, 0xfdc, 0xfdc, 0x24, 0x0, 0x507abe) /usr/local/go/src/os/file_unix.go:249 +0x4e os.(File).Read(0xc0003d2208, 0xc000738024, 0xfdc, 0xfdc, 0x4, 0xc0001a1720, 0xc000000300) /usr/local/go/src/os/file.go:108 +0x69 bufio.(Scanner).Scan(0xc0001a1730, 0xc0001a1720) /usr/local/go/src/bufio/scan.go:213 +0xa6 github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(Client).Start.func3(0xc0011f2630, 0xc000588de0, 0x222de40, 0xc0003d2208) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:642 +0x181 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(Client).Start /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:637 +0xdbe

goroutine 145 [semacquire, 5 minutes]: sync.runtime_Semacquire(0xc00016d388) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(WaitGroup).Wait(0xc00016d380) /usr/local/go/src/sync/waitgroup.go:130 +0x64 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run(0xc00020e480) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:174 +0x1ca created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.newInstanceManager /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:132 +0x271

goroutine 258 [semacquire, 5 minutes]: sync.runtime_Semacquire(0xc00016d2b8) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(WaitGroup).Wait(0xc00016d2b0) /usr/local/go/src/sync/waitgroup.go:130 +0x64 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run(0xc00020e600) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:174 +0x1ca created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.newInstanceManager /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:132 +0x271

goroutine 259 [semacquire, 5 minutes]: sync.runtime_Semacquire(0xc0002bd908) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(WaitGroup).Wait(0xc0002bd900) /usr/local/go/src/sync/waitgroup.go:130 +0x64 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run(0xc00020e6c0) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:174 +0x1ca created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.newInstanceManager /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:132 +0x271

goroutine 260 [semacquire, 5 minutes]: sync.runtime_Semacquire(0xc00016d2d8) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(WaitGroup).Wait(0xc00016d2d0) /usr/local/go/src/sync/waitgroup.go:130 +0x64 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run(0xc00020e780) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:174 +0x1ca created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.newInstanceManager /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:132 +0x271

goroutine 205 [select]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).fingerprint(0xc00020e900) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:295 +0x1f9 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run.func1(0xc00020e900, 0xc00049ed00) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:162 +0x2b created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:161 +0x179

goroutine 204 [select]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc0003bd1a0) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 207 [select]: github.com/hashicorp/nomad/drivers/rkt.(Driver).handleFingerprint(0xc0003bd200, 0x223f080, 0xc001174180, 0xc000532300) /opt/gopath/src/github.com/hashicorp/nomad/drivers/rkt/driver.go:258 +0x1a0 created by github.com/hashicorp/nomad/drivers/rkt.(Driver).Fingerprint /opt/gopath/src/github.com/hashicorp/nomad/drivers/rkt/driver.go:250 +0x7f

goroutine 338 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(addrConn).transportMonitor(0xc0000cadc0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/clientconn.go:1060 +0x1c1 github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(addrConn).connect.func1(0xc0000cadc0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/clientconn.go:632 +0x1b2 created by github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(*addrConn).connect /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/clientconn.go:623 +0xe1

goroutine 246 [select]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).fingerprint(0xc00020e6c0) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:295 +0x1f9 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run.func1(0xc00020e6c0, 0xc0002bd900) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:162 +0x2b created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:161 +0x179

goroutine 223 [IO wait, 5 minutes]: internal/poll.runtime_pollWait(0x7f986c0f9470, 0x72, 0xc00055fc58) /usr/local/go/src/runtime/netpoll.go:173 +0x66 internal/poll.(pollDesc).wait(0xc0003715d8, 0x72, 0xffffffffffffff01, 0x22276a0, 0x365f100) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a internal/poll.(pollDesc).waitRead(0xc0003715d8, 0xc0011fb101, 0xeb6, 0xeb6) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc0003715c0, 0xc0011fb14a, 0xeb6, 0xeb6, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x179 os.(File).read(0xc0003d2218, 0xc0011fb14a, 0xeb6, 0xeb6, 0x0, 0x3, 0xc000050a70) /usr/local/go/src/os/file_unix.go:249 +0x4e os.(File).Read(0xc0003d2218, 0xc0011fb14a, 0xeb6, 0xeb6, 0x135e9461, 0xc00055fdb0, 0xc00055fe18) /usr/local/go/src/os/file.go:108 +0x69 bufio.(Scanner).Scan(0xc00055ff38, 0xc00016e2c0) /usr/local/go/src/bufio/scan.go:213 +0xa6 github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(Client).logStderr(0xc0011f2630, 0x2225680, 0xc0003d2218) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:942 +0x156 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(Client).Start /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:631 +0xd32

goroutine 222 [syscall, 5 minutes]: syscall.Syscall6(0xf7, 0x1, 0x29f1, 0xc000757538, 0x1000004, 0x0, 0x0, 0x7ce50c, 0xc000236400, 0x2035f30) /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5 os.(Process).blockUntilWaitable(0xc001244060, 0x2, 0x2, 0xc000757764) /usr/local/go/src/os/wait_waitid.go:31 +0x98 os.(Process).wait(0xc001244060, 0x430283, 0xc000757688, 0xc000757764) /usr/local/go/src/os/exec_unix.go:22 +0x39 os.(Process).Wait(0xc001244060, 0x45d8c0, 0x8, 0xc000757678) /usr/local/go/src/os/exec.go:125 +0x2b os/exec.(Cmd).Wait(0xc0000c89a0, 0x2035f40, 0xc0011f26b0) /usr/local/go/src/os/exec/exec.go:465 +0x5b github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(Client).Start.func2(0xc0011f2630, 0xc0000c89a0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:607 +0xd1 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(Client).Start /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:595 +0xcc9

goroutine 296 [select]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).fingerprint(0xc00020e480) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:295 +0x1f9 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run.func1(0xc00020e480, 0xc00016d380) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:162 +0x2b created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:161 +0x179

goroutine 5927 [select]: github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers.func1(0xc0004982a0, 0xc001157b60, 0xc001157b00) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:469 +0xa0 created by github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:468 +0x17f

goroutine 311 [select, 5 minutes]: github.com/hashicorp/nomad/client.(Client).consulDiscovery(0xc000460fc0) /opt/gopath/src/github.com/hashicorp/nomad/client/client.go:2281 +0xa9 github.com/hashicorp/nomad/client.(Client).consulDiscovery-fm() /opt/gopath/src/github.com/hashicorp/nomad/client/client.go:407 +0x2a github.com/hashicorp/nomad/client.(group).Go.func1(0xc000461128, 0xc000075c50) /opt/gopath/src/github.com/hashicorp/nomad/client/client.go:2745 +0x4f created by github.com/hashicorp/nomad/client.(group).Go /opt/gopath/src/github.com/hashicorp/nomad/client/client.go:2743 +0x62

goroutine 247 [select, 5 minutes]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).handleEvents(0xc00020e6c0) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:425 +0x1f9 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run.func2(0xc00020e6c0, 0xc0002bd900) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:169 +0x2b created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:168 +0x1bc

goroutine 248 [select]: github.com/hashicorp/nomad/drivers/qemu.(Driver).handleFingerprint(0xc000797b00, 0x223f080, 0xc000797b80, 0xc0011568a0) /opt/gopath/src/github.com/hashicorp/nomad/drivers/qemu/driver.go:195 +0x173 created by github.com/hashicorp/nomad/drivers/qemu.(Driver).Fingerprint /opt/gopath/src/github.com/hashicorp/nomad/drivers/qemu/driver.go:188 +0x7f

goroutine 237 [select]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc001145620) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 239 [select, 5 minutes]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).handleEvents(0xc00020e600) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:425 +0x1f9 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run.func2(0xc00020e600, 0xc00016d2b0) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:169 +0x2b created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:168 +0x1bc

goroutine 206 [select, 5 minutes]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).handleEvents(0xc00020e900) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:425 +0x1f9 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run.func2(0xc00020e900, 0xc00049ed00) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:169 +0x2b created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:168 +0x1bc

goroutine 225 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000552d80) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x10f created by github.com/hashicorp/nomad/vendor/google.golang.org/grpc.newCCBalancerWrapper /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x13d

goroutine 290 [select]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).fingerprint(0xc00020e780) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:295 +0x1f9 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run.func1(0xc00020e780, 0xc00016d2d0) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:162 +0x2b created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:161 +0x179

goroutine 339 [IO wait, 5 minutes]: internal/poll.runtime_pollWait(0x7f986c0f96e0, 0x72, 0xc0005b7bb0) /usr/local/go/src/runtime/netpoll.go:173 +0x66 internal/poll.(pollDesc).wait(0xc000136498, 0x72, 0xffffffffffffff00, 0x22276a0, 0x365f100) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a internal/poll.(pollDesc).waitRead(0xc000136498, 0xc0005fe000, 0x8000, 0x8000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc000136480, 0xc0005fe000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x179 net.(netFD).Read(0xc000136480, 0xc0005fe000, 0x8000, 0x8000, 0x30, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc0003d2298, 0xc0005fe000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x68 bufio.(Reader).Read(0xc0003719e0, 0xc000216498, 0x9, 0x9, 0x203000, 0x100000000000000, 0x0) /usr/local/go/src/bufio/bufio.go:216 +0x22f io.ReadAtLeast(0x2222260, 0xc0003719e0, 0xc000216498, 0x9, 0x9, 0x9, 0x18, 0x18, 0xc000448d20) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(0x2222260, 0xc0003719e0, 0xc000216498, 0x9, 0x9, 0x4108af, 0xc0011c410c, 0xc0005b7e48) /usr/local/go/src/io/io.go:329 +0x58 github.com/hashicorp/nomad/vendor/golang.org/x/net/http2.readFrameHeader(0xc000216498, 0x9, 0x9, 0x2222260, 0xc0003719e0, 0x0, 0x7f9800000000, 0xc0004a25c0, 0xc000552e90) /opt/gopath/src/github.com/hashicorp/nomad/vendor/golang.org/x/net/http2/frame.go:237 +0x7b github.com/hashicorp/nomad/vendor/golang.org/x/net/http2.(Framer).ReadFrame(0xc000216460, 0xc0011c4100, 0xc0011c4100, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/golang.org/x/net/http2/frame.go:492 +0xa3 github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport.(http2Client).reader(0xc0004618c0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1193 +0x12e created by github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport/http2_client.go:272 +0xac6

goroutine 241 [select]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc0011457a0) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 291 [select, 5 minutes]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).handleEvents(0xc00020e780) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:425 +0x1f9 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run.func2(0xc00020e780, 0xc00016d2d0) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:169 +0x2b created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:168 +0x1bc

goroutine 292 [select]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc001145920) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 294 [select, 5 minutes]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).handleEvents(0xc00020e840) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:425 +0x1f9 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run.func2(0xc00020e840, 0xc00016d370) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:169 +0x2b created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:168 +0x1bc

goroutine 295 [select]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc001145aa0) /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0x123 created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer /opt/gopath/src/github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0xc6

goroutine 297 [select, 5 minutes]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).handleEvents(0xc00020e480) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:425 +0x1f9 github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(instanceManager).run.func2(0xc00020e480, 0xc00016d380) /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:169 +0x2b created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run /opt/gopath/src/github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:168 +0x1bc

goroutine 298 [select]: github.com/hashicorp/nomad/drivers/exec.(Driver).handleFingerprint(0xc0003a57c0, 0x223f080, 0xc0003dbc80, 0xc00114fb60) /opt/gopath/src/github.com/hashicorp/nomad/drivers/exec/driver.go:198 +0x1a0 created by github.com/hashicorp/nomad/drivers/exec.(Driver).Fingerprint /opt/gopath/src/github.com/hashicorp/nomad/drivers/exec/driver.go:190 +0x7f

goroutine 299 [select]: github.com/hashicorp/nomad/drivers/java.(Driver).handleFingerprint(0xc0003dbb00, 0x223f080, 0xc0003dbcc0, 0xc00114fc80) /opt/gopath/src/github.com/hashicorp/nomad/drivers/java/driver.go:184 +0x173 created by github.com/hashicorp/nomad/drivers/java.(Driver).Fingerprint /opt/gopath/src/github.com/hashicorp/nomad/drivers/java/driver.go:177 +0x7f

goroutine 300 [select]: github.com/hashicorp/nomad/drivers/docker.(Driver).handleFingerprint(0xc00035fd60, 0x223f080, 0xc0003dbd00, 0xc00114fda0) /opt/gopath/src/github.com/hashicorp/nomad/drivers/docker/fingerprint.go:61 +0x1a0 created by github.com/hashicorp/nomad/drivers/docker.(Driver).Fingerprint /opt/gopath/src/github.com/hashicorp/nomad/drivers/docker/fingerprint.go:17 +0x7f

goroutine 301 [select]: github.com/hashicorp/nomad/drivers/rawexec.(Driver).handleFingerprint(0xc0003a58b0, 0x223f080, 0xc0003dbd40, 0xc00114fec0) /opt/gopath/src/github.com/hashicorp/nomad/drivers/rawexec/driver.go:215 +0x188 created by github.com/hashicorp/nomad/drivers/rawexec.(Driver).Fingerprint /opt/gopath/src/github.com/hashicorp/nomad/drivers/rawexec/driver.go:207 +0x7f

goroutine 302 [chan receive, 5 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(Client).Start.func4.1(0xc0011f2630, 0xc000588de0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:655 +0x79 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(Client).Start.func4 /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:653 +0x49

goroutine 303 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(ccResolverWrapper).watcher(0xc000010270) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x142 created by github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(ccResolverWrapper).start /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x3f

goroutine 304 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(gRPCBrokerClientImpl).Recv(0xc000086b80, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/grpc_broker.go:231 +0xb3 github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(GRPCBroker).Run(0xc00036af50) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/grpc_broker.go:411 +0x71 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.newGRPCClient /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/grpc_client.go:56 +0x23e

goroutine 305 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport.(Stream).waitOnHeader(0xc00038e800, 0xc00117a4d8, 0x643d18) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport/transport.go:242 +0xcc github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport.(Stream).RecvCompress(0xc00038e800, 0xc000316240, 0xc001148f00) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport/transport.go:253 +0x2b github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(csAttempt).recvMsg(0xc000237a20, 0x1e23ec0, 0xc0001fa410, 0x223f140, 0xc0000115c0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:741 +0x5c0 github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(clientStream).RecvMsg.func1(0xc000237a20, 0xc0001fa410, 0x50) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:648 +0x3d github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(clientStream).withRetry(0xc000330c60, 0xc0011dfe48, 0xc0011dfe38, 0xc0011dfea0, 0xab754b) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:530 +0x281 github.com/hashicorp/nomad/vendor/google.golang.org/grpc.(clientStream).RecvMsg(0xc000330c60, 0x1e23ec0, 0xc0001fa410, 0x20, 0xc000544600) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:647 +0x8d github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/internal/proto.(gRPCBrokerStartStreamClient).Recv(0xc0004a2610, 0x2032ad0, 0xc00009a9c0, 0xc000086b80) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/internal/proto/grpc_broker.pb.go:149 +0x62 github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(gRPCBrokerClientImpl).StartStream(0xc000086b80, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/grpc_broker.go:194 +0x19f created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.newGRPCClient /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/grpc_client.go:57 +0x260

goroutine 340 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport.(controlBuffer).get(0xc000552e80, 0x1, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:321 +0x103 github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport.(loopyWriter).run(0xc000371c20, 0x0, 0x0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:439 +0x1ac github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0004618c0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport/http2_client.go:312 +0x7b created by github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/internal/transport/http2_client.go:310 +0xc25

goroutine 341 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/google.golang.org/grpc.newClientStream.func5(0xc000348000, 0xc000330c60, 0x223f140, 0xc0000115c0) /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:287 +0xd7 created by github.com/hashicorp/nomad/vendor/google.golang.org/grpc.newClientStream /opt/gopath/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:286 +0x842

goroutine 342 [select, 5 minutes]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(gRPCBrokerClientImpl).StartStream.func1(0xc00009a9c0, 0xc000086b80, 0x224cd60, 0xc0004a2610) /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/grpc_broker.go:181 +0x146 created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(gRPCBrokerClientImpl).StartStream /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/grpc_broker.go:179 +0x189

goroutine 5928 [select]: github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers.func1(0xc000498300, 0xc001157b60, 0xc001157b00) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:469 +0xa0 created by github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:468 +0x17f

goroutine 5912 [select]: github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers.func1(0xc000498240, 0xc0011579e0, 0xc001157980) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:469 +0xa0 created by github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:468 +0x17f

goroutine 5926 [select]: github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers.func1(0xc000498240, 0xc001157b60, 0xc001157b00) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:469 +0xa0 created by github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:468 +0x17f

goroutine 5925 [select]: github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers.func1(0xc00049c120, 0xc001157aa0, 0xc001157a40) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:469 +0xa0 created by github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:468 +0x17f

goroutine 5913 [select]: github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers.func1(0xc0004982a0, 0xc0011579e0, 0xc001157980) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:469 +0xa0 created by github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:468 +0x17f

goroutine 5914 [select]: github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers.func1(0xc000498300, 0xc0011579e0, 0xc001157980) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:469 +0xa0 created by github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:468 +0x17f

goroutine 5915 [select]: github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers.func1(0xc000498360, 0xc0011579e0, 0xc001157980) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:469 +0xa0 created by github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:468 +0x17f

goroutine 5929 [select]: github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers.func1(0xc000498360, 0xc001157b60, 0xc001157b00) /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:469 +0xa0 created by github.com/hashicorp/nomad/nomad.(EvalBroker).waitForSchedulers /opt/gopath/src/github.com/hashicorp/nomad/nomad/eval_broker.go:468 +0x17f

rax 0xfffffffffffffffc rbx 0x3b997ae9 rcx 0x463213 rdx 0x0 rdi 0x3763e00 rsi 0x80 rbp 0xc0002c76e0 rsp 0xc0002c7698 r8 0x0 r9 0x0 r10 0xc0002c76d0 r11 0x202 r12 0xb3c24841498 r13 0x3 r14 0xc00049c240 r15 0xc001211d58 rip 0x463213 rflags 0x202 cs 0x33 fs 0x0 gs 0x0

I can only recover nomad i've i stop it, stop and remove the docker container and kill all remaining nomad processes:

nomad agent -config=nomad.hcl.example ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation. ==> Loaded configuration from nomad.hcl.example ==> Starting Nomad agent... ==> Nomad agent configuration:

   Advertise Addrs: HTTP: 192.168.200.16:4646; RPC: 192.168.200.16:4647; Serf: 192.168.200.16:4648
        Bind Addrs: HTTP: 0.0.0.0:4646; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
            Client: true
         Log Level: debug
            Region: global (DC: dc1)
            Server: true
           Version: 0.9.0

==> Nomad agent started! Log data will stream in below:

2019-04-14T12:21:34.906+0200 [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/plugins
2019-04-14T12:21:34.906+0200 [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/plugins
2019-04-14T12:21:34.906+0200 [INFO ] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
2019-04-14T12:21:34.906+0200 [INFO ] agent: detected plugin: name=rkt type=driver plugin_version=0.1.0
2019-04-14T12:21:34.906+0200 [INFO ] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
2019-04-14T12:21:34.906+0200 [INFO ] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
2019-04-14T12:21:34.906+0200 [INFO ] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
2019-04-14T12:21:34.906+0200 [INFO ] agent: detected plugin: name=java type=driver plugin_version=0.1.0
2019-04-14T12:21:34.906+0200 [INFO ] agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
2019-04-14T12:21:34.915+0200 [INFO ] nomad: raft: Initial configuration (index=1): [{Suffrage:Voter ID:192.168.200.16:4647 Address:192.168.200.16:4647}]
2019-04-14T12:21:34.916+0200 [INFO ] nomad: raft: Node at 192.168.200.16:4647 [Follower] entering Follower state (Leader: "")
2019-04-14T12:21:34.923+0200 [INFO ] nomad: serf: EventMemberJoin: cassandra.global 192.168.200.16
2019-04-14T12:21:34.924+0200 [INFO ] nomad: starting scheduling worker(s): num_workers=6 schedulers="[system service batch _core]"
2019-04-14T12:21:34.924+0200 [WARN ] nomad: serf: Failed to re-join any previously known node
2019-04-14T12:21:34.924+0200 [INFO ] client: using state directory: state_dir=/var/lib/nomad/client
2019-04-14T12:21:34.924+0200 [INFO ] nomad: adding server: server="cassandra.global (Addr: 192.168.200.16:4647) (DC: dc1)"
2019-04-14T12:21:34.924+0200 [DEBUG] nomad: lost contact with Nomad quorum, falling back to Consul for server list
2019-04-14T12:21:34.924+0200 [INFO ] client: using alloc directory: alloc_dir=/var/lib/nomad/alloc
2019-04-14T12:21:34.925+0200 [ERROR] nomad: error looking up Nomad servers in Consul: error="server.nomad: unable to query Consul datacenters: Get http://127.0.0.1:8500/v1/catalog/datacenters: dial tcp 127.0.0.1:8500: connect: connection refused"
2019-04-14T12:21:34.927+0200 [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters="[arch cgroup consul cpu host memory network nomad signal storage vault env_aws env_gce]"
2019-04-14T12:21:34.927+0200 [INFO ] client.fingerprint_mgr.cgroup: cgroups are available
2019-04-14T12:21:34.927+0200 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=cgroup period=15s
2019-04-14T12:21:34.927+0200 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul period=15s
2019-04-14T12:21:34.928+0200 [DEBUG] client.fingerprint_mgr.cpu: detected cpu frequency: MHz=2207
2019-04-14T12:21:34.928+0200 [DEBUG] client.fingerprint_mgr.cpu: detected core count: cores=6
2019-04-14T12:21:34.933+0200 [WARN ] client.fingerprint_mgr.network: unable to parse speed: path=/usr/bin/ethtool device=lan0
2019-04-14T12:21:34.933+0200 [DEBUG] client.fingerprint_mgr.network: unable to parse link speed: path=/sys/class/net/lan0/speed
2019-04-14T12:21:34.933+0200 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected and no speed specified by user, falling back to default speed: mbits=1000
2019-04-14T12:21:34.933+0200 [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=lan0 IP=192.168.200.16
2019-04-14T12:21:34.940+0200 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault period=15s
2019-04-14T12:21:36.907+0200 [WARN ] nomad: raft: Heartbeat timeout from "" reached, starting election
2019-04-14T12:21:36.907+0200 [INFO ] nomad: raft: Node at 192.168.200.16:4647 [Candidate] entering Candidate state in term 6
2019-04-14T12:21:36.921+0200 [DEBUG] nomad: raft: Votes needed: 1
2019-04-14T12:21:36.921+0200 [DEBUG] nomad: raft: Vote granted from 192.168.200.16:4647 in term 6. Tally: 1
2019-04-14T12:21:36.921+0200 [INFO ] nomad: raft: Election won. Tally: 1
2019-04-14T12:21:36.921+0200 [INFO ] nomad: raft: Node at 192.168.200.16:4647 [Leader] entering Leader state
2019-04-14T12:21:36.921+0200 [INFO ] nomad: cluster leadership acquired
2019-04-14T12:21:36.933+0200 [DEBUG] nomad.fsm.service_sched: reconciled current state with desired state: eval_id=37b52f9b-ede3-8d84-9ff1-134b698b091a job_id=test namespace=default results="Total changes: (place 0) (destructive 0) (inplace 0) (stop 0)

Desired Changes for "test": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 1) (canary 0)" 2019-04-14T12:21:36.933+0200 [DEBUG] nomad.fsm.service_sched: setting eval status: eval_id=37b52f9b-ede3-8d84-9ff1-134b698b091a job_id=test namespace=default status=complete 2019-04-14T12:21:36.936+0200 [DEBUG] nomad.fsm.service_sched: reconciled current state with desired state: eval_id=8e73cf3e-edc3-fd45-1c55-d6f569a04341 job_id=test namespace=default results="Total changes: (place 0) (destructive 0) (inplace 0) (stop 0) Desired Changes for "test": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 1) (canary 0)" 2019-04-14T12:21:36.936+0200 [DEBUG] nomad.fsm.service_sched: setting eval status: eval_id=8e73cf3e-edc3-fd45-1c55-d6f569a04341 job_id=test namespace=default status=complete 2019-04-14T12:21:36.939+0200 [DEBUG] nomad.fsm.service_sched: reconciled current state with desired state: eval_id=8b9ab8a5-e6ff-fd83-2ca0-53a9d827c68c job_id=test namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 0) Desired Changes for "test": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)" 2019-04-14T12:21:36.939+0200 [DEBUG] nomad.fsm.service_sched: failed to place all allocations, blocked eval created: eval_id=8b9ab8a5-e6ff-fd83-2ca0-53a9d827c68c job_id=test namespace=default blocked_eval_id=f29a86ae-0031-3ab2-6c9f-d3fa4ef558ba 2019-04-14T12:21:36.939+0200 [DEBUG] nomad.fsm.service_sched: setting eval status: eval_id=8b9ab8a5-e6ff-fd83-2ca0-53a9d827c68c job_id=test namespace=default status=complete 2019-04-14T12:21:36.939+0200 [DEBUG] nomad: leader reconciling job summaries: index=46 2019-04-14T12:21:36.941+0200 [DEBUG] client.fingerprint_mgr.env_aws: error querying AWS Metadata URL, skipping 2019-04-14T12:21:36.945+0200 [DEBUG] nomad.fsm.service_sched: reconciled current state with desired state: eval_id=54acf91f-e127-4992-ecff-1bb24a6c83c3 job_id=test namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 0) Desired Changes for "test": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)" 2019-04-14T12:21:36.945+0200 [DEBUG] nomad.fsm.service_sched: failed to place all allocations, blocked eval created: eval_id=54acf91f-e127-4992-ecff-1bb24a6c83c3 job_id=test namespace=default blocked_eval_id=50753089-f13b-3ce7-8755-fdc8d2cdc102 2019-04-14T12:21:36.945+0200 [DEBUG] nomad.fsm.service_sched: setting eval status: eval_id=54acf91f-e127-4992-ecff-1bb24a6c83c3 job_id=test namespace=default status=complete 2019-04-14T12:21:38.942+0200 [DEBUG] client.fingerprint_mgr.env_gce: could not read value for attribute: attribute=machine-type error="Get http://169.254.169.254/computeMetadata/v1/instance/machine-type: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" 2019-04-14T12:21:38.942+0200 [DEBUG] client.fingerprint_mgr.env_gce: error querying GCE Metadata URL, skipping 2019-04-14T12:21:38.942+0200 [DEBUG] client.fingerprint_mgr: detected fingerprints: node_attrs="[arch cgroup cpu host network nomad signal storage]" 2019-04-14T12:21:38.942+0200 [INFO ] client.plugin: starting plugin manager: plugin-type=driver 2019-04-14T12:21:38.942+0200 [INFO ] client.plugin: starting plugin manager: plugin-type=device 2019-04-14T12:21:38.943+0200 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=driver 2019-04-14T12:21:38.943+0200 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=device 2019-04-14T12:21:38.948+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=qemu health=undetected description= 2019-04-14T12:21:38.948+0200 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=device 2019-04-14T12:21:38.948+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=exec health=healthy description=Healthy 2019-04-14T12:21:38.949+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=raw_exec health=undetected description=disabled 2019-04-14T12:21:38.949+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=rkt health=undetected description="Failed to execute rkt version: exec: "rkt": executable file not found in $PATH" 2019-04-14T12:21:38.959+0200 [ERROR] client.alloc_runner.task_runner: error recovering task; cleaning up: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 task=test error="failed to inspect container for id "35eede160c63deff0c813f6030599cd4cd301d5066c77f50e4ef5942d05e32b8": No such container: 35eede160c63deff0c813f6030599cd4cd301d5066c77f50e4ef5942d05e32b8" task_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29/test/e15bb1a5 2019-04-14T12:21:38.959+0200 [INFO ] client: started client: node_id=cad4d630-9775-9c83-7218-b78913955043 2019-04-14T12:21:38.965+0200 [ERROR] client.alloc_runner.task_runner.task_hook: failed to launch logmon process: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 task=test error="Reattachment process not found" 2019-04-14T12:21:38.969+0200 [INFO ] client: node registration complete 2019-04-14T12:21:38.970+0200 [DEBUG] client: updated allocations: index=42 total=1 pulled=1 filtered=0 2019-04-14T12:21:38.971+0200 [DEBUG] client: allocation updates: added=0 removed=0 updated=1 ignored=0 2019-04-14T12:21:38.972+0200 [ERROR] client.alloc_runner.task_runner: prestart failed: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 task=test error="prestart hook "logmon" failed: Reattachment process not found" 2019-04-14T12:21:38.972+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 task=test reason="Restart within policy" delay=15.633393259s 2019-04-14T12:21:38.976+0200 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=1 ignored=0 errors=0 2019-04-14T12:21:38.978+0200 [DEBUG] worker: dequeued evaluation: eval_id=a5b17764-64b7-0298-f2be-f614aaeb720c 2019-04-14T12:21:38.978+0200 [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=a5b17764-64b7-0298-f2be-f614aaeb720c job_id=test namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 0) Desired Changes for "test": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)" 2019-04-14T12:21:38.978+0200 [DEBUG] worker: reblocked evaluation: eval="<Eval "a5b17764-64b7-0298-f2be-f614aaeb720c" JobID: "test" Namespace: "default">" 2019-04-14T12:21:38.978+0200 [DEBUG] worker: ack evaluation: eval_id=a5b17764-64b7-0298-f2be-f614aaeb720c 2019-04-14T12:21:38.983+0200 [DEBUG] worker: dequeued evaluation: eval_id=ab064868-b44d-c14f-4b50-bda97834f824 2019-04-14T12:21:38.984+0200 [DEBUG] client: evaluations triggered by node update: num_evals=1 2019-04-14T12:21:38.984+0200 [DEBUG] client: state updated: node_status=ready 2019-04-14T12:21:38.984+0200 [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=ab064868-b44d-c14f-4b50-bda97834f824 job_id=test namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 0) Desired Changes for "test": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)" 2019-04-14T12:21:38.987+0200 [DEBUG] client.alloc_runner.task_runner: task run loop exiting: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 task=test 2019-04-14T12:21:38.988+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=bfb2fea6-c9e3-d8e8-8818-a497df8c8b29 2019-04-14T12:21:38.991+0200 [DEBUG] worker: created evaluation: eval="<Eval "d5d9747c-5bbe-1c36-c60e-3bf7a5f38ad6" JobID: "test" Namespace: "default">" 2019-04-14T12:21:38.991+0200 [DEBUG] worker.service_sched: failed to place all allocations, blocked eval created: eval_id=ab064868-b44d-c14f-4b50-bda97834f824 job_id=test namespace=default blocked_eval_id=d5d9747c-5bbe-1c36-c60e-3bf7a5f38ad6 2019-04-14T12:21:38.991+0200 [DEBUG] worker.service_sched: setting eval status: eval_id=ab064868-b44d-c14f-4b50-bda97834f824 job_id=test namespace=default status=complete 2019-04-14T12:21:38.996+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=docker health=healthy description=Healthy 2019-04-14T12:21:39.000+0200 [DEBUG] worker: updated evaluation: eval="<Eval "ab064868-b44d-c14f-4b50-bda97834f824" JobID: "test" Namespace: "default">" 2019-04-14T12:21:39.000+0200 [DEBUG] worker: ack evaluation: eval_id=ab064868-b44d-c14f-4b50-bda97834f824 2019-04-14T12:21:39.044+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=java health=healthy description=Healthy 2019-04-14T12:21:39.044+0200 [DEBUG] client.driver_mgr: detected drivers: drivers="[exec docker java]" 2019-04-14T12:21:39.044+0200 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=driver 2019-04-14T12:21:39.227+0200 [DEBUG] client: updated allocations: index=56 total=1 pulled=0 filtered=1 2019-04-14T12:21:39.227+0200 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1 2019-04-14T12:21:39.227+0200 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0 2019-04-14T12:21:44.151+0200 [DEBUG] client: state changed, updating node and re-registering 2019-04-14T12:21:44.159+0200 [DEBUG] worker: dequeued evaluation: eval_id=d5d9747c-5bbe-1c36-c60e-3bf7a5f38ad6 2019-04-14T12:21:44.159+0200 [INFO ] client: node registration complete 2019-04-14T12:21:44.160+0200 [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=d5d9747c-5bbe-1c36-c60e-3bf7a5f38ad6 job_id=test namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 0) Desired Changes for "test": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)" 2019-04-14T12:21:44.167+0200 [DEBUG] worker: submitted plan for evaluation: eval_id=d5d9747c-5bbe-1c36-c60e-3bf7a5f38ad6 2019-04-14T12:21:44.167+0200 [DEBUG] worker.service_sched: setting eval status: eval_id=d5d9747c-5bbe-1c36-c60e-3bf7a5f38ad6 job_id=test namespace=default status=complete 2019-04-14T12:21:44.168+0200 [DEBUG] client: updated allocations: index=58 total=2 pulled=1 filtered=1 2019-04-14T12:21:44.168+0200 [DEBUG] client: allocation updates: added=1 removed=0 updated=0 ignored=1 2019-04-14T12:21:44.171+0200 [DEBUG] worker: updated evaluation: eval="<Eval "d5d9747c-5bbe-1c36-c60e-3bf7a5f38ad6" JobID: "test" Namespace: "default">" 2019-04-14T12:21:44.171+0200 [DEBUG] worker: ack evaluation: eval_id=d5d9747c-5bbe-1c36-c60e-3bf7a5f38ad6 2019-04-14T12:21:44.173+0200 [DEBUG] client: allocation updates applied: added=1 removed=0 updated=0 ignored=1 errors=0 2019-04-14T12:21:44.186+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=d05edc8e-e541-c320-6b33-227a4c4d539c task=test path=/usr/bin/nomad args="[/usr/bin/nomad logmon]" 2019-04-14T12:21:44.186+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=d05edc8e-e541-c320-6b33-227a4c4d539c task=test path=/usr/bin/nomad pid=12027 2019-04-14T12:21:44.186+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=d05edc8e-e541-c320-6b33-227a4c4d539c task=test path=/usr/bin/nomad 2019-04-14T12:21:44.214+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=d05edc8e-e541-c320-6b33-227a4c4d539c task=test @module=logmon address=/tmp/plugin612254164 network=unix timestamp=2019-04-14T12:21:44.214+0200 2019-04-14T12:21:44.214+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=d05edc8e-e541-c320-6b33-227a4c4d539c task=test version=2 2019-04-14T12:21:44.215+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=d05edc8e-e541-c320-6b33-227a4c4d539c task=test path=/var/lib/nomad/alloc/d05edc8e-e541-c320-6b33-227a4c4d539c/alloc/logs/.test.stdout.fifo @module=logmon timestamp=2019-04-14T12:21:44.215+0200 2019-04-14T12:21:44.215+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=d05edc8e-e541-c320-6b33-227a4c4d539c task=test @module=logmon path=/var/lib/nomad/alloc/d05edc8e-e541-c320-6b33-227a4c4d539c/alloc/logs/.test.stderr.fifo timestamp=2019-04-14T12:21:44.215+0200 2019-04-14T12:21:44.225+0200 [DEBUG] client.driver_mgr.docker: image reference count incremented: driver=docker image_name=redis:3.2 image_id=sha256:87856cc39862cec77541d68382e4867d7ccb29a85a17221446c857ddaebca916 references=1 2019-04-14T12:21:44.225+0200 [DEBUG] client.driver_mgr.docker: configured resources: driver=docker task_name=test memory=314572800 cpu_shares=100 cpu_quota=0 cpu_period=0 2019-04-14T12:21:44.225+0200 [DEBUG] client.driver_mgr.docker: binding directories: driver=docker task_name=test binds="[]string{"/var/lib/nomad/alloc/d05edc8e-e541-c320-6b33-227a4c4d539c/alloc:/alloc", "/var/lib/nomad/alloc/d05edc8e-e541-c320-6b33-227a4c4d539c/test/local:/local", "/var/lib/nomad/alloc/d05edc8e-e541-c320-6b33-227a4c4d539c/test/secrets:/secrets"}" 2019-04-14T12:21:44.225+0200 [DEBUG] client.driver_mgr.docker: networking mode not specified; using default: driver=docker task_name=test network_mode=bridge 2019-04-14T12:21:44.225+0200 [DEBUG] client.driver_mgr.docker: no network interfaces are available: driver=docker task_name=test 2019-04-14T12:21:44.225+0200 [DEBUG] client.driver_mgr.docker: setting container name: driver=docker task_name=test container_name=test-d05edc8e-e541-c320-6b33-227a4c4d539c 2019-04-14T12:21:44.263+0200 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=1341065b7cf6b74d684e3d59088bc12b62c567209c88a634719d71a91b1fe3e7 2019-04-14T12:21:44.418+0200 [DEBUG] client: updated allocations: index=60 total=2 pulled=0 filtered=2 2019-04-14T12:21:44.418+0200 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=2 2019-04-14T12:21:44.418+0200 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=2 errors=0 2019-04-14T12:21:44.890+0200 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=1341065b7cf6b74d684e3d59088bc12b62c567209c88a634719d71a91b1fe3e7 2019-04-14T12:21:44.890+0200 [DEBUG] client.driver_mgr.docker.docker_logger: starting plugin: driver=docker path=/usr/bin/nomad args="[/usr/bin/nomad docker_logger]" 2019-04-14T12:21:44.890+0200 [DEBUG] client.driver_mgr.docker.docker_logger: plugin started: driver=docker path=/usr/bin/nomad pid=12111 2019-04-14T12:21:44.890+0200 [DEBUG] client.driver_mgr.docker.docker_logger: waiting for RPC address: driver=docker path=/usr/bin/nomad 2019-04-14T12:21:44.933+0200 [DEBUG] client.driver_mgr.docker.docker_logger: using plugin: driver=docker version=2 2019-04-14T12:21:44.933+0200 [DEBUG] client.driver_mgr.docker.docker_logger.nomad: plugin address: driver=docker network=unix @module=docker_logger address=/tmp/plugin268989702 timestamp=2019-04-14T12:21:44.930+0200 2019-04-14T12:21:44.934+0200 [DEBUG] client.driver_mgr.docker.docker_logger.nomad: using client connection initialized from environment: driver=docker @module=docker_logger timestamp=2019-04-14T12:21:44.933+0200 2019-04-14T12:21:45.022+0200 [DEBUG] client: updated allocations: index=61 total=2 pulled=0 filtered=2 2019-04-14T12:21:45.022+0200 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=2 2019-04-14T12:21:45.022+0200 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=2 errors=0 2019-04-14T12:21:55.022+0200 [DEBUG] client: updated allocations: index=62 total=2 pulled=0 filtered=2 2019-04-14T12:21:55.022+0200 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=2 2019-04-14T12:21:55.022+0200 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=2 errors=0

[MorphBonehunter]

Maybe an addition: i know that in the deploy guide is mentioned to use KillMode=process with systemd. But also if this is used and someone/something killed the additonal nomad processes the startup stucks, so treat my systemd stuff above just as an example.

[schmichael]

Despite this issue coming first, closing in favor of #5566. Thanks @MorphBonehunter!

[MorphBonehunter]

No Problem ;) Thanks.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.