search

hashicorp / nomad-autoscaler

Nomad Autoscaler brings autoscaling to your Nomad workloads.
Mozilla Public License 2.0
420 stars 85 forks source link

Nomad Autoscaler Idle forever with context.canceled error #711

Open bernardoVale opened 10 months ago

bernardoVale commented 10 months ago

I couldn't find a way to reproduce this error, but sometimes, autoscaler gets stuck and returns context canceled to all policy evaluations:

2023-09-19T18:08:24.247Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=af2adcdd-ffc6-8c54-e71d-dbbf55543a77 error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:24.349Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=9418971a-8261-cbd7-7730-5f8748f455d0 error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:24.447Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=04db11b8-6242-c03d-cb93-010c35e35285 error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:24.551Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=3e2681aa-68d2-41ae-f4e0-cd1dcd0b8ba3 error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:24.848Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=3c10001b-0402-743b-5bfd-73577527d9ac error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:24.948Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=236fcef8-6a4f-be5d-ae61-fd56eabfe92f error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:25.148Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=fa573b85-9ab6-952c-8deb-611a59f64d32 error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:25.249Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=8bf225c1-5a57-7ce1-d13e-ce63f7de45de error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:25.345Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=8ea0ab80-bbaa-35b0-7a50-69bc38a719fb error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:25.348Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=ee9ea733-6563-d=c3767a43-77c9-d8a4-3101-1bd6a6cb6c8f error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:26.248Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=8ff31dca-fce8-0624-3669-2839a768752e error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:26.348Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=df860074-418c-fb31-d2c4-e970b1f95d39 error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:52.520Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=32725454-2121-9bea-29b5-f81405eeb3e2 error="rpc error: code = Canceled desc = context canceled"

Each eval loop, it prints one context canceled error per policy. Once it gets into this state, only a restart fixes it.

Here's the SIGABRT dump:

SIGABRT: abort

PC=0x46d261 m=0 sigcode=0

goroutine 0 [idle]:

runtime.futex()

    runtime/sys_linux_amd64.s:519 +0x21

runtime.futexsleep(0xc000052000, 0x57000, 0x1970b5f)

    runtime/os_linux.go:44 +0x36

runtime.notesleep(0x2b6ef30)

    runtime/lock_futex.go:160 +0x87

runtime.mPark()

    runtime/proc.go:1441 +0x2a

runtime.stopm()

    runtime/proc.go:2408 +0x78

runtime.findrunnable()

    runtime/proc.go:2984 +0x865

runtime.schedule()

    runtime/proc.go:3367 +0x239

runtime.goexit0(0xc00099c9c0)

    runtime/proc.go:3691 +0x21e

runtime.mcall()

    runtime/asm_amd64.s:307 +0x43

goroutine 1 [chan receive, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/agent.(*Agent).handleSignals(0xc0004299d0)

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:270 +0xe5

github.com/hashicorp/nomad-autoscaler/agent.(*Agent).Run(0xc0004299d0)

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:100 +0x446

github.com/hashicorp/nomad-autoscaler/command.(*AgentCommand).Run(0xc0002a7860, {0xc0001a8020, 0x203000, 0x0})

    github.com/hashicorp/nomad-autoscaler/command/agent.go:333 +0x8e5

github.com/mitchellh/cli.(*CLI).Run(0xc0001e9b80)

    github.com/mitchellh/cli@v1.1.2/cli.go:262 +0x5f8

main.main()

    github.com/hashicorp/nomad-autoscaler/main.go:26 +0x24d

goroutine 5 [select]:

go.opencensus.io/stats/view.(*worker).start(0xc000380380)

    go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xb9

created by go.opencensus.io/stats/view.init.0

    go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x92

goroutine 7 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8d2e33d8, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc00020cd00, 0x4172e6, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Accept(0xc00020cd00)

    internal/poll/fd_unix.go:402 +0x22c

net.(*netFD).accept(0xc00020cd00)

    net/fd_unix.go:173 +0x35

net.(*TCPListener).accept(0xc00000c9d8)

    net/tcpsock_posix.go:140 +0x28

net.(*TCPListener).Accept(0xc00000c9d8)

    net/tcpsock.go:262 +0x3d

net/http.(*Server).Serve(0xc0003ece00, {0x1e39d40, 0xc00000c9d8})

    net/http/server.go:3002 +0x394

github.com/hashicorp/nomad-autoscaler/agent/http.(*Server).Start(0xc0001b1b80)

    github.com/hashicorp/nomad-autoscaler/agent/http/server.go:124 +0xd2

created by github.com/hashicorp/nomad-autoscaler/command.(*AgentCommand).Run

    github.com/hashicorp/nomad-autoscaler/command/agent.go:330 +0x896

goroutine 8 [IO wait, 28822 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e3108, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc0005bf7a0, 0xc000640000, 0x1)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc0005bf7a0, {0xc000640000, 0x10000, 0x10000})

    internal/poll/fd_unix.go:167 +0x25a

os.(*File).read(...)

    os/file_posix.go:32

os.(*File).Read(0xc0001a2ae0, {0xc000640000, 0x10000c00063bde8, 0x7f4c8d23cd40})

    os/file.go:119 +0x5e

bufio.(*Reader).fill(0xc00063bf40)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).ReadSlice(0xc00063bf40, 0x0)

    bufio/bufio.go:360 +0x2f

bufio.(*Reader).ReadLine(0xc00063bf40)

    bufio/bufio.go:389 +0x27

github.com/hashicorp/go-plugin.(*Client).logStderr(0xc0003e2960, {0x1e17260, 0xc0001a2ae0})

    github.com/hashicorp/go-plugin@v1.0.1/client.go:956 +0x2aa

created by github.com/hashicorp/go-plugin.(*Client).Start

    github.com/hashicorp/go-plugin@v1.0.1/client.go:601 +0x14af

goroutine 9 [semacquire, 28822 minutes]:

sync.runtime_Semacquire(0x0)

    runtime/sema.go:56 +0x25

sync.(*WaitGroup).Wait(0x0)

    sync/waitgroup.go:130 +0x71

github.com/hashicorp/go-plugin.(*Client).Start.func2()

    github.com/hashicorp/go-plugin@v1.0.1/client.go:617 +0xc8

created by github.com/hashicorp/go-plugin.(*Client).Start

    github.com/hashicorp/go-plugin@v1.0.1/client.go:604 +0x152f

goroutine 10 [IO wait, 28822 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e32e8, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc0005bf6e0, 0xc000256025, 0x1)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc0005bf6e0, {0xc000256025, 0xfdb, 0xfdb})

    internal/poll/fd_unix.go:167 +0x25a

os.(*File).read(...)

    os/file_posix.go:32

os.(*File).Read(0xc0001a2ad0, {0xc000256025, 0xc00060eea8, 0x40592d})

    os/file.go:119 +0x5e

bufio.(*Scanner).Scan(0xc00060ef40)

    bufio/scan.go:215 +0x865

github.com/hashicorp/go-plugin.(*Client).Start.func3()

    github.com/hashicorp/go-plugin@v1.0.1/client.go:650 +0x12f

created by github.com/hashicorp/go-plugin.(*Client).Start

    github.com/hashicorp/go-plugin@v1.0.1/client.go:645 +0x15f4

goroutine 97 [chan receive, 28822 minutes]:

github.com/hashicorp/go-plugin.(*Client).Start.func4.1()

    github.com/hashicorp/go-plugin@v1.0.1/client.go:663 +0x7f

created by github.com/hashicorp/go-plugin.(*Client).Start.func4

    github.com/hashicorp/go-plugin@v1.0.1/client.go:661 +0x85

goroutine 98 [select, 28822 minutes]:

google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0006d6080)

    google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:112 +0x79

created by google.golang.org/grpc.newCCBalancerWrapper

    google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:73 +0x22f

goroutine 32 [select, 28822 minutes]:

google.golang.org/grpc.newClientStreamWithParams.func4()

    google.golang.org/grpc@v1.46.0/stream.go:341 +0x98

created by google.golang.org/grpc.newClientStreamWithParams

    google.golang.org/grpc@v1.46.0/stream.go:340 +0xb65

goroutine 100 [select, 28822 minutes]:

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).Recv(0x0)

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:231 +0x6d

github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0xc0006d0140)

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:411 +0x48

created by github.com/hashicorp/go-plugin.newGRPCClient

    github.com/hashicorp/go-plugin@v1.0.1/grpc_client.go:62 +0x287

goroutine 101 [select, 28822 minutes]:

google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc000316240)

    google.golang.org/grpc@v1.46.0/internal/transport/transport.go:324 +0x85

google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)

    google.golang.org/grpc@v1.46.0/internal/transport/transport.go:339

google.golang.org/grpc.(*csAttempt).recvMsg(0xc000318000, {0x1aab320, 0xc000310140}, 0x8)

    google.golang.org/grpc@v1.46.0/stream.go:969 +0xbb

google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x203000)

    google.golang.org/grpc@v1.46.0/stream.go:834 +0x25

google.golang.org/grpc.(*clientStream).withRetry(0xc000316000, 0xc000713e90, 0xc000713e60)

    google.golang.org/grpc@v1.46.0/stream.go:688 +0x2f6

google.golang.org/grpc.(*clientStream).RecvMsg(0xc000316000, {0x1aab320, 0xc000310140})

    google.golang.org/grpc@v1.46.0/stream.go:833 +0x11f

github.com/hashicorp/go-plugin/internal/plugin.(*gRPCBrokerStartStreamClient).Recv(0xc00018b940)

    github.com/hashicorp/go-plugin@v1.0.1/internal/plugin/grpc_broker.pb.go:149 +0x4c

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream(0xc0006d6200)

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:194 +0x1dd

created by github.com/hashicorp/go-plugin.newGRPCClient

    github.com/hashicorp/go-plugin@v1.0.1/grpc_client.go:63 +0x2c9

goroutine 11 [IO wait, 23 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e3018, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc00020cf00, 0xc0005e0000, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc00020cf00, {0xc0005e0000, 0x8000, 0x8000})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc00020cf00, {0xc0005e0000, 0x60100000000, 0x8})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc0001a2af0, {0xc0005e0000, 0x4693ae, 0x439c47})

    net/net.go:183 +0x45

bufio.(*Reader).Read(0xc0005bfa40, {0xc0003ed000, 0x9, 0xc00062dd18})

    bufio/bufio.go:227 +0x1b4

io.ReadAtLeast({0x1e13d00, 0xc0005bfa40}, {0xc0003ed000, 0x9, 0x9}, 0x9)

    io/io.go:328 +0x9a

io.ReadFull(...)

    io/io.go:347

golang.org/x/net/http2.readFrameHeader({0xc0003ed000, 0x9, 0xc00062dd6f}, {0x1e13d00, 0xc0005bfa40})

    golang.org/x/net@v0.0.0-20220425223048-2871e0cb64e4/http2/frame.go:237 +0x6e

golang.org/x/net/http2.(*Framer).ReadFrame(0xc0003ecfc0)

    golang.org/x/net@v0.0.0-20220425223048-2871e0cb64e4/http2/frame.go:498 +0x95

google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00000a1e0)

    google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:1498 +0x41f

created by google.golang.org/grpc/internal/transport.newHTTP2Client

    google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:365 +0x194f

goroutine 12 [select, 23 minutes]:

google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0005de0a0, 0x1)

    google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:407 +0x11b

google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0005bfbc0)

    google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:534 +0x85

google.golang.org/grpc/internal/transport.newHTTP2Client.func3()

    google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:415 +0x65

created by google.golang.org/grpc/internal/transport.newHTTP2Client

    google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:413 +0x1fa5

goroutine 113 [select, 28822 minutes]:

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream.func1()

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:181 +0x10e

created by github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:179 +0x17a

goroutine 129 [IO wait, 2414 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e2e38, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc0007321e0, 0xc000010000, 0x1)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc0007321e0, {0xc000010000, 0x10000, 0x10000})

    internal/poll/fd_unix.go:167 +0x25a

os.(*File).read(...)

    os/file_posix.go:32

os.(*File).Read(0xc00071c050, {0xc000010000, 0xc00299ade8, 0x4bc716})

    os/file.go:119 +0x5e

bufio.(*Reader).fill(0xc00299af40)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).ReadSlice(0xc00299af40, 0xb0)

    bufio/bufio.go:360 +0x2f

bufio.(*Reader).ReadLine(0xc00299af40)

    bufio/bufio.go:389 +0x27

github.com/hashicorp/go-plugin.(*Client).logStderr(0xc00075e000, {0x1e17260, 0xc00071c050})

    github.com/hashicorp/go-plugin@v1.0.1/client.go:956 +0x2aa

created by github.com/hashicorp/go-plugin.(*Client).Start

    github.com/hashicorp/go-plugin@v1.0.1/client.go:601 +0x14af

goroutine 130 [semacquire, 28822 minutes]:

sync.runtime_Semacquire(0xc000619f20)

    runtime/sema.go:56 +0x25

sync.(*WaitGroup).Wait(0x1)

    sync/waitgroup.go:130 +0x71

github.com/hashicorp/go-plugin.(*Client).Start.func2()

    github.com/hashicorp/go-plugin@v1.0.1/client.go:617 +0xc8

created by github.com/hashicorp/go-plugin.(*Client).Start

    github.com/hashicorp/go-plugin@v1.0.1/client.go:604 +0x152f

goroutine 131 [IO wait, 28822 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e31f8, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000732120, 0xc000252025, 0x1)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000732120, {0xc000252025, 0xfdb, 0xfdb})

    internal/poll/fd_unix.go:167 +0x25a

os.(*File).read(...)

    os/file_posix.go:32

os.(*File).Read(0xc00071c040, {0xc000252025, 0xc0007796a8, 0x40592d})

    os/file.go:119 +0x5e

bufio.(*Scanner).Scan(0xc000779740)

    bufio/scan.go:215 +0x865

github.com/hashicorp/go-plugin.(*Client).Start.func3()

    github.com/hashicorp/go-plugin@v1.0.1/client.go:650 +0x12f

created by github.com/hashicorp/go-plugin.(*Client).Start

    github.com/hashicorp/go-plugin@v1.0.1/client.go:645 +0x15f4

goroutine 82 [chan receive, 28822 minutes]:

github.com/hashicorp/go-plugin.(*Client).Start.func4.1()

    github.com/hashicorp/go-plugin@v1.0.1/client.go:663 +0x7f

created by github.com/hashicorp/go-plugin.(*Client).Start.func4

    github.com/hashicorp/go-plugin@v1.0.1/client.go:661 +0x85

goroutine 83 [select, 28822 minutes]:

google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00063e140)

    google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:112 +0x79

created by google.golang.org/grpc.newCCBalancerWrapper

    google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:73 +0x22f

goroutine 86 [select, 28822 minutes]:

google.golang.org/grpc.newClientStreamWithParams.func4()

    google.golang.org/grpc@v1.46.0/stream.go:341 +0x98

created by google.golang.org/grpc.newClientStreamWithParams

    google.golang.org/grpc@v1.46.0/stream.go:340 +0xb65

goroutine 14 [select, 28822 minutes]:

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).Recv(0x0)

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:231 +0x6d

github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0xc0005de230)

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:411 +0x48

created by github.com/hashicorp/go-plugin.newGRPCClient

    github.com/hashicorp/go-plugin@v1.0.1/grpc_client.go:62 +0x287

goroutine 15 [select, 28822 minutes]:

google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc00002e120)

    google.golang.org/grpc@v1.46.0/internal/transport/transport.go:324 +0x85

google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)

    google.golang.org/grpc@v1.46.0/internal/transport/transport.go:339

google.golang.org/grpc.(*csAttempt).recvMsg(0xc0003858c0, {0x1aab320, 0xc0000f24b0}, 0x8)

    google.golang.org/grpc@v1.46.0/stream.go:969 +0xbb

google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x203000)

    google.golang.org/grpc@v1.46.0/stream.go:834 +0x25

google.golang.org/grpc.(*clientStream).withRetry(0xc0005d06c0, 0xc00070fe90, 0xc00070fe60)

    google.golang.org/grpc@v1.46.0/stream.go:688 +0x2f6

google.golang.org/grpc.(*clientStream).RecvMsg(0xc0005d06c0, {0x1aab320, 0xc0000f24b0})

    google.golang.org/grpc@v1.46.0/stream.go:833 +0x11f

github.com/hashicorp/go-plugin/internal/plugin.(*gRPCBrokerStartStreamClient).Recv(0xc000634800)

    github.com/hashicorp/go-plugin@v1.0.1/internal/plugin/grpc_broker.pb.go:149 +0x4c

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream(0xc0005f8180)

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:194 +0x1dd

created by github.com/hashicorp/go-plugin.newGRPCClient

    github.com/hashicorp/go-plugin@v1.0.1/grpc_client.go:63 +0x2c9

goroutine 84 [IO wait, 13 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e2d48, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000380200, 0xc000144000, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000380200, {0xc000144000, 0x8000, 0x8000})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000380200, {0xc000144000, 0x60100000000, 0x8})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc000632068, {0xc000144000, 0x4693ae, 0x439c47})

    net/net.go:183 +0x45

bufio.(*Reader).Read(0xc000650300, {0xc00002a040, 0x9, 0xc0005d6d18})

    bufio/bufio.go:227 +0x1b4

io.ReadAtLeast({0x1e13d00, 0xc000650300}, {0xc00002a040, 0x9, 0x9}, 0x9)

    io/io.go:328 +0x9a

io.ReadFull(...)

    io/io.go:347

golang.org/x/net/http2.readFrameHeader({0xc00002a040, 0x9, 0xc0005d6d6f}, {0x1e13d00, 0xc000650300})

    golang.org/x/net@v0.0.0-20220425223048-2871e0cb64e4/http2/frame.go:237 +0x6e

golang.org/x/net/http2.(*Framer).ReadFrame(0xc00002a000)

    golang.org/x/net@v0.0.0-20220425223048-2871e0cb64e4/http2/frame.go:498 +0x95

google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00002c000)

    google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:1498 +0x41f

created by google.golang.org/grpc/internal/transport.newHTTP2Client

    google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:365 +0x194f

goroutine 85 [select, 13 minutes]:

google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000f2370, 0x1)

    google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:407 +0x11b

google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000650420)

    google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:534 +0x85

google.golang.org/grpc/internal/transport.newHTTP2Client.func3()

    google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:415 +0x65

created by google.golang.org/grpc/internal/transport.newHTTP2Client

    google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:413 +0x1fa5

goroutine 87 [select, 28822 minutes]:

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream.func1()

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:181 +0x10e

created by github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:179 +0x17a

goroutine 162 [select, 2338 minutes]:

google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0005f8500)

    google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:112 +0x79

created by google.golang.org/grpc.newCCBalancerWrapper

    google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:73 +0x22f

goroutine 170 [chan receive]:

github.com/hashicorp/nomad-autoscaler/plugins/builtin/target/nomad/plugin.(*TargetPlugin).garbageCollectionLoop(0xc0005fccc0)

    github.com/hashicorp/nomad-autoscaler/plugins/builtin/target/nomad/plugin/plugin.go:213 +0xc9

created by github.com/hashicorp/nomad-autoscaler/plugins/builtin/target/nomad/plugin.(*TargetPlugin).SetConfig

    github.com/hashicorp/nomad-autoscaler/plugins/builtin/target/nomad/plugin/plugin.go:94 +0xef

goroutine 88 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8d2e2898, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000312540, 0xc00033e000, 0x1)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000312540, {0xc00033e000, 0x10000, 0x10000})

    internal/poll/fd_unix.go:167 +0x25a

os.(*File).read(...)

    os/file_posix.go:32

os.(*File).Read(0xc00000e048, {0xc00033e000, 0x10000c0009cbde8, 0x7f4c8d07d5c0})

    os/file.go:119 +0x5e

bufio.(*Reader).fill(0xc0009cbf40)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).ReadSlice(0xc0009cbf40, 0x70)

    bufio/bufio.go:360 +0x2f

bufio.(*Reader).ReadLine(0xc0009cbf40)

    bufio/bufio.go:389 +0x27

github.com/hashicorp/go-plugin.(*Client).logStderr(0xc00042a3c0, {0x1e17260, 0xc00000e048})

    github.com/hashicorp/go-plugin@v1.0.1/client.go:956 +0x2aa

created by github.com/hashicorp/go-plugin.(*Client).Start

    github.com/hashicorp/go-plugin@v1.0.1/client.go:601 +0x14af

goroutine 89 [semacquire, 28822 minutes]:

sync.runtime_Semacquire(0x0)

    runtime/sema.go:56 +0x25

sync.(*WaitGroup).Wait(0x0)

    sync/waitgroup.go:130 +0x71

github.com/hashicorp/go-plugin.(*Client).Start.func2()

    github.com/hashicorp/go-plugin@v1.0.1/client.go:617 +0xc8

created by github.com/hashicorp/go-plugin.(*Client).Start

    github.com/hashicorp/go-plugin@v1.0.1/client.go:604 +0x152f

goroutine 90 [IO wait, 28822 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e2c58, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000312480, 0xc000026025, 0x1)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000312480, {0xc000026025, 0xfdb, 0xfdb})

    internal/poll/fd_unix.go:167 +0x25a

os.(*File).read(...)

    os/file_posix.go:32

os.(*File).Read(0xc00000e038, {0xc000026025, 0xc000718ea8, 0x40592d})

    os/file.go:119 +0x5e

bufio.(*Scanner).Scan(0xc000718f40)

    bufio/scan.go:215 +0x865

github.com/hashicorp/go-plugin.(*Client).Start.func3()

    github.com/hashicorp/go-plugin@v1.0.1/client.go:650 +0x12f

created by github.com/hashicorp/go-plugin.(*Client).Start

    github.com/hashicorp/go-plugin@v1.0.1/client.go:645 +0x15f4

goroutine 114 [chan receive, 28822 minutes]:

github.com/hashicorp/go-plugin.(*Client).Start.func4.1()

    github.com/hashicorp/go-plugin@v1.0.1/client.go:663 +0x7f

created by github.com/hashicorp/go-plugin.(*Client).Start.func4

    github.com/hashicorp/go-plugin@v1.0.1/client.go:661 +0x85

goroutine 115 [select, 28822 minutes]:

google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00063e580)

    google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:112 +0x79

created by google.golang.org/grpc.newCCBalancerWrapper

    google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:73 +0x22f

goroutine 149 [syscall, 28822 minutes]:

os/signal.signal_recv()

    runtime/sigqueue.go:169 +0x98

os/signal.loop()

    os/signal/signal_unix.go:24 +0x19

created by os/signal.Notify.func1.1

    os/signal/signal.go:151 +0x2c

goroutine 117 [select, 28822 minutes]:

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).Recv(0x0)

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:231 +0x6d

github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0xc0000f2870)

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:411 +0x48

created by github.com/hashicorp/go-plugin.newGRPCClient

    github.com/hashicorp/go-plugin@v1.0.1/grpc_client.go:62 +0x287

goroutine 118 [select, 28822 minutes]:

google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc00002e7e0)

    google.golang.org/grpc@v1.46.0/internal/transport/transport.go:324 +0x85

google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)

    google.golang.org/grpc@v1.46.0/internal/transport/transport.go:339

google.golang.org/grpc.(*csAttempt).recvMsg(0xc000384a50, {0x1aab320, 0xc0003104b0}, 0x8)

    google.golang.org/grpc@v1.46.0/stream.go:969 +0xbb

google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x203000)

    google.golang.org/grpc@v1.46.0/stream.go:834 +0x25

google.golang.org/grpc.(*clientStream).withRetry(0xc0006d2900, 0xc0001bfe90, 0xc0001bfe60)

    google.golang.org/grpc@v1.46.0/stream.go:688 +0x2f6

google.golang.org/grpc.(*clientStream).RecvMsg(0xc0006d2900, {0x1aab320, 0xc0003104b0})

    google.golang.org/grpc@v1.46.0/stream.go:833 +0x11f

github.com/hashicorp/go-plugin/internal/plugin.(*gRPCBrokerStartStreamClient).Recv(0xc0000eca40)

    github.com/hashicorp/go-plugin@v1.0.1/internal/plugin/grpc_broker.pb.go:149 +0x4c

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream(0xc00063e6c0)

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:194 +0x1dd

created by github.com/hashicorp/go-plugin.newGRPCClient

    github.com/hashicorp/go-plugin@v1.0.1/grpc_client.go:63 +0x2c9

goroutine 91 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8d2e27a8, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc0006d8400, 0xc0002d6000, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc0006d8400, {0xc0002d6000, 0x8000, 0x8000})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc0006d8400, {0xc0002d6000, 0x1060100000000, 0x8})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc00000e0f0, {0xc0002d6000, 0x97b8b0, 0x1})

    net/net.go:183 +0x45

bufio.(*Reader).Read(0xc000312720, {0xc00075c200, 0x9, 0x18})

    bufio/bufio.go:227 +0x1b4

io.ReadAtLeast({0x1e13d00, 0xc000312720}, {0xc00075c200, 0x9, 0x9}, 0x9)

    io/io.go:328 +0x9a

io.ReadFull(...)

    io/io.go:347

golang.org/x/net/http2.readFrameHeader({0xc00075c200, 0x9, 0x624d25f18c414}, {0x1e13d00, 0xc000312720})

    golang.org/x/net@v0.0.0-20220425223048-2871e0cb64e4/http2/frame.go:237 +0x6e

golang.org/x/net/http2.(*Framer).ReadFrame(0xc00075c1c0)

    golang.org/x/net@v0.0.0-20220425223048-2871e0cb64e4/http2/frame.go:498 +0x95

google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00002c1e0)

    google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:1498 +0x41f

created by google.golang.org/grpc/internal/transport.newHTTP2Client

    google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:365 +0x194f

goroutine 92 [select]:

google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000310370, 0x1)

    google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:407 +0x11b

google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0003128a0)

    google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:534 +0x85

google.golang.org/grpc/internal/transport.newHTTP2Client.func3()

    google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:415 +0x65

created by google.golang.org/grpc/internal/transport.newHTTP2Client

    google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:413 +0x1fa5

goroutine 93 [select, 28822 minutes]:

google.golang.org/grpc.newClientStreamWithParams.func4()

    google.golang.org/grpc@v1.46.0/stream.go:341 +0x98

created by google.golang.org/grpc.newClientStreamWithParams

    google.golang.org/grpc@v1.46.0/stream.go:340 +0xb65

goroutine 94 [select, 28822 minutes]:

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream.func1()

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:181 +0x10e

created by github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream

    github.com/hashicorp/go-plugin@v1.0.1/grpc_broker.go:179 +0x17a

goroutine 150 [select, 28822 minutes]:

github.com/armon/go-metrics.(*InmemSignal).run(0xc0005f8c40)

    github.com/armon/go-metrics@v0.3.11/inmem_signal.go:64 +0x6c

created by github.com/armon/go-metrics.NewInmemSignal

    github.com/armon/go-metrics@v0.3.11/inmem_signal.go:38 +0x174

goroutine 152 [sleep]:

time.Sleep(0x3b9aca00)

    runtime/time.go:193 +0x12e

github.com/armon/go-metrics.(*Metrics).collectStats(0xc0001fa2d0)

    github.com/armon/go-metrics@v0.3.11/metrics.go:234 +0x25

created by github.com/armon/go-metrics.New

    github.com/armon/go-metrics@v0.3.11/start.go:84 +0x17b

goroutine 154 [select]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e640, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f

goroutine 155 [select]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e690, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f

goroutine 156 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e6e0, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f

goroutine 157 [select, 5 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e730, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f

goroutine 158 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e780, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f

goroutine 159 [select]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e7d0, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f

goroutine 160 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e820, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f

goroutine 177 [select]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e870, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f

goroutine 178 [select, 5 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e8c0, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f

goroutine 179 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e910, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f

goroutine 180 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e960, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b

goroutine 181 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e9b0, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b

goroutine 182 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089ea00, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b

goroutine 183 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089ea50, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b

goroutine 184 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089eaa0, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b

goroutine 185 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089eaf0, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b

goroutine 186 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089eb40, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b

goroutine 187 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089eb90, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b

goroutine 188 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089ebe0, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b

goroutine 189 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

    github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089ec30, {0x1e4cfb0, 0xc0001a7cc0})

    github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b

goroutine 190 [select]:

github.com/hashicorp/nomad-autoscaler/agent.(*Agent).runEvalHandler(0xc0004299d0, {0x1e4cfb0, 0xc0001a7cc0}, 0xc0008b4720)

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:106 +0xb2

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).Run

    github.com/hashicorp/nomad-autoscaler/agent/agent.go:97 +0x439

goroutine 4504214 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc002c98240)

    net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

    net/http/transport.go:1748 +0x1e65

goroutine 4543502 [select]:

net/http.(*persistConn).roundTrip(0xc001970000, 0xc0022e0340)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc0007cf400)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc0007cf400, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x417901, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc0007cf400, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc0007cf400)

    net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

    net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc000b91e40)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x24, {0xc000575b80, 0xc000b6e4c0}, {0x1879000, 0xc0001b9340}, 0xc13a9754473257de)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc00102e7b0, {0xc0022d1200, 0x24}, 0xc001fc0480)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367

goroutine 4504291 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8ce2f2d8, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000380180, 0xc002416600, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000380180, {0xc002416600, 0x109c, 0x109c})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000380180, {0xc002416600, 0xc002416605, 0x28c})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc000a20008, {0xc002416600, 0x6, 0xc0019c87f8})

    net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc000630048, {0xc002416600, 0x0, 0x409f2d})

    crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc000192278, {0x1e13f00, 0xc000630048})

    bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc000192000, {0x1e16e20, 0xc000a20008}, 0x109c)

    crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc000192000, 0x0)

    crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

    crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc000192000, {0xc002458000, 0x1000, 0x0})

    crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc00072e000, {0xc002458000, 0xc002642960, 0xc0019c8d30})

    net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc002e830e0)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc002e830e0, 0x1)

    bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc00072e000)

    net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

    net/http/transport.go:1747 +0x1e05

goroutine 4504310 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc0006d26c0)

    net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

    net/http/transport.go:1748 +0x1e65

goroutine 4543255 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc0020da000, 0xc0015281c0)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc0009aa500)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc0009aa500, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc0009aa500, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc0009aa500)

    net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

    net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc001644d10)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0xc001986748, {0xc00110ed40, 0x2ba0fe0}, {0x1879000, 0xc000bba770}, 0xc001838058)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc0019867b0, {0xc0022d0d50, 0x24}, 0xc001fc39a0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367

goroutine 4414570 [IO wait, 3 minutes]:

internal/poll.runtime_pollWait(0x7f4c8cf06cf0, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000296180, 0xc002310000, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000296180, {0xc002310000, 0x109c, 0x109c})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000296180, {0xc002310000, 0xc002310005, 0x267})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc001724000, {0xc002310000, 0x6, 0xc001f4a7f8})

    net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0020ee0d8, {0xc002310000, 0x0, 0x409f2d})

    crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc001730278, {0x1e13f00, 0xc0020ee0d8})

    bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc001730000, {0x1e16e20, 0xc001724000}, 0x109c)

    crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc001730000, 0x0)

    crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

    crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc001730000, {0xc002338000, 0x1000, 0x0})

    crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc0020da000, {0xc002338000, 0xc000314c00, 0xc001f4ad30})

    net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc002b8cf00)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc002b8cf00, 0x1)

    bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc0020da000)

    net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

    net/http/transport.go:1747 +0x1e05

goroutine 1552064 [select]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc001789580}, {{0xc0022d0c60, 0x24}, 0xc000b311a0, 0xc000b312c0, 0xc000b31140})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 1553913 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc000b80c80}, {{0xc0022d0e10, 0x24}, 0xc002546780, 0xc0025468a0, 0xc002546720})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 1554082 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorIDs(0xc0008b0400, {0x1e4cfb0, 0xc000c31440}, {0xc002075140, 0xc0020750e0})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:113 +0x26d

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:67 +0x1f3

goroutine 4542337 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc000ca2120)

    net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

    net/http/transport.go:1748 +0x1e65

goroutine 1554093 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002450780, {0x1e4cfb0, 0xc0001a7cc0}, 0xc0008c6b00)

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0e10, 0x24})

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3

goroutine 4543253 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc001970120, 0xc001332180)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc00118dc00)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc00118dc00, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc00118dc00, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc00118dc00)

    net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

    net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc0001616b0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc0010d4d80, 0x62493e4ec443e}, {0x1879000, 0xc00067c1c0}, 0x62493e4ec443e)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc0011457b0, {0xc0022d0e40, 0x24}, 0x0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367

goroutine 1554162 [select]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc0022c2300}, {{0xc0022d0f30, 0x24}, 0xc0025472c0, 0xc0025473e0, 0xc002547260})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 1554091 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002450640, {0x1e4cfb0, 0xc0001a7cc0}, 0xc001980840)

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0db0, 0x24})

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3

goroutine 1553755 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc001111040}, {{0xc0022d0de0, 0x24}, 0xc0025465a0, 0xc0025466c0, 0xc002546540})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 1554055 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc002503680}, {{0xc0022d0d50, 0x24}, 0xc000b31b60, 0xc000b31c80, 0xc000b31b00})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 1553845 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc00063fe80}, {{0xc0022d0f90, 0x24}, 0xc002547680, 0xc0025477a0, 0xc002547620})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 1553923 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc0004abe40}, {{0xc0022d10b0, 0x24}, 0xc001793a40, 0xc001793b60, 0xc0017939e0})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 4504279 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8ce2fe18, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000528080, 0xc002304c00, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000528080, {0xc002304c00, 0x109c, 0x109c})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000528080, {0xc002304c00, 0xc002304c05, 0x279})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc001100008, {0xc002304c00, 0x6, 0xc0026107f8})

    net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0018ca078, {0xc002304c00, 0x0, 0x409f2d})

    crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc000130cf8, {0x1e13f00, 0xc0018ca078})

    bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc000130a80, {0x1e16e20, 0xc001100008}, 0x109c)

    crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc000130a80, 0x0)

    crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

    crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc000130a80, {0xc0024d6000, 0x1000, 0x0})

    crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc0006d2240, {0xc0024d6000, 0xc001f94d20, 0xc002610d30})

    net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc0005c5da0)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc0005c5da0, 0x1)

    bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc0006d2240)

    net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

    net/http/transport.go:1747 +0x1e05

goroutine 1554092 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc0024506e0, {0x1e4cfb0, 0xc0001a7cc0}, 0x1e4cfb0)

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0de0, 0x24})

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3

goroutine 4504216 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc001970120)

    net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

    net/http/transport.go:1748 +0x1e65

goroutine 1554084 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc0024501e0, {0x1e4cfb0, 0xc0001a7cc0}, 0x1e4cfb0)

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0c60, 0x24})

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3

goroutine 4214783 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8ce2f788, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000f18300, 0xc002300000, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000f18300, {0xc002300000, 0x109c, 0x109c})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000f18300, {0xc002300000, 0xc002300005, 0x26b})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc0008c8020, {0xc002300000, 0x6, 0xc001e267f8})

    net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc00000c300, {0xc002300000, 0x0, 0x409f2d})

    crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc0018b6278, {0x1e13f00, 0xc00000c300})

    bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc0018b6000, {0x1e16e20, 0xc0008c8020}, 0x109c)

    crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc0018b6000, 0x0)

    crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

    crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc0018b6000, {0xc002494000, 0x1000, 0x0})

    crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc002b20000, {0xc002494000, 0xc00112c240, 0xc001e26d30})

    net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc0004ce6c0)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc0004ce6c0, 0x1)

    bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc002b20000)

    net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

    net/http/transport.go:1747 +0x1e05

goroutine 4542326 [IO wait, 3 minutes]:

internal/poll.runtime_pollWait(0x7f4c8ce2fa58, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000e12400, 0xc002483900, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000e12400, {0xc002483900, 0x109c, 0x109c})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000e12400, {0xc002483900, 0xc002483905, 0x26c})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc00091a0a0, {0xc002483900, 0x6, 0xc0029967f8})

    net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0013b40a8, {0xc002483900, 0x0, 0x409f2d})

    crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc0016365f8, {0x1e13f00, 0xc0013b40a8})

    bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc001636380, {0x1e16e20, 0xc00091a0a0}, 0x109c)

    crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc001636380, 0x0)

    crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

    crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc001636380, {0xc000af6000, 0x1000, 0x0})

    crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc00093e000, {0xc000af6000, 0xc000181d40, 0xc002996d30})

    net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc0002b9a40)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc0002b9a40, 0x1)

    bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc00093e000)

    net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

    net/http/transport.go:1747 +0x1e05

goroutine 4543171 [select, 5 minutes]:

net/http.(*persistConn).roundTrip(0xc0006ec360, 0xc000b6e140)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc000170400)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc000170400, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc000170400, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc000170400)

    net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

    net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc0014ac840)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc000d92880, 0x62470f845a44d}, {0x1879000, 0xc00067c0e0}, 0x62470f845a44d)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc001a71fb0, {0xc0022d10e0, 0x24}, 0xc0006d2120)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367

goroutine 1554098 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002450b40, {0x1e4cfb0, 0xc0001a7cc0}, 0xc000319290)

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0f00, 0x24})

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3

goroutine 1554130 [select]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc000915a40}, {{0xc0022d0e70, 0x24}, 0xc002546b40, 0xc002546c60, 0xc002546ae0})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 4542313 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc0005f8c00}, {{0xc0013fd200, 0x24}, 0xc0010da480, 0xc0010da600, 0xc0010da420})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 1554097 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002450aa0, {0x1e4cfb0, 0xc0001a7cc0}, 0xc0015246e0)

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0ed0, 0x24})

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3

goroutine 1554087 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc0024503c0, {0x1e4cfb0, 0xc0001a7cc0}, 0xc001860210)

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0cf0, 0x24})

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3

goroutine 1554112 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002451400, {0x1e4cfb0, 0xc0001a7cc0}, 0xc000161290)

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d11a0, 0x24})

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3

goroutine 4543338 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc002c98120, 0xc00063e440)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc0007c5300)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc0007c5300, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc0007c5300, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc0007c5300)

    net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

    net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc000918bb0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x0, {0xc0015850c0, 0x14c8942}, {0x1879000, 0xc000589570}, 0x624afd586b04d)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc0033b9fb0, {0xc0022d1050, 0x24}, 0xc002e8a220)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367

goroutine 4543219 [select, 5 minutes]:

net/http.(*persistConn).roundTrip(0xc002b20120, 0xc002502300)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc000302b00)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc000302b00, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc000302b00, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc000302b00)

    net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

    net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc00113a9a0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc00188cfc0, 0x62493ba612df8}, {0x1879000, 0xc00049a4d0}, 0x62493ba612df8)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc0018187b0, {0xc0022d0db0, 0x24}, 0x0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367

goroutine 1553869 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc000735b40}, {{0xc0022d0e40, 0x24}, 0xc002546960, 0xc002546a80, 0xc002546900})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 4272239 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8d2e2a78, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000544300, 0xc002414000, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000544300, {0xc002414000, 0x109c, 0x109c})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000544300, {0xc002414000, 0xc002414005, 0x266})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc000112060, {0xc002414000, 0x6, 0xc0019c47f8})

    net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc000132078, {0xc002414000, 0x0, 0x409f2d})

    crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc000532278, {0x1e13f00, 0xc000132078})

    bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc000532000, {0x1e16e20, 0xc000112060}, 0x109c)

    crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc000532000, 0x0)

    crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

    crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc000532000, {0xc00243a000, 0x1000, 0x0})

    crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc0005d05a0, {0xc00243a000, 0xc00112c2a0, 0xc0019c4d30})

    net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc000650ea0)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc000650ea0, 0x1)

    bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc0005d05a0)

    net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

    net/http/transport.go:1747 +0x1e05

goroutine 4542385 [IO wait, 3 minutes]:

internal/poll.runtime_pollWait(0x7f4c8cf06b10, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000f18280, 0xc002314c00, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000f18280, {0xc002314c00, 0x109c, 0x109c})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000f18280, {0xc002314c00, 0xc002314c05, 0x58a})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc00071c060, {0xc002314c00, 0xc000217500, 0xc0003b47f8})

    net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0008ae090, {0xc002314c00, 0x0, 0x409f2d})

    crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc0014813f8, {0x1e13f00, 0xc0008ae090})

    bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc001481180, {0x1e16e20, 0xc00071c060}, 0x109c)

    crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc001481180, 0x0)

    crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

    crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc001481180, {0xc000af9000, 0x1000, 0x0})

    crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc002a00240, {0xc000af9000, 0xc000426de0, 0xc0003b4d30})

    net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc002e83380)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc002e83380, 0x1)

    bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc002a00240)

    net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

    net/http/transport.go:1747 +0x1e05

goroutine 4487210 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc00216e000)

    net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

    net/http/transport.go:1748 +0x1e65

goroutine 4543367 [select]:

net/http.(*persistConn).roundTrip(0xc0005d05a0, 0xc000b80140)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc000c98200)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc000c98200, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc000c98200, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc000c98200)

    net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

    net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc00113ab00)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0xc00176f748, {0xc00110e3c0, 0x14c8942}, {0x1879000, 0xc0003fe0e0}, 0xc000536118)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc00176f7b0, {0xc0022d0f00, 0x24}, 0xc002e8a220)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367

goroutine 1553949 [select]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc0017d7700}, {{0xc0022d0f00, 0x24}, 0xc0025470e0, 0xc002547200, 0xc002547080})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 4543307 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc00093e000, 0xc000c30300)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc000bdf700)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc000bdf700, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc000bdf700, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc000bdf700)

    net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

    net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc0006ee2c0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc00147cac0, 0x62485ec8ab0d8}, {0x1879000, 0xc0002b4070}, 0x62485ec8ab0d8)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc000604fb0, {0xc0013fced0, 0x24}, 0x0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367

goroutine 1554106 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002451040, {0x1e4cfb0, 0xc0001a7cc0}, 0xc000d99000)

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d1080, 0x24})

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3

goroutine 1553840 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc0022c2240}, {{0xc0022d0d80, 0x24}, 0xc000b31d40, 0xc000b31e60, 0xc000b31ce0})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 4504293 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8cf05a30, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc0004d0100, 0xc002417900, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc0004d0100, {0xc002417900, 0x109c, 0x109c})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc0004d0100, {0xc002417900, 0xc002417905, 0x269})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc00051c010, {0xc002417900, 0x6, 0xc0024617f8})

    net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc002736048, {0xc002417900, 0x0, 0x409f2d})

    crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc0003b1078, {0x1e13f00, 0xc002736048})

    bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc0003b0e00, {0x1e16e20, 0xc00051c010}, 0x109c)

    crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc0003b0e00, 0x0)

    crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

    crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc0003b0e00, {0xc002474000, 0x1000, 0x0})

    crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc001970000, {0xc002474000, 0xc002642ba0, 0xc002461d30})

    net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc0027c3e60)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc0027c3e60, 0x1)

    bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc001970000)

    net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

    net/http/transport.go:1747 +0x1e05

goroutine 4543353 [select]:

net/http.(*persistConn).roundTrip(0xc002a00120, 0xc001332140)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc0009aa000)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc0009aa000, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc0009aa000, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc0009aa000)

    net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

    net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc0001b5ef0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc001512640, 0x620faa667a5f8}, {0x1879000, 0xc0004449a0}, 0x620faa667a5f8)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc001022fb0, {0xc0022d1110, 0x24}, 0xc0025b2000)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367

goroutine 4543292 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc002a00000, 0xc000914180)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc002052200)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc002052200, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc002052200, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc002052200)

    net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

    net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc001fc6370)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc00188c6c0, 0x62493ba612df8}, {0x1879000, 0xc0005928c0}, 0x62493ba612df8)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc0018187b0, {0xc0022d10b0, 0x24}, 0x0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367

goroutine 4272240 [select]:

net/http.(*persistConn).writeLoop(0xc0005d05a0)

    net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

    net/http/transport.go:1748 +0x1e65

goroutine 4504030 [IO wait, 3 minutes]:

internal/poll.runtime_pollWait(0x7f4c8ce2f5a8, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc002e92180, 0xc00082a000, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc002e92180, {0xc00082a000, 0x109c, 0x109c})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc002e92180, {0xc00082a000, 0xc00082a005, 0x27d})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc0009ac0b8, {0xc00082a000, 0x6, 0xc0020217f8})

    net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0018ca060, {0xc00082a000, 0x0, 0x409f2d})

    crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc0013a6278, {0x1e13f00, 0xc0018ca060})

    bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc0013a6000, {0x1e16e20, 0xc0009ac0b8}, 0x109c)

    crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc0013a6000, 0x0)

    crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

    crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc0013a6000, {0xc001009000, 0x1000, 0x0})

    crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc0006d2120, {0xc001009000, 0xc000181200, 0xc002021d30})

    net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc0020c4720)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc0020c4720, 0x1)

    bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc0006d2120)

    net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

    net/http/transport.go:1747 +0x1e05

goroutine 4504313 [IO wait, 3 minutes]:

internal/poll.runtime_pollWait(0x7f4c8cf05760, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc002228780, 0xc002279300, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc002228780, {0xc002279300, 0x109c, 0x109c})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc002228780, {0xc002279300, 0xc002279305, 0x272})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc001082030, {0xc002279300, 0x6, 0xc00260c7f8})

    net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0014020a8, {0xc002279300, 0x0, 0x409f2d})

    crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc000c22278, {0x1e13f00, 0xc0014020a8})

    bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc000c22000, {0x1e16e20, 0xc001082030}, 0x109c)

    crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc000c22000, 0x0)

    crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

    crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc000c22000, {0xc002624000, 0x1000, 0x0})

    crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc002c98360, {0xc002624000, 0xc0010e4660, 0xc00260cd30})

    net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc002155da0)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc002155da0, 0x1)

    bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc002c98360)

    net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

    net/http/transport.go:1747 +0x1e05

goroutine 1553915 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc000b80d80}, {{0xc0022d0f60, 0x24}, 0xc0025474a0, 0xc0025475c0, 0xc002547440})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 4504312 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc0006d2000)

    net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

    net/http/transport.go:1748 +0x1e65

goroutine 1553968 [select, 19 minutes]:

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run(0xc00089e5a0, {0x1e4cfb0, 0xc0001a7cc0}, 0xc0008b4720)

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:72 +0x475

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:154 +0x847

goroutine 4487206 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc0001ce480)

    net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

    net/http/transport.go:1748 +0x1e65

goroutine 4543304 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc0006d2120, 0xc0010de180)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc000ac1a00)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc000ac1a00, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc000ac1a00, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc000ac1a00)

    net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

    net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc0003193f0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc0010d4bc0, 0x62485ec8ab0d8}, {0x1879000, 0xc000428690}, 0x62485ec8ab0d8)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc000604fb0, {0xc0022d1080, 0x24}, 0x0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367

goroutine 1554086 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002450320, {0x1e4cfb0, 0xc0001a7cc0}, 0xc0008c6b00)

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0cc0, 0x24})

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3

goroutine 3878095 [IO wait, 3 minutes]:

internal/poll.runtime_pollWait(0x7f4c8cf061b0, 0x72)

    runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc00075a300, 0xc002284000, 0x0)

    internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

    internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc00075a300, {0xc002284000, 0x109c, 0x109c})

    internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc00075a300, {0xc002284000, 0xc002284005, 0x76d})

    net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc000e18078, {0xc002284000, 0x6, 0xc001f0f7f8})

    net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0002e6060, {0xc002284000, 0x0, 0x409f2d})

    crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc001256278, {0x1e13f00, 0xc0002e6060})

    bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc001256000, {0x1e16e20, 0xc000e18078}, 0x109c)

    crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc001256000, 0x0)

    crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

    crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc001256000, {0xc0022a6000, 0x1000, 0x0})

    crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc0001ce360, {0xc0022a6000, 0xc001522240, 0xc001f0fd30})

    net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc0023963c0)

    bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc0023963c0, 0x1)

    bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc0001ce360)

    net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

    net/http/transport.go:1747 +0x1e05

goroutine 1554101 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002450d20, {0x1e4cfb0, 0xc0001a7cc0}, 0xc000918790)

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0f90, 0x24})

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

    github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3

goroutine 4214784 [select]:

net/http.(*persistConn).writeLoop(0xc002b20000)

    net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

    net/http/transport.go:1748 +0x1e65

goroutine 4543412 [select]:

net/http.(*persistConn).roundTrip(0xc002c98000, 0xc001110240)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc000302e00)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0xc002999a28, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc000302e00, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc000302e00, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc000302e00)

    net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

    net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc000e6cfd0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc001696e80, 0x620faa667a5f8}, {0x1879000, 0xc00059c000}, 0x620faa667a5f8)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc001022fb0, {0xc0022d1020, 0x24}, 0x0)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367

goroutine 4543421 [select]:

net/http.(*persistConn).roundTrip(0xc0006ec000, 0xc0006d6380)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc0007c4600)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc0007c4600, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc0007c4600, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc0007c4600)

    net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

    net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc001784840)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc0010d4840, 0x61f98f2350c7c}, {0x1879000, 0xc0003e0700}, 0x61f98f2350c7c)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc00184b7b0, {0xc0022d11a0, 0x24}, 0xc0006e8130)

    github.com/hashicorp/nomad/api@v0.0.0-20220519231241-2b054e38e91a/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367

goroutine 1553945 [select, 5 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc0017d7500}, {{0xc0022d0cc0, 0x24}, 0xc000b31560, 0xc000b316e0, 0xc000b31500})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 4504308 [select, 5 minutes]:

net/http.(*persistConn).writeLoop(0xc0005d0a20)

    net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

    net/http/transport.go:1748 +0x1e65

goroutine 1553911 [select, 5 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc000b80b80}, {{0xc0022d0c90, 0x24}, 0xc000b31380, 0xc000b314a0, 0xc000b31320})

    github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

    github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5

goroutine 4543308 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc002c98240, 0xc002556140)

    net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc0007c5000)

    net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

    net/http/roundtrip.go:18 +0x19

net/http.send(0xc0007c5000, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

    net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc0007c5000, {0x203000, 0x203000, 0x0})

    net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc0007c5000)

    net/http/client.go:725 +0x908
lgfa29 commented 7 months ago

Hi @bernardoVale 👋

What kind of policies are these? Are they application scaling defined in jobs?

If that's the case, it seems to be coming from Nomad itself, more specifically from here: https://github.com/hashicorp/nomad/blob/dd15bdff9c35be3fe2d0c470feea016967acea9d/helper/pool/pool.go#L512

Have you noticed any errors in the Nomad log when this happens?

bernardoVale commented 6 months ago

What kind of policies are these? Are they application scaling defined in jobs?

Correct, using nomad target plugin

Have you noticed any errors in the Nomad log when this happens?

It happened again today, and not, there was no error log in nomad server/clients

nick-kentik commented 3 weeks ago

We had this recur today, and I thought to dig into the nomad-autoscaler logs to see what was happening at the time the errors started. It looked like:

[...]
2024-07-01T03:09:20.575Z [INFO]  external_plugin.nomad-meta-apm-v2: query response: query="Meta contains \"run_foo-agent\" and SchedulingEligibility == \"eligible\" and Attributes contains \"driver.docker\"" total=10 ineligible=0 job=foo pool=all timestamp=2024-07-01T03:09:20.575Z
2024-07-01T03:09:20.581Z [INFO]  external_plugin.nomad-meta-apm-v2: query response: query="Meta contains \"run_bar-gateway\" and SchedulingEligibility == \"eligible\" and Attributes contains \"driver.docker\"" total=1 ineligible=0 job=bar-gateway pool=default timestamp=2024-07-01T03:09:20.581Z
2024-07-01T03:09:20.590Z [INFO]  external_plugin.nomad-meta-apm-v2: query response: ineligible=0 job=baz pool=default query="Meta contains \"node_type\" and Meta[\"node_type\"] == \"storage\" and SchedulingEligibility == \"eligible\" and Attributes contains \"driver.docker\"" total=3 timestamp=2024-07-01T03:09:20.590Z
2024-07-01T03:10:16.280Z [ERROR] policy_manager.policy_handler: rpc error: code = Unavailable desc = error reading from server: EOF: policy_id=a53e053c-0c60-1296-3e35-8e6ca0aa1a6f
2024-07-01T03:10:16.280Z [ERROR] external_plugin: plugin process exited: plugin=/opt/autoscaler-plugins/kentik-nomad-target id=17 error="exit status 2"
2024-07-01T03:11:17.874Z [ERROR] policy_manager.policy_handler: rpc error: code = Canceled desc = context canceled: policy_id=854225b7-34cd-e54c-7f0f-c06fd8be8903
2024-07-01T03:11:17.874Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=854225b7-34cd-e54c-7f0f-c06fd8be8903 error="rpc error: code = Canceled desc = context canceled"
[...]

So it begins to happen after the external plugin (nomad target) exits with an error. I went looking for the code that restarts the plugin on error, and couldn't find it. I did find this: https://github.com/hashicorp/nomad-autoscaler/blob/main/plugins/manager/manager.go#L202

    // TODO(jrasell) if we do not find the instance, we should probably try and
    //  dispense the plugin. We should also check the plugin instance has not
    //  exited.

So is it the case that nomad-autoscaler simply can't recover if an external plugin dies? The 'rpc error:' text is generic enough that I wonder if perhaps the error message is coming from the gRPC communication between plugin client and server, rather than between plugin server and nomad API?

nick-kentik commented 3 weeks ago

OK, I just tried this out by manually running kill <pid of kentik-nomad-target process>

The logs were a fine copy of the above:

[...]
2024-07-02T12:43:13.258Z [INFO]  external_plugin.nomad-meta-apm-v2: query response: job=foo pool=default query="Meta contains \"run_foo\" and SchedulingEligibility == \"eligible\" and Attributes contains \"driver.docker\"" total=2 ineligible=0 timestamp=2024-07-02T12:43:13.258Z
2024-07-02T12:44:22.364Z [ERROR] external_plugin: plugin process exited: plugin=/opt/autoscaler-plugins/kentik-nomad-target id=40 error="signal: terminated"
2024-07-02T12:45:10.344Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=ff2ef4dc-b2b7-297e-ed89-e91cfd65cace error="rpc error: code = Canceled desc = context canceled"
[...]

So in terms of possible solutions, I guess one of:

?

nick-kentik commented 3 weeks ago

I've got a test case and first pass at a fix in https://github.com/hashicorp/nomad-autoscaler/pull/931