search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.02k stars 2.09k forks source link

fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?) #7507

Open frakev opened 3 months ago

frakev commented 3 months ago

Thanos, Prometheus and Golang version used: Thanos 0.32.5, Prometheus v2.49.1

Object Storage Provider: -

What happened: Thanos receive router are crashing (see screenshot)

What you expected to happen: No crash

How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

Logs

``` ts=2024-07-02T11:29:37.130702059Z caller=handler.go:354 level=info component=receive component=receive-handler msg="Serving plain HTTP" address=0.0.0.0:19291 ts=2024-07-02T11:29:38.13067296Z caller=limiter.go:110 level=info component=receive component=receive-limiter msg="reloading limit config" runtime: pointer 0xc0293d04f7 to unallocated span span.base()=0xc02934c000 span.limit=0xc0293dd88b span.state=0 runtime: found in object at *(0xc0135eda40+0xb0) object=0xc0135eda40 s.base()=0xc0135ec000 s.limit=0xc0135edfe0 s.spanclass=50 s.elemsize=480 s.state=mSpanInUse *(object+0) = 0xc0293d0461 *(object+8) = 0x8 *(object+16) = 0xc0293d046b *(object+24) = 0x15 *(object+32) = 0xc0293d0484 *(object+40) = 0x8 *(object+48) = 0xc0293d048e *(object+56) = 0xf *(object+64) = 0xc0293d04a1 *(object+72) = 0x3 *(object+80) = 0xc0293d04a6 *(object+88) = 0xe *(object+96) = 0xc0293d04b8 *(object+104) = 0xd *(object+112) = 0xc0293d04c7 *(object+120) = 0x9 *(object+128) = 0xc0293d04d4 *(object+136) = 0xd *(object+144) = 0xc0293d04e3 *(object+152) = 0x8 *(object+160) = 0xc0293d04ef *(object+168) = 0x6 *(object+176) = 0xc0293d04f7 <== *(object+184) = 0x4 *(object+192) = 0xc0293d04ff *(object+200) = 0x7 *(object+208) = 0xc0293d0508 *(object+216) = 0x1 *(object+224) = 0xc0293d050d *(object+232) = 0x4 *(object+240) = 0xc0293d0513 *(object+248) = 0x5 *(object+256) = 0xc0293d051c *(object+264) = 0x7 *(object+272) = 0xc0293d0525 *(object+280) = 0xd *(object+288) = 0xc0293d0536 *(object+296) = 0x4 *(object+304) = 0xc0293d053c *(object+312) = 0x3 *(object+320) = 0x0 *(object+328) = 0x0 *(object+336) = 0x0 *(object+344) = 0x0 *(object+352) = 0x0 *(object+360) = 0x0 *(object+368) = 0x0 *(object+376) = 0x0 *(object+384) = 0x0 *(object+392) = 0x0 *(object+400) = 0x0 *(object+408) = 0x0 *(object+416) = 0x0 *(object+424) = 0x0 *(object+432) = 0x0 *(object+440) = 0x0 *(object+448) = 0xc0293d054f *(object+456) = 0x8 *(object+464) = 0xc0293d0559 *(object+472) = 0x14 fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?) runtime stack: runtime.throw({0x26fb0a5?, 0x6?}) /usr/local/go/src/runtime/panic.go:1077 +0x5c fp=0xc001b13df8 sp=0xc001b13dc8 pc=0x43b45c runtime.badPointer(0x7fc822d574f0, 0xc001b13e78?, 0xc0135eda40, 0xc000801c00?) /usr/local/go/src/runtime/mbitmap.go:321 +0x165 fp=0xc001b13e50 sp=0xc001b13df8 pc=0x4172e5 runtime.findObject(0xc0135eda40?, 0xc001b13ec0?, 0x472f2d?) /usr/local/go/src/runtime/mbitmap.go:364 +0xa6 fp=0xc001b13e88 sp=0xc001b13e50 pc=0x417466 runtime.scanobject(0xc0135eda40, 0xc000086140) /usr/local/go/src/runtime/mgcmark.go:1335 +0x14c fp=0xc001b13f10 sp=0xc001b13e88 pc=0x42344c runtime.gcDrainN(0xc000086140, 0x6528) /usr/local/go/src/runtime/mgcmark.go:1192 +0x17a fp=0xc001b13f40 sp=0xc001b13f10 pc=0x4230da runtime.gcAssistAlloc1(0xc000ca2ea0, 0xc00028ed50?) /usr/local/go/src/runtime/mgcmark.go:568 +0x112 fp=0xc001b13fa8 sp=0xc001b13f40 pc=0x421c92 runtime.gcAssistAlloc.func1() /usr/local/go/src/runtime/mgcmark.go:476 +0x1b fp=0xc001b13fc8 sp=0xc001b13fa8 pc=0x421b5b traceback: unexpected SPWRITE function runtime.systemstack runtime.systemstack() /usr/local/go/src/runtime/asm_amd64.s:509 +0x4a fp=0xc001b13fd8 sp=0xc001b13fc8 pc=0x46f70a goroutine 469698 [GC assist marking]: runtime.systemstack_switch() /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc00099f0f8 sp=0xc00099f0e8 pc=0x46f6a8 runtime.gcAssistAlloc(0xc000ca2ea0) /usr/local/go/src/runtime/mgcmark.go:475 +0x198 fp=0xc00099f158 sp=0xc00099f0f8 pc=0x421a18 runtime.deductAssistCredit(0x411bcc?) /usr/local/go/src/runtime/malloc.go:1287 +0x54 fp=0xc00099f180 sp=0xc00099f158 pc=0x410854 runtime.mallocgc(0x10, 0x233d120, 0x1) /usr/local/go/src/runtime/malloc.go:1002 +0xc9 fp=0xc00099f1e8 sp=0xc00099f180 pc=0x410089 runtime.newobject(0x2617b00?) /usr/local/go/src/runtime/malloc.go:1324 +0x25 fp=0xc00099f210 sp=0xc00099f1e8 pc=0x410945 google.golang.org/protobuf/internal/impl.(*MessageInfo).MessageOf(0xc0003438c0, {0x2557620, 0x42a32c0}) /go/pkg/mod/google.golang.org/protobuf@v1.31.0/internal/impl/message_reflect.go:405 +0x7f fp=0xc00099f278 sp=0xc00099f210 pc=0x8525ff google.golang.org/protobuf/internal/impl.legacyWrapMessage({0x2557620?, 0x42a32c0?, 0x5?}) /go/pkg/mod/google.golang.org/protobuf@v1.31.0/internal/impl/legacy_message.go:30 +0x119 fp=0xc00099f2f0 sp=0xc00099f278 pc=0x8475d9 google.golang.org/protobuf/internal/impl.Export.ProtoMessageV2Of({}, {0x2557620, 0x42a32c0}) /go/pkg/mod/google.golang.org/protobuf@v1.31.0/internal/impl/api_export.go:134 +0x89 fp=0xc00099f328 sp=0xc00099f2f0 pc=0x811c69 github.com/golang/protobuf/proto.MessageV2(...) /go/pkg/mod/github.com/golang/protobuf@v1.5.3/proto/proto.go:61 github.com/golang/protobuf/proto.UnmarshalMerge({0x0, 0x0, 0x0}, {0x7fc8232ec240?, 0x42a32c0?}) /go/pkg/mod/github.com/golang/protobuf@v1.5.3/proto/wire.go:63 +0x56 fp=0xc00099f408 sp=0xc00099f328 pc=0x8bd856 github.com/golang/protobuf/proto.Unmarshal({0x0, 0x0, 0x0}, {0x7fc8232ec240, 0x42a32c0?}) /go/pkg/mod/github.com/golang/protobuf@v1.5.3/proto/wire.go:58 +0x53 fp=0xc00099f448 sp=0xc00099f408 pc=0x8bd7b3 google.golang.org/grpc/encoding/proto.codec.Unmarshal({}, {0x0, 0x0, 0x0}, {0x2557620, 0x42a32c0}) /go/pkg/mod/google.golang.org/grpc@v1.45.0/encoding/proto/proto.go:53 +0x5f fp=0xc00099f490 sp=0xc00099f448 pc=0xf8543f google.golang.org/grpc/encoding/proto.(*codec).Unmarshal(0xc00099f530?, {0x0?, 0x0?, 0x0?}, {0x2557620?, 0x42a32c0?}) :1 +0x3f fp=0xc00099f4c8 sp=0xc00099f490 pc=0xf855ff google.golang.org/grpc.recv(0x6?, {0x7fc8235886b8, 0x42a32c0}, 0x5?, {0x0?, 0x0?}, {0x2557620, 0x42a32c0}, 0xc02527ed10?, 0x0, ...) /go/pkg/mod/google.golang.org/grpc@v1.45.0/rpc_util.go:760 +0xaf fp=0xc00099f540 sp=0xc00099f4c8 pc=0xf9d3af google.golang.org/grpc.(*csAttempt).recvMsg(0xc0009616b0, {0x2557620?, 0x42a32c0}, 0xc0001943c0?) /go/pkg/mod/google.golang.org/grpc@v1.45.0/stream.go:977 +0x274 fp=0xc00099f640 sp=0xc00099f540 pc=0xfae994 google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xe000?) /go/pkg/mod/google.golang.org/grpc@v1.45.0/stream.go:828 +0x1f fp=0xc00099f670 sp=0xc00099f640 pc=0xfadddf google.golang.org/grpc.(*clientStream).withRetry(0xc01940c500, 0xc00099f740, 0xc00099f710) /go/pkg/mod/google.golang.org/grpc@v1.45.0/stream.go:686 +0xcb fp=0xc00099f6c8 sp=0xc00099f670 pc=0xfac9eb google.golang.org/grpc.(*clientStream).RecvMsg(0xc01940c500, {0x2557620?, 0x42a32c0?}) /go/pkg/mod/google.golang.org/grpc@v1.45.0/stream.go:827 +0x10a fp=0xc00099f770 sp=0xc00099f6c8 pc=0xfadb2a google.golang.org/grpc.invoke({0x2c7db70?, 0xc014fcbb60?}, {0x26af870?, 0x1a087fe308c00811?}, {0x25574e0, 0xc014fcb740}, {0x2557620, 0x42a32c0}, 0x30?, {0xc0008567c0, ...}) /go/pkg/mod/google.golang.org/grpc@v1.45.0/call.go:73 +0xcb fp=0xc00099f7d8 sp=0xc00099f770 pc=0xf8e1eb github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/tracing.UnaryClientInterceptor.UnaryClientInterceptor.func1({0x2c7db70, 0xc014fcb710}, {0x26af870, 0x22}, {0x25574e0, 0xc014fcb740}, {0x2557620, 0x42a32c0}, 0xc0001b5350?, 0x275d3a8, ...) /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/client.go:22 +0x273 fp=0xc00099f918 sp=0xc00099f7d8 pc=0x15d8713 github.com/thanos-io/thanos/pkg/extgrpc.StoreClientGRPCOpts.ChainUnaryClient.func4.1.1({0x2c7db70?, 0xc014fcb710?}, {0x26af870?, 0x30?}, {0x25574e0?, 0xc014fcb740?}, {0x2557620?, 0x42a32c0?}, 0xc01529b6f8?, {0xc0008567c0, ...}) /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:74 +0x84 fp=0xc00099f990 sp=0xc00099f918 pc=0x1deda64 github.com/thanos-io/thanos/pkg/extgrpc.StoreClientGRPCOpts.(*ClientMetrics).UnaryClientInterceptor.func3({0x2c7db70, 0xc014fcb710}, {0x26af870, 0x22}, {0x25574e0, 0xc014fcb740}, {0x2557620, 0x42a32c0}, 0x0?, 0xc01529b6e0, ...) /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/client_metrics.go:112 +0x10b fp=0xc00099fa20 sp=0xc00099f990 pc=0x1ded70b github.com/thanos-io/thanos/pkg/extgrpc.StoreClientGRPCOpts.ChainUnaryClient.func4.1.1({0x2c7db70?, 0xc014fcb710?}, {0x26af870?, 0x7fc86a0ef1d8?}, {0x25574e0?, 0xc014fcb740?}, {0x2557620?, 0x42a32c0?}, 0xc0040edb18?, {0xc0008567c0, ...}) /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:74 +0x84 fp=0xc00099fa98 sp=0xc00099fa20 pc=0x1deda64 github.com/thanos-io/thanos/pkg/extgrpc.StoreClientGRPCOpts.ChainUnaryClient.func4({0x2c7db70, 0xc014fcb710}, {0x26af870, 0x22}, {0x25574e0, 0xc014fcb740}, {0x2557620, 0x42a32c0}, 0x2499960?, 0x275d3a8, ...) /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:83 +0x143 fp=0xc00099fb28 sp=0xc00099fa98 pc=0x1ded583 google.golang.org/grpc.(*ClientConn).Invoke(0xc000223400, {0x2c7db70?, 0xc014fcb710?}, {0x26af870?, 0xc0040edc20?}, {0x25574e0?, 0xc014fcb740?}, {0x2557620?, 0x42a32c0?}, {0x0, ...}) /go/pkg/mod/google.golang.org/grpc@v1.45.0/call.go:35 +0x205 fp=0xc00099fbc0 sp=0xc00099fb28 pc=0xf8e065 github.com/thanos-io/thanos/pkg/store/storepb.(*writeableStoreClient).RemoteWrite(0x21f48e0?, {0x2c7db70?, 0xc014fcb710?}, 0x2236fa0?, {0x0?, 0xc023f8de90?, 0x4272d20?}) /app/pkg/store/storepb/rpc.pb.go:1197 +0x5d fp=0xc00099fc30 sp=0xc00099fbc0 pc=0x15b1b1d github.com/thanos-io/thanos/pkg/receive.(*Handler).fanoutForward.func3.2({0x2c7db70, 0xc014fcb710}) /app/pkg/receive/handler.go:730 +0x131 fp=0xc00099fcc0 sp=0xc00099fc30 pc=0x1e81ad1 github.com/thanos-io/thanos/pkg/tracing.DoInSpan({0x2c7dc18?, 0xc00068ff10?}, {0x2674d4b?, 0x11?}, 0xc0040edf48, {0x0?, 0xe000?, 0x436d17?}) /app/pkg/tracing/tracing.go:95 +0x98 fp=0xc00099fd30 sp=0xc00099fcc0 pc=0x16065f8 github.com/thanos-io/thanos/pkg/receive.(*Handler).fanoutForward.func3({{0xc000844018, 0x11}, 0x0}) /app/pkg/receive/handler.go:728 +0x49e fp=0xc00099ffb8 sp=0xc00099fd30 pc=0x1e812de github.com/thanos-io/thanos/pkg/receive.(*Handler).fanoutForward.func7() /app/pkg/receive/handler.go:762 +0x2f fp=0xc00099ffe0 sp=0xc00099ffb8 pc=0x1e80e0f runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00099ffe8 sp=0xc00099ffe0 pc=0x471501 created by github.com/thanos-io/thanos/pkg/receive.(*Handler).fanoutForward in goroutine 8444 /app/pkg/receive/handler.go:688 +0x5ab goroutine 1 [chan receive]: runtime.gopark(0xc00093fb68?, 0x4105c5?, 0xe0?, 0xb7?, 0x20?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000de9b00 sp=0xc000de9ae0 pc=0x43e2ae runtime.chanrecv(0xc000c40900, 0xc00093fc00, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000de9b78 sp=0xc000de9b00 pc=0x4099ad runtime.chanrecv1(0x10?, 0xa?) /usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc000de9ba0 sp=0xc000de9b78 pc=0x4095b2 github.com/oklog/run.(*Group).Run(0xc0007e4a38) /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:43 +0x155 fp=0xc000de9c20 sp=0xc000de9ba0 pc=0x621fd5 main.main() /app/cmd/thanos/main.go:159 +0x1878 fp=0xc000de9f40 sp=0xc000de9c20 pc=0x1f0d278 runtime.main() /usr/local/go/src/runtime/proc.go:267 +0x2bb fp=0xc000de9fe0 sp=0xc000de9f40 pc=0x43de3b runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000de9fe8 sp=0xc000de9fe0 pc=0x471501 goroutine 2 [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000090fa8 sp=0xc000090f88 pc=0x43e2ae runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:404 runtime.forcegchelper() /usr/local/go/src/runtime/proc.go:322 +0xb3 fp=0xc000090fe0 sp=0xc000090fa8 pc=0x43e113 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000090fe8 sp=0xc000090fe0 pc=0x471501 created by runtime.init.6 in goroutine 1 /usr/local/go/src/runtime/proc.go:310 +0x1a goroutine 3 [runnable]: runtime.lock2(0x4270c20?) /usr/local/go/src/runtime/lock_futex.go:51 +0x170 fp=0xc000091778 sp=0xc000091770 pc=0x40eb70 runtime.lockWithRank(...) /usr/local/go/src/runtime/lockrank_off.go:24 runtime.lock(...) /usr/local/go/src/runtime/lock_futex.go:48 runtime.bgsweep(0x0?) /usr/local/go/src/runtime/mgcsweep.go:312 +0xa6 fp=0xc0000917c8 sp=0xc000091778 pc=0x4284c6 runtime.gcenable.func1() /usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc0000917e0 sp=0xc0000917c8 pc=0x41d645 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000917e8 sp=0xc0000917e0 pc=0x471501 created by runtime.gcenable in goroutine 1 /usr/local/go/src/runtime/mgc.go:200 +0x66 ... [thanos.log](https://github.com/user-attachments/files/16078901/thanos.log) goroutine 159 [IO wait]: runtime.gopark(0x100?, 0x0?, 0x78?, 0x7b?, 0x4ec5bd?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000b87b08 sp=0xc000b87ae8 pc=0x43e2ae runtime.netpollblock(0x0?, 0x407de6?, 0x0?) /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc000b87b40 sp=0xc000b87b08 pc=0x436d17 internal/poll.runtime_pollWait(0x7fc823380ba0, 0x72) /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc000b87b60 sp=0xc000b87b40 pc=0x46b9a5 internal/poll.(*pollDesc).wait(0xc000320180?, 0x418608?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000b87b88 sp=0xc000b87b60 pc=0x4e5227 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc000320180) /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc000b87c30 sp=0xc000b87b88 pc=0x4ea70c net.(*netFD).accept(0xc000320180) /usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc000b87ce8 sp=0xc000b87c30 pc=0x58a849 net.(*TCPListener).accept(0xc0006dc040) /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc000b87d10 sp=0xc000b87ce8 pc=0x5a3e3e net.(*TCPListener).Accept(0xc0006dc040) /usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc000b87d40 sp=0xc000b87d10 pc=0x5a2ff0 google.golang.org/grpc.(*Server).Serve(0xc000537340, {0x2c73bd0, 0xc0006dc040}) /go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:779 +0x358 fp=0xc000b87e58 sp=0xc000b87d40 pc=0xfa00b8 github.com/thanos-io/thanos/pkg/server/grpc.(*Server).ListenAndServe(0xc0007b5d60) /app/pkg/server/grpc/grpc.go:132 +0x196 fp=0xc000b87ef8 sp=0xc000b87e58 pc=0x1df48d6 main.runReceive.func5() /app/cmd/thanos/receive.go:373 +0x153 fp=0xc000b87f88 sp=0xc000b87ef8 pc=0x1f23973 github.com/oklog/run.(*Group).Run.func1({0xc000c46140?, 0xc0001db620?}) /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29 fp=0xc000b87fc0 sp=0xc000b87f88 pc=0x622149 github.com/oklog/run.(*Group).Run.func2() /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2b fp=0xc000b87fe0 sp=0xc000b87fc0 pc=0x6220eb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000b87fe8 sp=0xc000b87fe0 pc=0x471501 created by github.com/oklog/run.(*Group).Run in goroutine 1 /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67 goroutine 160 [IO wait]: runtime.gopark(0xa7bca5?, 0x7fc8227766a0?, 0xf8?, 0x5a?, 0x4ec5bd?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc006bc5a88 sp=0xc006bc5a68 pc=0x43e2ae runtime.netpollblock(0xc006bc5ae8?, 0x407de6?, 0x0?) /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc006bc5ac0 sp=0xc006bc5a88 pc=0x436d17 internal/poll.runtime_pollWait(0x7fc8233809b0, 0x72) /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc006bc5ae0 sp=0xc006bc5ac0 pc=0x46b9a5 internal/poll.(*pollDesc).wait(0xc00068c080?, 0x0?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc006bc5b08 sp=0xc006bc5ae0 pc=0x4e5227 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc00068c080) /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc006bc5bb0 sp=0xc006bc5b08 pc=0x4ea70c net.(*netFD).accept(0xc00068c080) /usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc006bc5c68 sp=0xc006bc5bb0 pc=0x58a849 net.(*TCPListener).accept(0xc0000b28e0) /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc006bc5c90 sp=0xc006bc5c68 pc=0x5a3e3e net.(*TCPListener).Accept(0xc0000b28e0) /usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc006bc5cc0 sp=0xc006bc5c90 pc=0x5a2ff0 github.com/mwitkow/go-conntrack.(*connTrackListener).Accept(0xc00036e0f0) /go/pkg/mod/github.com/mwitkow/go-conntrack@v0.0.0-20190716064945-2f068394615f/listener_wrapper.go:100 +0x62 fp=0xc006bc5d28 sp=0xc006bc5cc0 pc=0xa85722 net/http.(*onceCloseListener).Accept(0xc00e998360?) :1 +0x24 fp=0xc006bc5d40 sp=0xc006bc5d28 pc=0x7c82a4 net/http.(*Server).Serve(0xc000864000, {0x2c76ab0, 0xc00036e0f0}) /usr/local/go/src/net/http/server.go:3056 +0x364 fp=0xc006bc5e70 sp=0xc006bc5d40 pc=0x7a47e4 github.com/thanos-io/thanos/pkg/receive.(*Handler).Run(0xc00021a7e0) /app/pkg/receive/handler.go:355 +0x5dc fp=0xc006bc5f58 sp=0xc006bc5e70 pc=0x1e7d89c main.runReceive.func7() /app/cmd/thanos/receive.go:388 +0x17 fp=0xc006bc5f88 sp=0xc006bc5f58 pc=0x1f23697 github.com/oklog/run.(*Group).Run.func1({0xc00077f9b0?, 0xc00077f9c0?}) /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29 fp=0xc006bc5fc0 sp=0xc006bc5f88 pc=0x622149 github.com/oklog/run.(*Group).Run.func2() /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2b fp=0xc006bc5fe0 sp=0xc006bc5fc0 pc=0x6220eb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc006bc5fe8 sp=0xc006bc5fe0 pc=0x471501 created by github.com/oklog/run.(*Group).Run in goroutine 1 /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67 goroutine 161 [select]: runtime.gopark(0xc0003056d0?, 0x2?, 0x0?, 0x0?, 0xc0003056a4?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000305548 sp=0xc000305528 pc=0x43e2ae runtime.selectgo(0xc0003056d0, 0xc0003056a0, 0xc0003056f8?, 0x0, 0xc000694608?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000305668 sp=0xc000305548 pc=0x44e765 github.com/thanos-io/thanos/pkg/runutil.Repeat(0xc0006945f0?, 0xc00097c0c0, 0xc000305748) /app/pkg/runutil/runutil.go:77 +0xcb fp=0xc000305708 sp=0xc000305668 pc=0x1608d0b main.runReceive.func11() /app/cmd/thanos/receive.go:417 +0xac fp=0xc000305788 sp=0xc000305708 pc=0x1f2322c github.com/oklog/run.(*Group).Run.func1({0xc000c37f50?, 0xc00077f9e0?}) /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29 fp=0xc0003057c0 sp=0xc000305788 pc=0x622149 github.com/oklog/run.(*Group).Run.func2() /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2b fp=0xc0003057e0 sp=0xc0003057c0 pc=0x6220eb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003057e8 sp=0xc0003057e0 pc=0x471501 created by github.com/oklog/run.(*Group).Run in goroutine 1 /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67 goroutine 162 [chan receive]: runtime.gopark(0x4a0725?, 0x40e44c?, 0xf8?, 0x5e?, 0x4a0699?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000305e68 sp=0xc000305e48 pc=0x43e2ae runtime.chanrecv(0xc000196540, 0x0, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000305ee0 sp=0xc000305e68 pc=0x4099ad runtime.chanrecv1(0xc000694690?, 0x2c7dba8?) /usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc000305f08 sp=0xc000305ee0 pc=0x4095b2 main.runReceive.func13() /app/cmd/thanos/receive.go:436 +0x110 fp=0xc000305f88 sp=0xc000305f08 pc=0x1f23130 github.com/oklog/run.(*Group).Run.func1({0xc000c37f80?, 0xc00077fa00?}) /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29 fp=0xc000305fc0 sp=0xc000305f88 pc=0x622149 github.com/oklog/run.(*Group).Run.func2() /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2b fp=0xc000305fe0 sp=0xc000305fc0 pc=0x6220eb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000305fe8 sp=0xc000305fe0 pc=0x471501 created by github.com/oklog/run.(*Group).Run in goroutine 1 /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67 goroutine 163 [select]: runtime.gopark(0xc0003066f0?, 0x2?, 0xb0?, 0x65?, 0xc0003066bc?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000306558 sp=0xc000306538 pc=0x43e2ae runtime.selectgo(0xc0003066f0, 0xc0003066b8, 0x2?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000306678 sp=0xc000306558 pc=0x44e765 main.interrupt({0x2c5ec80, 0xc0006c6080}, 0xc0007dafc0) /app/cmd/thanos/main.go:170 +0x105 fp=0xc000306760 sp=0xc000306678 pc=0x1f0dce5 main.main.func4() /app/cmd/thanos/main.go:143 +0x1f fp=0xc000306788 sp=0xc000306760 pc=0x1f0d65f github.com/oklog/run.(*Group).Run.func1({0xc0001db6a0?, 0xc00077fa20?}) /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29 fp=0xc0003067c0 sp=0xc000306788 pc=0x622149 github.com/oklog/run.(*Group).Run.func2() /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2b fp=0xc0003067e0 sp=0xc0003067c0 pc=0x6220eb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003067e8 sp=0xc0003067e0 pc=0x471501 created by github.com/oklog/run.(*Group).Run in goroutine 1 /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67 goroutine 164 [select]: runtime.gopark(0xc000306f08?, 0x2?, 0xa0?, 0x6d?, 0xc000306eb4?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000306d48 sp=0xc000306d28 pc=0x43e2ae runtime.selectgo(0xc000306f08, 0xc000306eb0, 0x1?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000306e68 sp=0xc000306d48 pc=0x44e765 main.reload({0x2c5ec80, 0xc0006c6080}, 0xc0007db020, 0x0?) /app/cmd/thanos/main.go:183 +0xf2 fp=0xc000306f58 sp=0xc000306e68 pc=0x1f0dfb2 main.main.func6() /app/cmd/thanos/main.go:153 +0x25 fp=0xc000306f88 sp=0xc000306f58 pc=0x1f0d5e5 github.com/oklog/run.(*Group).Run.func1({0xc000c48000?, 0xc00077fa30?}) /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29 fp=0xc000306fc0 sp=0xc000306f88 pc=0x622149 github.com/oklog/run.(*Group).Run.func2() /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2b fp=0xc000306fe0 sp=0xc000306fc0 pc=0x6220eb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000306fe8 sp=0xc000306fe0 pc=0x471501 created by github.com/oklog/run.(*Group).Run in goroutine 1 /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67 goroutine 165 [select, locked to thread]: runtime.gopark(0xc0003077a8?, 0x2?, 0x49?, 0xe5?, 0xc0003077a4?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000307638 sp=0xc000307618 pc=0x43e2ae runtime.selectgo(0xc0003077a8, 0xc0003077a0, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000307758 sp=0xc000307638 pc=0x44e765 runtime.ensureSigM.func1() /usr/local/go/src/runtime/signal_unix.go:1014 +0x19f fp=0xc0003077e0 sp=0xc000307758 pc=0x467f3f runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003077e8 sp=0xc0003077e0 pc=0x471501 created by runtime.ensureSigM in goroutine 164 /usr/local/go/src/runtime/signal_unix.go:997 +0xc8 goroutine 48 [select]: runtime.gopark(0xc000289fb0?, 0x2?, 0xf2?, 0x1?, 0xc000289f8c?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000289e30 sp=0xc000289e10 pc=0x43e2ae runtime.selectgo(0xc000289fb0, 0xc000289f88, 0x8e02ac?, 0x0, 0x26637c9?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000289f50 sp=0xc000289e30 pc=0x44e765 github.com/thanos-io/thanos/pkg/extkingpin.(*pollingEngine).start.func2() /app/pkg/extkingpin/path_content_reloader.go:70 +0xa9 fp=0xc000289fe0 sp=0xc000289f50 pc=0x16478a9 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000289fe8 sp=0xc000289fe0 pc=0x471501 created by github.com/thanos-io/thanos/pkg/extkingpin.(*pollingEngine).start in goroutine 162 /app/pkg/extkingpin/path_content_reloader.go:68 +0xd0 goroutine 166 [syscall]: runtime.notetsleepg(0x0?, 0x2df00000000?) /usr/local/go/src/runtime/lock_futex.go:236 +0x29 fp=0xc000307fa0 sp=0xc000307f68 pc=0x40f069 os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc000307fc0 sp=0xc000307fa0 pc=0x46db09 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc000307fe0 sp=0xc000307fc0 pc=0x621cd3 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000307fe8 sp=0xc000307fe0 pc=0x471501 created by os/signal.Notify.func1.1 in goroutine 164 /usr/local/go/src/os/signal/signal.go:151 +0x1f goroutine 93 [select]: runtime.gopark(0x1?, 0xc02bbc967e?, 0x7?, 0x0?, 0xc02bbc9687?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00e0a0538 sp=0xc00e0a0518 pc=0x43e2ae panic during panic SIGSEGV: segmentation violation PC=0x461a25 m=15 sigcode=1 ... goroutine 49 [GC worker (idle)]: runtime.gopark(0x42a5860?, 0x1?, 0x8c?, 0x5c?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000584750 sp=0xc000584730 pc=0x43e2ae runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005847e0 sp=0xc000584750 pc=0x41f1c5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005847e8 sp=0xc0005847e0 pc=0x471501 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 50 [GC worker (idle)]: runtime.gopark(0x42a5860?, 0x1?, 0xb2?, 0xa?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000584f50 sp=0xc000584f30 pc=0x43e2ae runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000584fe0 sp=0xc000584f50 pc=0x41f1c5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000584fe8 sp=0xc000584fe0 pc=0x471501 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 51 [GC worker (idle)]: runtime.gopark(0x2aee71ac8ec898?, 0x1?, 0x79?, 0x34?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000585750 sp=0xc000585730 pc=0x43e2ae runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005857e0 sp=0xc000585750 pc=0x41f1c5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005857e8 sp=0xc0005857e0 pc=0x471501 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 52 [GC worker (idle)]: runtime.gopark(0x2aee71ac802ee0?, 0x3?, 0xe?, 0xcc?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000585f50 sp=0xc000585f30 pc=0x43e2ae runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000585fe0 sp=0xc000585f50 pc=0x41f1c5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000585fe8 sp=0xc000585fe0 pc=0x471501 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 53 [GC worker (idle)]: runtime.gopark(0x2aee71ac8ac9b4?, 0x1?, 0x79?, 0x10?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000586750 sp=0xc000586730 pc=0x43e2ae runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005867e0 sp=0xc000586750 pc=0x41f1c5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005867e8 sp=0xc0005867e0 pc=0x471501 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 34 [GC worker (idle)]: runtime.gopark(0x2aee71ad415db0?, 0x1?, 0x22?, 0xe7?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000588f50 sp=0xc000588f30 pc=0x43e2ae runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000588fe0 sp=0xc000588f50 pc=0x41f1c5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000588fe8 sp=0xc000588fe0 pc=0x471501 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 21 [GC worker (idle)]: runtime.gopark(0x2aee71599a34ca?, 0x1?, 0xac?, 0x87?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00008df50 sp=0xc00008df30 pc=0x43e2ae runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00008dfe0 sp=0xc00008df50 pc=0x41f1c5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00008dfe8 sp=0xc00008dfe0 pc=0x471501 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 54 [GC worker (idle)]: runtime.gopark(0x2aee7054330790?, 0x1?, 0xce?, 0x7b?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000586f50 sp=0xc000586f30 pc=0x43e2ae runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000586fe0 sp=0xc000586f50 pc=0x41f1c5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000586fe8 sp=0xc000586fe0 pc=0x471501 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 35 [GC worker (idle)]: runtime.gopark(0x2aee71ac8b822e?, 0x1?, 0xbc?, 0xa5?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000589750 sp=0xc000589730 pc=0x43e2ae runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005897e0 sp=0xc000589750 pc=0x41f1c5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005897e8 sp=0xc0005897e0 pc=0x471501 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 36 [GC worker (active)]: runtime.systemstack_switch() /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc000589f50 sp=0xc000589f40 pc=0x46f6a8 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1353 +0x1f2 fp=0xc000589fe0 sp=0xc000589f50 pc=0x41f2d2 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000589fe8 sp=0xc000589fe0 pc=0x471501 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 37 [GC worker (idle)]: runtime.gopark(0x2aee71ac872e01?, 0x1?, 0xce?, 0x81?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00058a750 sp=0xc00058a730 pc=0x43e2ae runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00058a7e0 sp=0xc00058a750 pc=0x41f1c5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00058a7e8 sp=0xc00058a7e0 pc=0x471501 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 38 [GC worker (idle)]: runtime.gopark(0x2aee71b352e55a?, 0x1?, 0x27?, 0xfe?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00058af50 sp=0xc00058af30 pc=0x43e2ae runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00058afe0 sp=0xc00058af50 pc=0x41f1c5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00058afe8 sp=0xc00058afe0 pc=0x471501 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 39 [GC worker (idle)]: runtime.gopark(0x2aee71ac8c2357?, 0x1?, 0x15?, 0xb4?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00058b750 sp=0xc00058b730 pc=0x43e2ae runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00058b7e0 sp=0xc00058b750 pc=0x41f1c5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00058b7e8 sp=0xc00058b7e0 pc=0x471501 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 55 [select]: runtime.gopark(0xc000092788?, 0x3?, 0x68?, 0xce?, 0xc000092772?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000092618 sp=0xc0000925f8 pc=0x43e2ae runtime.selectgo(0xc000092788, 0xc00009276c, 0xc00024ea80?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000092738 sp=0xc000092618 pc=0x44e765 go.opencensus.io/stats/view.(*worker).start(0xc00024ea80) /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc0000927c8 sp=0xc000092738 pc=0x1079ebf go.opencensus.io/stats/view.init.0.func1() /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc0000927e0 sp=0xc0000927c8 pc=0x10791e5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000927e8 sp=0xc0000927e0 pc=0x471501 created by go.opencensus.io/stats/view.init.0 in goroutine 1 /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d goroutine 85 [IO wait]: runtime.gopark(0x21b8b40?, 0xb?, 0x0?, 0x0?, 0x7?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000b89a70 sp=0xc000b89a50 pc=0x43e2ae runtime.netpollblock(0x487158?, 0x407de6?, 0x0?) /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc000b89aa8 sp=0xc000b89a70 pc=0x436d17 internal/poll.runtime_pollWait(0x7fc823380d90, 0x72) /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc000b89ac8 sp=0xc000b89aa8 pc=0x46b9a5 internal/poll.(*pollDesc).wait(0xc000320380?, 0xc00086c000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000b89af0 sp=0xc000b89ac8 pc=0x4e5227 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc000320380, {0xc00086c000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc000b89b88 sp=0xc000b89af0 pc=0x4e651a net.(*netFD).Read(0xc000320380, {0xc00086c000?, 0x1060100000000?, 0x8?}) /usr/local/go/src/net/fd_posix.go:55 +0x25 fp=0xc000b89bd0 sp=0xc000b89b88 pc=0x588825 net.(*conn).Read(0xc0005e6058, {0xc00086c000?, 0xc0007fe290?, 0xc001214d80?}) /usr/local/go/src/net/net.go:179 +0x45 fp=0xc000b89c18 sp=0xc000b89bd0 pc=0x59ae25 net.(*TCPConn).Read(0x800010601?, {0xc00086c000?, 0xc000b89cc0?, 0x800010601?}) :1 +0x25 fp=0xc000b89c48 sp=0xc000b89c18 pc=0x5ac905 bufio.(*Reader).Read(0xc00036a420, {0xc00084a3c0, 0x9, 0x7fc86a0f0948?}) /usr/local/go/src/bufio/bufio.go:244 +0x197 fp=0xc000b89c80 sp=0xc000b89c48 pc=0x563337 io.ReadAtLeast({0x2c5efc0, 0xc00036a420}, {0xc00084a3c0, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:335 +0x90 fp=0xc000b89cc8 sp=0xc000b89c80 pc=0x4df190 io.ReadFull(...) /usr/local/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0xc00084a3c0, 0x9, 0x4271ce0?}, {0x2c5efc0?, 0xc00036a420?}) /go/pkg/mod/golang.org/x/net@v0.17.0/http2/frame.go:237 +0x65 fp=0xc000b89d18 sp=0xc000b89cc8 pc=0xa9d065 golang.org/x/net/http2.(*Framer).ReadFrame(0xc00084a380) /go/pkg/mod/golang.org/x/net@v0.17.0/http2/frame.go:498 +0x85 fp=0xc000b89dc0 sp=0xc000b89d18 pc=0xa9d7a5 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000874000) /go/pkg/mod/google.golang.org/grpc@v1.45.0/internal/transport/http2_client.go:1499 +0x43d fp=0xc000b89fc8 sp=0xc000b89dc0 pc=0xf703fd google.golang.org/grpc/internal/transport.newHTTP2Client.func7() /go/pkg/mod/google.golang.org/grpc@v1.45.0/internal/transport/http2_client.go:364 +0x25 fp=0xc000b89fe0 sp=0xc000b89fc8 pc=0xf66b05 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000b89fe8 sp=0xc000b89fe0 pc=0x471501 created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 105 /go/pkg/mod/google.golang.org/grpc@v1.45.0/internal/transport/http2_client.go:364 +0x193d goroutine 26 [IO wait]: runtime.gopark(0x4892f0?, 0xc000b88a98?, 0xe8?, 0x8a?, 0x4ec5bd?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000b88a78 sp=0xc000b88a58 pc=0x43e2ae runtime.netpollblock(0x46f6f2?, 0x407de6?, 0x0?) /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc000b88ab0 sp=0xc000b88a78 pc=0x436d17 internal/poll.runtime_pollWait(0x7fc823380aa8, 0x72) /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc000b88ad0 sp=0xc000b88ab0 pc=0x46b9a5 internal/poll.(*pollDesc).wait(0xc00068c000?, 0x4?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000b88af8 sp=0xc000b88ad0 pc=0x4e5227 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc00068c000) /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc000b88ba0 sp=0xc000b88af8 pc=0x4ea70c net.(*netFD).accept(0xc00068c000) /usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc000b88c58 sp=0xc000b88ba0 pc=0x58a849 net.(*TCPListener).accept(0xc0006dc060) /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc000b88c80 sp=0xc000b88c58 pc=0x5a3e3e net.(*TCPListener).Accept(0xc0006dc060) /usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc000b88cb0 sp=0xc000b88c80 pc=0x5a2ff0 net/http.(*onceCloseListener).Accept(0xc0257e39e0?) :1 +0x24 fp=0xc000b88cc8 sp=0xc000b88cb0 pc=0x7c82a4 net/http.(*Server).Serve(0xc0008c6780, {0x2c73bd0, 0xc0006dc060}) /usr/local/go/src/net/http/server.go:3056 +0x364 fp=0xc000b88df8 sp=0xc000b88cc8 pc=0x7a47e4 github.com/prometheus/exporter-toolkit/web.Serve({0x2c73bd0, 0xc0006dc060}, 0xc0008c6780, 0xc00051e120, {0x2c5ec40?, 0xc00085def0}) /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:278 +0x34e fp=0xc000b88f38 sp=0xc000b88df8 pc=0x151edae github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1() /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:232 +0x2b fp=0xc000b88f78 sp=0xc000b88f38 pc=0x151e52b golang.org/x/sync/errgroup.(*Group).Go.func1() /go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x56 fp=0xc000b88fe0 sp=0xc000b88f78 pc=0xa60816 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000b88fe8 sp=0xc000b88fe0 pc=0x471501 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 158 /go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x96 goroutine 84 [select]: runtime.gopark(0xc000637f58?, 0x4?, 0xc5?, 0x5?, 0xc000637ed0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000288d60 sp=0xc000288d40 pc=0x43e2ae runtime.selectgo(0xc000288f58, 0xc000637ec8, 0xc00085de50?, 0x0, 0xc0005877d0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000288e80 sp=0xc000288d60 pc=0x44e765 github.com/thanos-io/thanos/pkg/receive.(*ConfigWatcher).Run(0xc0007ecf20, {0x2c7dba8, 0xc00085de50}) /app/pkg/receive/config.go:171 +0x193 fp=0xc000288fb8 sp=0xc000288e80 pc=0x1e79f53 github.com/thanos-io/thanos/pkg/receive.ConfigFromWatcher.func2() /app/pkg/receive/config.go:284 +0x28 fp=0xc000288fe0 sp=0xc000288fb8 pc=0x1e7b1a8 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000288fe8 sp=0xc000288fe0 pc=0x471501 created by github.com/thanos-io/thanos/pkg/receive.ConfigFromWatcher in goroutine 156 /app/pkg/receive/config.go:284 +0xef goroutine 104 [select]: runtime.gopark(0xc000b86f38?, 0x2?, 0xc0?, 0x6d?, 0xc000b86ee4?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000b86d80 sp=0xc000b86d60 pc=0x43e2ae runtime.selectgo(0xc000b86f38, 0xc000b86ee0, 0xc0007f8530?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000b86ea0 sp=0xc000b86d80 pc=0x44e765 google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00085c9b0) /go/pkg/mod/google.golang.org/grpc@v1.45.0/balancer_conn_wrappers.go:77 +0x95 fp=0xc000b86fc8 sp=0xc000b86ea0 pc=0xf8c555 google.golang.org/grpc.newCCBalancerWrapper.func1() /go/pkg/mod/google.golang.org/grpc@v1.45.0/balancer_conn_wrappers.go:67 +0x25 fp=0xc000b86fe0 sp=0xc000b86fc8 pc=0xf8c485 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000b86fe8 sp=0xc000b86fe0 pc=0x471501 created by google.golang.org/grpc.newCCBalancerWrapper in goroutine 1 /go/pkg/mod/google.golang.org/grpc@v1.45.0/balancer_conn_wrappers.go:67 +0x21b goroutine 42757 [select]: runtime.gopark(0xc00a194790?, 0x2?, 0x8?, 0xd1?, 0xc00a1946b4?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00a194538 sp=0xc00a194518 pc=0x43e2ae runtime.selectgo(0xc00a194790, 0xc00a1946b0, 0xc00a194750?, 0x0, 0x2191580?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00a194658 sp=0xc00a194538 pc=0x44e765 github.com/thanos-io/thanos/pkg/receive.(*Handler).fanoutForward(0xc00021a7e0, {0x2c7db70, 0xc01fd35e60}, {0x2671fb8, 0xe}, 0xc01fd35e90, 0x7d0, 0x0) /app/pkg/receive/handler.go:789 +0x971 fp=0xc00a194880 sp=0xc00a194658 pc=0x1e807f1 github.com/thanos-io/thanos/pkg/receive.(*Handler).forward(0xc00021a7e0, {0x2c7db70?, 0xc001bba720?}, {0x2671fb8, 0xe}, {0x0?, 0x2?}, 0xc01fd35d10) /app/pkg/receive/handler.go:612 +0x3cc fp=0xc00a194a38 sp=0xc00a194880 pc=0x1e7f90c github.com/thanos-io/thanos/pkg/receive.(*Handler).handleRequest(0xc00021a7e0, {0x2c7db70, 0xc001bba720}, 0x0, {0x2671fb8, 0xe}, 0x0?) /app/pkg/receive/handler.go:417 +0x313 fp=0xc00a194ad0 sp=0xc00a194a38 pc=0x1e7dc53 github.com/thanos-io/thanos/pkg/receive.(*Handler).receiveHTTP(0xc00021a7e0, {0x2c72b80, 0xc002482040}, 0xc022ee7700) /app/pkg/receive/handler.go:539 +0x1105 fp=0xc00a194e20 sp=0xc00a194ad0 pc=0x1e7edc5 github.com/thanos-io/thanos/pkg/receive.(*Handler).receiveHTTP-fm({0x2c72b80?, 0xc002482040?}, 0xc022ee7700?) :1 +0x36 fp=0xc00a194e50 sp=0xc00a194e20 pc=0x1e98c76 net/http.HandlerFunc.ServeHTTP(0xc022ee7600?, {0x2c72b80?, 0xc002482040?}, 0xc003b3cc00?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc00a194e78 sp=0xc00a194e50 pc=0x7a16a9 github.com/thanos-io/thanos/pkg/receive.NewHandler.RequestID.func2({0x2c72b80, 0xc002482040}, 0xc022ee7600) /app/pkg/server/http/middleware/request_id.go:40 +0x148 fp=0xc00a194ef0 sp=0xc00a194e78 pc=0x1e7c908 github.com/thanos-io/thanos/pkg/receive.(*Handler).testReady-fm.(*Handler).testReady.func1({0x2c72b80, 0xc002482040}, 0x55555555555550f9?) /app/pkg/receive/handler.go:263 +0x155 fp=0xc00a194f60 sp=0xc00a194ef0 pc=0x1e96955 net/http.HandlerFunc.ServeHTTP(0x20e25e0?, {0x2c72b80?, 0xc002482040?}, 0x76?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc00a194f88 sp=0xc00a194f60 pc=0x7a16a9 github.com/thanos-io/thanos/pkg/extprom/http.httpInstrumentationHandler.func1({0x7fc8235760c0?, 0xc006122730}, 0xc022ee7600) /app/pkg/extprom/http/instrument_server.go:75 +0x103 fp=0xc00a195288 sp=0xc00a194f88 pc=0x14dd963 net/http.HandlerFunc.ServeHTTP(0x7fc8235760c0?, {0x7fc8235760c0?, 0xc006122730?}, 0xc00a1952f0?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc00a1952b0 sp=0xc00a195288 pc=0x7a16a9 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1({0x7fc8235760c0?, 0xc006122690?}, 0xc022ee7600) /go/pkg/mod/github.com/prometheus/client_golang@v1.16.0/prometheus/promhttp/instrument_server.go:296 +0xca fp=0xc00a1953d0 sp=0xc00a1952b0 pc=0x14b83aa net/http.HandlerFunc.ServeHTTP(0x7fc8235760c0?, {0x7fc8235760c0?, 0xc006122690?}, 0xc00bf8f080?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc00a1953f8 sp=0xc00a1953d0 pc=0x7a16a9 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x7fc8235760c0?, 0xc006122640?}, 0xc022ee7600) /go/pkg/mod/github.com/prometheus/client_golang@v1.16.0/prometheus/promhttp/instrument_server.go:147 +0xb6 fp=0xc00a195508 sp=0xc00a1953f8 pc=0x14b7436 net/http.HandlerFunc.ServeHTTP(0x22375a0?, {0x7fc8235760c0?, 0xc006122640?}, 0x6?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc00a195530 sp=0xc00a195508 pc=0x7a16a9 github.com/thanos-io/thanos/pkg/extprom/http.httpInstrumentationHandler.instrumentHandlerInFlight.func2({0x7fc8235760c0, 0xc006122640}, 0xc022ee7600) /app/pkg/extprom/http/instrument_server.go:164 +0x167 fp=0xc00a1955c8 sp=0xc00a195530 pc=0x14dd7a7 net/http.HandlerFunc.ServeHTTP(0x2c73090?, {0x7fc8235760c0?, 0xc006122640?}, 0x20?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc00a1955f0 sp=0xc00a1955c8 pc=0x7a16a9 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1({0x2c73090?, 0xc002ddc1c0?}, 0xc022ee7600) /go/pkg/mod/github.com/prometheus/client_golang@v1.16.0/prometheus/promhttp/instrument_server.go:243 +0xca fp=0xc00a195718 sp=0xc00a1955f0 pc=0x14b7d8a net/http.HandlerFunc.ServeHTTP(...) /usr/local/go/src/net/http/server.go:2136 github.com/thanos-io/thanos/pkg/extprom/http.(*tenantInstrumentationMiddleware).NewHandler.func1({0x2c73090, 0xc002ddc1c0}, 0xc022ee7600) /app/pkg/extprom/http/instrument_tenant_server.go:38 +0x156 fp=0xc00a195788 sp=0xc00a195718 pc=0x14de516 net/http.HandlerFunc.ServeHTTP(0x2c7db70?, {0x2c73090?, 0xc002ddc1c0?}, 0xc0245184e0?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc00a1957b0 sp=0xc00a195788 pc=0x7a16a9 github.com/thanos-io/thanos/pkg/tracing.HTTPMiddleware.func1({0x2c73090, 0xc002ddc1c0}, 0xc022ee7500) /app/pkg/tracing/http.go:62 +0x9ad fp=0xc00a195978 sp=0xc00a1957b0 pc=0x160598d github.com/prometheus/common/route.(*Router).handle.func1({0x2c73090, 0xc002ddc1c0}, 0xc022ee7400, {0x0, 0x0, 0x1?}) /go/pkg/mod/github.com/prometheus/common@v0.44.0/route/route.go:83 +0x1f6 fp=0xc00a195a10 sp=0xc00a195978 pc=0x8e9836 github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc000b500c0, {0x2c73090, 0xc002ddc1c0}, 0xc022ee7400) /go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0x7eb fp=0xc00a195b18 sp=0xc00a195a10 pc=0x8e66eb github.com/prometheus/common/route.(*Router).ServeHTTP(0x410945?, {0x2c73090?, 0xc002ddc1c0?}, 0xa5555596955aff26?) /go/pkg/mod/github.com/prometheus/common@v0.44.0/route/route.go:126 +0x25 fp=0xc00a195b48 sp=0xc00a195b18 pc=0x8e9e25 net/http.serverHandler.ServeHTTP({0x2c6c600?}, {0x2c73090?, 0xc002ddc1c0?}, 0x6?) /usr/local/go/src/net/http/server.go:2938 +0x8e fp=0xc00a195b78 sp=0xc00a195b48 pc=0x7a438e net/http.(*conn).serve(0xc011662ab0, {0x2c7db70, 0xc0008541b0}) /usr/local/go/src/net/http/server.go:2009 +0x5f4 fp=0xc00a195fb8 sp=0xc00a195b78 pc=0x7a0274 net/http.(*Server).Serve.func3() /usr/local/go/src/net/http/server.go:3086 +0x28 fp=0xc00a195fe0 sp=0xc00a195fb8 pc=0x7a4ba8 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00a195fe8 sp=0xc00a195fe0 pc=0x471501 created by net/http.(*Server).Serve in goroutine 160 /usr/local/go/src/net/http/server.go:3086 +0x5cb goroutine 106 [select]: runtime.gopark(0xc00028ef50?, 0x3?, 0x10?, 0x14?, 0xc00028ef0a?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00028ed80 sp=0xc00028ed60 pc=0x43e2ae runtime.selectgo(0xc00028ef50, 0xc00028ef04, 0xc000814000?, 0x0, 0x3?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00028eea0 sp=0xc00028ed80 pc=0x44e765 go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0xc0007b41e0) /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.16.0/trace/batch_span_processor.go:293 +0x125 fp=0xc00028efa0 sp=0xc00028eea0 pc=0x15e0c65 go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1() /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.16.0/trace/batch_span_processor.go:128 +0x54 fp=0xc00028efe0 sp=0xc00028efa0 pc=0x15dff54 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00028efe8 sp=0xc00028efe0 pc=0x471501 created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor in goroutine 1 /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.16.0/trace/batch_span_processor.go:126 +0x2e5 goroutine 140 [IO wait]: runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0x3?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000651ce0 sp=0xc000651cc0 pc=0x43e2ae runtime.netpollblock(0x487158?, 0x407de6?, 0x0?) /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc000651d18 sp=0xc000651ce0 pc=0x436d17 internal/poll.runtime_pollWait(0x7fc823380e88, 0x72) /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc000651d38 sp=0xc000651d18 pc=0x46b9a5 internal/poll.(*pollDesc).wait(0xc000b50a20?, 0xc000651ed0?, 0x1) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000651d60 sp=0xc000651d38 pc=0x4e5227 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc000b50a20, {0xc000651ed0, 0x10000, 0x10000}) /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc000651df8 sp=0xc000651d60 pc=0x4e651a os.(*File).read(...) /usr/local/go/src/os/file_posix.go:29 os.(*File).Read(0xc0001b54f0, {0xc000651ed0?, 0xc00084d1f0?, 0xa?}) /usr/local/go/src/os/file.go:118 +0x52 fp=0xc000651e38 sp=0xc000651df8 pc=0x4f1e72 github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc00085db80) /go/pkg/mod/github.com/fsnotify/fsnotify@v1.6.0/backend_inotify.go:356 +0xd2 fp=0xc000661fc8 sp=0xc000651e38 pc=0x1b01c12 github.com/fsnotify/fsnotify.NewWatcher.func1() /go/pkg/mod/github.com/fsnotify/fsnotify@v1.6.0/backend_inotify.go:150 +0x25 fp=0xc000661fe0 sp=0xc000661fc8 pc=0x1b010a5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000661fe8 sp=0xc000661fe0 pc=0x471501 created by github.com/fsnotify/fsnotify.NewWatcher in goroutine 1 /go/pkg/mod/github.com/fsnotify/fsnotify@v1.6.0/backend_inotify.go:150 +0x186 goroutine 28 [select]: runtime.gopark(0xc006c86790?, 0x2?, 0x98?, 0x4?, 0xc006c866b4?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc006c86538 sp=0xc006c86518 pc=0x43e2ae runtime.selectgo(0xc006c86790, 0xc006c866b0, 0xc006c86750?, 0x0, 0x2191580?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc006c86658 sp=0xc006c86538 pc=0x44e765 github.com/thanos-io/thanos/pkg/receive.(*Handler).fanoutForward(0xc00021a7e0, {0x2c7db70, 0xc005728960}, {0x2671fb8, 0xe}, 0xc021247410, 0x7d0, 0x0) /app/pkg/receive/handler.go:789 +0x971 fp=0xc006c86880 sp=0xc006c86658 pc=0x1e807f1 github.com/thanos-io/thanos/pkg/receive.(*Handler).forward(0xc00021a7e0, {0x2c7db70?, 0xc0249ff110?}, {0x2671fb8, 0xe}, {0x0?, 0x2?}, 0xc005728810) /app/pkg/receive/handler.go:612 +0x3cc fp=0xc006c86a38 sp=0xc006c86880 pc=0x1e7f90c github.com/thanos-io/thanos/pkg/receive.(*Handler).handleRequest(0xc00021a7e0, {0x2c7db70, 0xc0249ff110}, 0x0, {0x2671fb8, 0xe}, 0x0?) /app/pkg/receive/handler.go:417 +0x313 fp=0xc006c86ad0 sp=0xc006c86a38 pc=0x1e7dc53 github.com/thanos-io/thanos/pkg/receive.(*Handler).receiveHTTP(0xc00021a7e0, {0x2c72b80, 0xc022308300}, 0xc015434300) /app/pkg/receive/handler.go:539 +0x1105 fp=0xc006c86e20 sp=0xc006c86ad0 pc=0x1e7edc5 github.com/thanos-io/thanos/pkg/receive.(*Handler).receiveHTTP-fm({0x2c72b80?, 0xc022308300?}, 0x42a64a0?) :1 +0x36 fp=0xc006c86e50 sp=0xc006c86e20 pc=0x1e98c76 net/http.HandlerFunc.ServeHTTP(0xc015434200?, {0x2c72b80?, 0xc022308300?}, 0xc00452a080?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc006c86e78 sp=0xc006c86e50 pc=0x7a16a9 github.com/thanos-io/thanos/pkg/receive.NewHandler.RequestID.func2({0x2c72b80, 0xc022308300}, 0xc015434200) /app/pkg/server/http/middleware/request_id.go:40 +0x148 fp=0xc006c86ef0 sp=0xc006c86e78 pc=0x1e7c908 github.com/thanos-io/thanos/pkg/receive.(*Handler).testReady-fm.(*Handler).testReady.func1({0x2c72b80, 0xc022308300}, 0x495401?) /app/pkg/receive/handler.go:263 +0x155 fp=0xc006c86f60 sp=0xc006c86ef0 pc=0x1e96955 net/http.HandlerFunc.ServeHTTP(0x20e25e0?, {0x2c72b80?, 0xc022308300?}, 0x76?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc006c86f88 sp=0xc006c86f60 pc=0x7a16a9 github.com/thanos-io/thanos/pkg/extprom/http.httpInstrumentationHandler.func1({0x7fc8235760c0?, 0xc007f965f0}, 0xc015434200) /app/pkg/extprom/http/instrument_server.go:75 +0x103 fp=0xc006c87288 sp=0xc006c86f88 pc=0x14dd963 net/http.HandlerFunc.ServeHTTP(0x7fc8235760c0?, {0x7fc8235760c0?, 0xc007f965f0?}, 0x7fc86a0fe918?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc006c872b0 sp=0xc006c87288 pc=0x7a16a9 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1({0x7fc8235760c0?, 0xc007f96550?}, 0xc015434200) /go/pkg/mod/github.com/prometheus/client_golang@v1.16.0/prometheus/promhttp/instrument_server.go:296 +0xca fp=0xc006c873d0 sp=0xc006c872b0 pc=0x14b83aa net/http.HandlerFunc.ServeHTTP(0x7fc8235760c0?, {0x7fc8235760c0?, 0xc007f96550?}, 0xc0249feed0?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc006c873f8 sp=0xc006c873d0 pc=0x7a16a9 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x7fc8235760c0?, 0xc007f964b0?}, 0xc015434200) /go/pkg/mod/github.com/prometheus/client_golang@v1.16.0/prometheus/promhttp/instrument_server.go:147 +0xb6 fp=0xc006c87508 sp=0xc006c873f8 pc=0x14b7436 net/http.HandlerFunc.ServeHTTP(0x22375a0?, {0x7fc8235760c0?, 0xc007f964b0?}, 0x6?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc006c87530 sp=0xc006c87508 pc=0x7a16a9 github.com/thanos-io/thanos/pkg/extprom/http.httpInstrumentationHandler.instrumentHandlerInFlight.func2({0x7fc8235760c0, 0xc007f964b0}, 0xc015434200) /app/pkg/extprom/http/instrument_server.go:164 +0x167 fp=0xc006c875c8 sp=0xc006c87530 pc=0x14dd7a7 net/http.HandlerFunc.ServeHTTP(0x2c73090?, {0x7fc8235760c0?, 0xc007f964b0?}, 0x20?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc006c875f0 sp=0xc006c875c8 pc=0x7a16a9 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1({0x2c73090?, 0xc0024540e0?}, 0xc015434200) /go/pkg/mod/github.com/prometheus/client_golang@v1.16.0/prometheus/promhttp/instrument_server.go:243 +0xca fp=0xc006c87718 sp=0xc006c875f0 pc=0x14b7d8a net/http.HandlerFunc.ServeHTTP(...) /usr/local/go/src/net/http/server.go:2136 github.com/thanos-io/thanos/pkg/extprom/http.(*tenantInstrumentationMiddleware).NewHandler.func1({0x2c73090, 0xc0024540e0}, 0xc015434200) /app/pkg/extprom/http/instrument_tenant_server.go:38 +0x156 fp=0xc006c87788 sp=0xc006c87718 pc=0x14de516 net/http.HandlerFunc.ServeHTTP(0x2c7db70?, {0x2c73090?, 0xc0024540e0?}, 0xc0249fe1e0?) /usr/local/go/src/net/http/server.go:2136 +0x29 fp=0xc006c877b0 sp=0xc006c87788 pc=0x7a16a9 github.com/thanos-io/thanos/pkg/tracing.HTTPMiddleware.func1({0x2c73090, 0xc0024540e0}, 0xc015434100) /app/pkg/tracing/http.go:62 +0x9ad fp=0xc006c87978 sp=0xc006c877b0 pc=0x160598d github.com/prometheus/common/route.(*Router).handle.func1({0x2c73090, 0xc0024540e0}, 0xc015434000, {0x0, 0x0, 0x1?}) /go/pkg/mod/github.com/prometheus/common@v0.44.0/route/route.go:83 +0x1f6 fp=0xc006c87a10 sp=0xc006c87978 pc=0x8e9836 github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc000b500c0, {0x2c73090, 0xc0024540e0}, 0xc015434000) /go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0x7eb fp=0xc006c87b18 sp=0xc006c87a10 pc=0x8e66eb github.com/prometheus/common/route.(*Router).ServeHTTP(0x410945?, {0x2c73090?, 0xc0024540e0?}, 0xc002454001?) /go/pkg/mod/github.com/prometheus/common@v0.44.0/route/route.go:126 +0x25 fp=0xc006c87b48 sp=0xc006c87b18 pc=0x8e9e25 net/http.serverHandler.ServeHTTP({0x2c6c600?}, {0x2c73090?, 0xc0024540e0?}, 0x6?) /usr/local/go/src/net/http/server.go:2938 +0x8e fp=0xc006c87b78 sp=0xc006c87b48 pc=0x7a438e net/http.(*conn).serve(0xc00011fb90, {0x2c7db70, 0xc0008541b0}) /usr/local/go/src/net/http/server.go:2009 +0x5f4 fp=0xc006c87fb8 sp=0xc006c87b78 pc=0x7a0274 net/http.(*Server).Serve.func3() /usr/local/go/src/net/http/server.go:3086 +0x28 fp=0xc006c87fe0 sp=0xc006c87fb8 pc=0x7a4ba8 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc006c87fe8 sp=0xc006c87fe0 pc=0x471501 created by net/http.(*Server).Serve in goroutine 160 /usr/local/go/src/net/http/server.go:3086 +0x5cb ... goroutine 163 [select]: runtime.gopark(0xc0003066f0?, 0x2?, 0xb0?, 0x65?, 0xc0003066bc?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000306558 sp=0xc000306538 pc=0x43e2ae runtime.selectgo(0xc0003066f0, 0xc0003066b8, 0x2?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000306678 sp=0xc000306558 pc=0x44e765 main.interrupt({0x2c5ec80, 0xc0006c6080}, 0xc0007dafc0) /app/cmd/thanos/main.go:170 +0x105 fp=0xc000306760 sp=0xc000306678 pc=0x1f0dce5 main.main.func4() /app/cmd/thanos/main.go:143 +0x1f fp=0xc000306788 sp=0xc000306760 pc=0x1f0d65f github.com/oklog/run.(*Group).Run.func1({0xc0001db6a0?, 0xc00077fa20?}) /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29 fp=0xc0003067c0 sp=0xc000306788 pc=0x622149 github.com/oklog/run.(*Group).Run.func2() /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2b fp=0xc0003067e0 sp=0xc0003067c0 pc=0x6220eb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003067e8 sp=0xc0003067e0 pc=0x471501 created by github.com/oklog/run.(*Group).Run in goroutine 1 /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67 goroutine 164 [select]: runtime.gopark(0xc000306f08?, 0x2?, 0xa0?, 0x6d?, 0xc000306eb4?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000306d48 sp=0xc000306d28 pc=0x43e2ae runtime.selectgo(0xc000306f08, 0xc000306eb0, 0x1?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000306e68 sp=0xc000306d48 pc=0x44e765 main.reload({0x2c5ec80, 0xc0006c6080}, 0xc0007db020, 0x0?) /app/cmd/thanos/main.go:183 +0xf2 fp=0xc000306f58 sp=0xc000306e68 pc=0x1f0dfb2 main.main.func6() /app/cmd/thanos/main.go:153 +0x25 fp=0xc000306f88 sp=0xc000306f58 pc=0x1f0d5e5 github.com/oklog/run.(*Group).Run.func1({0xc000c48000?, 0xc00077fa30?}) /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29 fp=0xc000306fc0 sp=0xc000306f88 pc=0x622149 github.com/oklog/run.(*Group).Run.func2() /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2b fp=0xc000306fe0 sp=0xc000306fc0 pc=0x6220eb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000306fe8 sp=0xc000306fe0 pc=0x471501 created by github.com/oklog/run.(*Group).Run in goroutine 1 /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67 goroutine 165 [select, locked to thread]: runtime.gopark(0xc0003077a8?, 0x2?, 0x49?, 0xe5?, 0xc0003077a4?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000307638 sp=0xc000307618 pc=0x43e2ae runtime.selectgo(0xc0003077a8, 0xc0003077a0, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000307758 sp=0xc000307638 pc=0x44e765 runtime.ensureSigM.func1() /usr/local/go/src/runtime/signal_unix.go:1014 +0x19f fp=0xc0003077e0 sp=0xc000307758 pc=0x467f3f runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003077e8 sp=0xc0003077e0 pc=0x471501 created by runtime.ensureSigM in goroutine 164 /usr/local/go/src/runtime/signal_unix.go:997 +0xc8 goroutine 48 [select]: runtime.gopark(0xc000289fb0?, 0x2?, 0xf2?, 0x1?, 0xc000289f8c?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000289e30 sp=0xc000289e10 pc=0x43e2ae runtime.selectgo(0xc000289fb0, 0xc000289f88, 0x8e02ac?, 0x0, 0x26637c9?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000289f50 sp=0xc000289e30 pc=0x44e765 github.com/thanos-io/thanos/pkg/extkingpin.(*pollingEngine).start.func2() /app/pkg/extkingpin/path_content_reloader.go:70 +0xa9 fp=0xc000289fe0 sp=0xc000289f50 pc=0x16478a9 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000289fe8 sp=0xc000289fe0 pc=0x471501 created by github.com/thanos-io/thanos/pkg/extkingpin.(*pollingEngine).start in goroutine 162 /app/pkg/extkingpin/path_content_reloader.go:68 +0xd0 goroutine 166 [syscall]: runtime.notetsleepg(0x0?, 0x2df00000000?) /usr/local/go/src/runtime/lock_futex.go:236 +0x29 fp=0xc000307fa0 sp=0xc000307f68 pc=0x40f069 os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc000307fc0 sp=0xc000307fa0 pc=0x46db09 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc000307fe0 sp=0xc000307fc0 pc=0x621cd3 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000307fe8 sp=0xc000307fe0 pc=0x471501 created by os/signal.Notify.func1.1 in goroutine 164 /usr/local/go/src/os/signal/signal.go:151 +0x1f goroutine 93 [select]: runtime.gopark(0x1?, 0xc02bbc967e?, 0x7?, 0x0?, 0xc02bbc9687?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00e0a0538 sp=0xc00e0a0518 pc=0x43e2ae ts=2024-07-02T11:30:38.483963978Z caller=factory.go:43 level=info msg="loading tracing configuration" ts=2024-07-02T11:30:38.484997279Z caller=receive.go:134 level=info component=receive mode=RouterOnly msg="running receive" ```

Anything else we need to know: We have set limits to our Thanos Receive router.

write:
  global:
    meta_monitoring_url: "http://x.x.x.x/thanos/"
    meta_monitoring_limit_query: "sum(prometheus_tsdb_head_series) by (tenant)"
  default:
    head_series_limit: 0    #Specifies the total number of active (head) series for any tenant, across all replicas (including data replication), allowed by Thanos Receive.

We use tracing. Logs are truncated in this message (see log attached)

frakev commented 3 months ago

thanos.log