Possible goroutines leak for query and query-frontend

[query] [query-frontend] Possible goroutine leak

Thanos, Prometheus and Golang version used: thanos: 0.29,0.30-rc0 prometheus: 2.37 go version: go1.19.4

Object Storage Provider: s3

What happened: Thanos quey and qury frontend seems to consume too much gorotuines and memory and generally slow in querying after upgrading from 0.26

What you expected to happen: goroutines and memory consumption to decrease

How to reproduce it (as minimally and precisely as possible): query config

/opt/thanos/current/thanos query --http-address 0.0.0.0:19198 --grpc-address 0.0.0.0:19182  --query.replica-label replica --query.replica-label prometheus_replica --log.format=json --request.logging-config-file=/etc/thanos/logging_config.yaml --store dnssrv+_grpc._tcp.thanos-store.monitoring --store 10.20.76.32:19187 --store 10.40.208.29:19181 --store 10.20.76.32:19185 --store 10.40.208.29:19185  --query.timeout=10s --store.response-timeout=15s --grpc-grace-period=10s --query.active-query-path="/mnt/data/thanos" --query.promql-engine=thanos

query frontend config

/opt/thanos/current/thanos query-frontend --http-address 0.0.0.0:19192 --query-frontend.downstream-url="http://0.0.0.0:19198"  --log.format=json --labels.response-cache-config-file=/etc/thanos/cache.yaml  --query-range.response-cache-config-file=/etc/thanos/cache.yaml

Full logs to relevant components: query frontend log

Dec 26 16:41:46 thanos thanos-query-frontend[8462]: {"caller":"retry.go:81","err":"rpc error: code = Code(500) desc = error decoding response: context canceled","level":"error","msg":"error processing request","org_id":"anonymous","try":0,"ts":"2022-12-26T13:41:46.771161848Z"}
Dec 26 16:41:46 thanos thanos-query-frontend[8462]: {"caller":"retry.go:81","err":"rpc error: code = Code(500) desc = error decoding response: context canceled","level":"error","msg":"error processing request","org_id":"anonymous","try":0,"ts":"2022-12-26T13:41:46.994252914Z"}
Dec 26 16:52:42 thanos thanos-query-frontend[8462]: {"caller":"retry.go:81","err":"rpc error: code = Code(500) desc = error decoding response: context canceled","level":"error","msg":"error processing request","org_id":"anonymous","try":0,"ts":"2022-12-26T13:52:42.746838668Z"}
Dec 26 16:54:20 thanos thanos-query-frontend[8462]: {"caller":"retry.go:81","err":"rpc error: code = Code(500) desc = error decoding response: context canceled","level":"error","msg":"error processing request","org_id":"anonymous","try":0,"ts":"2022-12-26T13:54:20.773531266Z"}
Dec 26 17:00:08 thanos thanos-query-frontend[8462]: {"caller":"retry.go:81","err":"rpc error: code = Code(500) desc = error decoding response: context canceled","level":"error","msg":"error processing request","org_id":"anonymous","try":0,"ts":"2022-12-26T14:00:08.791952817Z"}
Dec 26 17:00:33 thanos thanos-query-frontend[8462]: {"caller":"retry.go:81","err":"rpc error: code = Code(503) desc = {\"status\":\"error\",\"errorType\":\"timeout\",\"error\":\"query timed out in expression evaluation\"}\n","level":"error","msg":"error processing request","org_id":"anonymous","try":0,"ts":"2022-12-26T14:00:33.088199575Z"}
Dec 26 17:00:43 thanos thanos-query-frontend[8462]: {"caller":"retry.go:81","err":"rpc error: code = Code(503) desc = {\"status\":\"error\",\"errorType\":\"timeout\",\"error\":\"query timed out in expression evaluation\"}\n","level":"error","msg":"error processing request","org_id":"anonymous","try":1,"ts":"2022-12-26T14:00:43.289702988Z"}
Dec 26 17:00:53 thanos thanos-query-frontend[8462]: {"caller":"retry.go:81","err":"rpc error: code = Code(503) desc = {\"status\":\"error\",\"errorType\":\"timeout\",\"error\":\"query timed out in expression evaluation\"}\n","level":"error","msg":"error processing request","org_id":"anonymous","try":2,"ts":"2022-12-26T14:00:53.337183823Z"}
Dec 26 17:01:04 thanos thanos-query-frontend[8462]: {"caller":"retry.go:81","err":"rpc error: code = Code(503) desc = {\"status\":\"error\",\"errorType\":\"timeout\",\"error\":\"query timed out in expression evaluation\"}\n","level":"error","msg":"error processing request","org_id":"anonymous","try":3,"ts":"2022-12-26T14:01:04.768546602Z"}
Dec 26 17:01:17 thanos thanos-query-frontend[8462]: {"caller":"retry.go:81","err":"rpc error: code = Code(503) desc = {\"status\":\"error\",\"errorType\":\"timeout\",\"error\":\"query timed out in expression evaluation\"}\n","level":"error","msg":"error processing request","org_id":"anonymous","try":4,"ts":"2022-12-26T14:01:17.216522678Z"}

query log

Dec 26 17:05:49 thanos thanos-query[7293]: {"caller":"engine.go:361","err":"runtime error: index out of range [28531] with length 28527","expr":"count by (node) ((kube_pod_status_phase{job=\"kube-state-metrics\",phase=\"Running\"} == 1) * on (instance, pod, namespace, cluster) group_left (node) topk by (cluster, instance, namespace, pod) (1, kube_pod_info{job=\"kube-state-metrics\"})) / max by (node) (kube_node_status_capacity{job=\"kube-state-metrics\",resource=\"pods\"} != 1) > 0.95","level":"error","msg":"runtime panic in engine","stacktrace":"goroutine 514304827 [running]:\ngithub.com/thanos-community/promql-engine/engine.recoverEngine({0x2bd9640, 0xc00069fa40}, {0x2bf78e0, 0xc054dff900}, 0xc142a6d340)\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/engine/engine.go:359 +0xc6\npanic({0x2507ae0, 0xc11b2461c8})\n\t/usr/local/go/src/runtime/panic.go:884 +0x212\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).join(0xc0af8fc5a0?, 0xc32efd0000?, 0x6f80?, 0xc2ce7b5000?, 0xc36513c690, {0xc0af890ac0, 0x1, 0x1})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:242 +0x71a\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).initOutputs(0xc0af8fc5a0, {0x2bf41c0, 0xc142a6d380})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:109 +0x2a5\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).Series.func1()\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:78 +0x2e\nsync.(*Once).doSlow(0x10?, 0xc6b58b2758?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).Series(0xc0af8fc5a0?, {0x2bf41c0?, 0xc142a6d380?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:78 +0x7e\ngithub.com/thanos-community/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables(0xc745e8f6c0, {0x2bf41c0?, 0xc142a6d380?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/aggregate/hashaggregate.go:200 +0x51\ngithub.com/thanos-community/promql-engine/execution/aggregate.(*aggregate).initializeTables(0xc745e8f6c0, {0x2bf41c0, 0xc142a6d380})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/aggregate/hashaggregate.go:168 +0x45\ngithub.com/thanos-community/promql-engine/execution/aggregate.(*aggregate).Series.func1()\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/aggregate/hashaggregate.go:91 +0x2e\nsync.(*Once).doSlow(0x28?, 0xc0aa25c118?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/thanos-community/promql-engine/execution/aggregate.(*aggregate).Series(0xc745e8f6c0?, {0x2bf41c0?, 0xc142a6d380?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/aggregate/hashaggregate.go:91 +0x7e\ngithub.com/thanos-community/promql-engine/execution/exchange.(*concurrencyOperator).Series(0xc054dff800?, {0x2bf41c0?, 0xc142a6d380?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/exchange/concurrent.go:41 +0x2c\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).initOutputs(0xc0b4382460, {0x2bf41c0, 0xc142a6d380})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:88 +0x52\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).Series.func1()\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:78 +0x2e\nsync.(*Once).doSlow(0x20301c?, 0x20301c?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).Series(0xc0b4382460?, {0x2bf41c0?, 0xc142a6d380?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:78 +0x7e\ngithub.com/thanos-community/promql-engine/execution/binary.(*scalarOperator).loadSeries(0xc077369570, {0x2bf41c0?, 0xc142a6d380?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/scalar.go:148 +0x4f\ngithub.com/thanos-community/promql-engine/execution/binary.(*scalarOperator).Series.func1()\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/scalar.go:78 +0x2e\nsync.(*Once).doSlow(0x10?, 0xc0aa25c6b0?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/thanos-community/promql-engine/execution/binary.(*scalarOperator).Series(0xc077369570?, {0x2bf41c0?, 0xc142a6d380?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/scalar.go:78 +0x76\ngithub.com/thanos-community/promql-engine/engine.(*compatibilityQuery).Exec(0xc0af8fe3c0, {0x2bf4268, 0xc0c65523f0})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/engine/engine.go:204 +0x1c2\ngithub.com/thanos-io/thanos/pkg/api/query.(*QueryAPI).query(0xc000a89a40, 0xc0af8be300)\n\t/app/pkg/api/query/v1.go:441 +0x92b\ngithub.com/thanos-io/thanos/pkg/api.GetInstr.func1.1({0x2bee450, 0xc0a5d79060}, 0xc04623d320?)\n\t/app/pkg/api/api.go:211 +0x50\nnet/http.HandlerFunc.ServeHTTP(0xc0a5d2a8c0?, {0x2bee450?, 0xc0a5d79060?}, 0x2bce434?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/logging.(*HTTPServerMiddleware).HTTPMiddleware.func1({0x2be99e0?, 0xc077369420}, 0xc0af8be300)\n\t/app/pkg/logging/http.go:73 +0x2d8\nnet/http.HandlerFunc.ServeHTTP(0x262d920?, {0x2be99e0?, 0xc077369420?}, 0x2bceca8?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/server/http/middleware.RequestID.func1({0x2be99e0, 0xc077369420}, 0xc0af8be200)\n\t/app/pkg/server/http/middleware/request_id.go:40 +0x542\nnet/http.HandlerFunc.ServeHTTP(0x2184f60?, {0x2be99e0?, 0xc077369420?}, 0x4?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1({0x2beddc0, 0xc0a5d79040}, 0xc4b071d390?)\n\t/go/pkg/mod/github.com/!n!y!times/gziphandler@v1.1.1/gzip.go:338 +0x26f\nnet/http.HandlerFunc.ServeHTTP(0x2?, {0x2beddc0?, 0xc0a5d79040?}, 0x0?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/extprom/http.httpInstrumentationHandler.func1({0x7fb68d310e38?, 0xc0a5d55450}, 0xc0af8be200)\n\t/app/pkg/extprom/http/instrument_server.go:75 +0x10b\nnet/http.HandlerFunc.ServeHTTP(0x7fb68d310e38?, {0x7fb68d310e38?, 0xc0a5d55450?}, 0xc0c6552180?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1({0x7fb68d310e38?, 0xc0a5d55400?}, 0xc0af8be200)\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:288 +0xc5\nnet/http.HandlerFunc.ServeHTTP(0x7fb68d310e38?, {0x7fb68d310e38?, 0xc0a5d55400?}, 0x0?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x7fb68d310e38?, 0xc0a5d553b0?}, 0xc0af8be200)\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:146 +0xb8\nnet/http.HandlerFunc.ServeHTTP(0x22c9b80?, {0x7fb68d310e38?, 0xc0a5d553b0?}, 0x6?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/extprom/http.instrumentHandlerInFlight.func1({0x7fb68d310e38, 0xc0a5d553b0}, 0xc0af8be200)\n\t/app/pkg/extprom/http/instrument_server.go:162 +0x169\nnet/http.HandlerFunc.ServeHTTP(0x2bf1390?, {0x7fb68d310e38?, 0xc0a5d553b0?}, 0xc2a19eb698?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1({0x2bf1390?, 0xc0164320e0?}, 0xc0af8be200)\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:238 +0xc5\nnet/http.HandlerFunc.ServeHTTP(0x262d920?, {0x2bf1390?, 0xc0164320e0?}, 0x417d240?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/tracing.HTTPMiddleware.func1({0x2bf1390, 0xc0164320e0}, 0xc0af8be100)\n\t/app/pkg/tracing/http.go:62 +0x9a2\ngithub.com/prometheus/common/route.(*Router).handle.func1({0x2bf1390, 0xc0164320e0}, 0xc0af8be000, {0x0, 0x0, 0x478d4e?})\n\t/go/pkg/mod/github.com/prometheus/common@v0.37.1/route/route.go:83 +0x18d\ngithub.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc000b8a360, {0x2bf1390, 0xc0164320e0}, 0xc0af8be000)\n\t/go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0x81c\ngithub.com/prometheus/common/route.(*Router).ServeHTTP(0xc0164320e0?, {0x2bf1390?, 0xc0164320e0?}, 0x26a03c8?)\n\t/go/pkg/mod/github.com/prometheus/common@v0.37.1/route/route.go:126 +0x26\nnet/http.(*ServeMux).ServeHTTP(0x0?, {0x2bf1390, 0xc0164320e0}, 0xc0af8be000)\n\t/usr/local/go/src/net/http/server.go:2487 +0x149\nnet/http.serverHandler.ServeHTTP({0x2be7c00?}, {0x2bf1390, 0xc0164320e0}, 0xc0af8be000)\n\t/usr/local/go/src/net/http/server.go:2947 +0x30c\nnet/http.(*conn).serve(0xc05f840780, {0x2bf4268, 0xc0009044b0})\n\t/usr/local/go/src/net/http/server.go:1991 +0x607\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:3102 +0x4db\n","ts":"2022-12-26T14:05:49.893591238Z"}
Dec 26 17:05:51 thanos thanos-query[7293]: {"caller":"engine.go:361","err":"runtime error: index out of range [28534] with length 28527","expr":"count by (node) ((kube_pod_status_phase{job=\"kube-state-metrics\",phase=\"Running\"} == 1) * on (instance, pod, namespace, cluster) group_left (node) topk by (cluster, instance, namespace, pod) (1, kube_pod_info{job=\"kube-state-metrics\"})) / max by (node) (kube_node_status_capacity{job=\"kube-state-metrics\",resource=\"pods\"} != 1) > 0.95","level":"error","msg":"runtime panic in engine","stacktrace":"goroutine 514338125 [running]:\ngithub.com/thanos-community/promql-engine/engine.recoverEngine({0x2bd9640, 0xc00069fa40}, {0x2bf78e0, 0xc7ea7de240}, 0xc78cfe4640)\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/engine/engine.go:359 +0xc6\npanic({0x2507ae0, 0xcb10e36570})\n\t/usr/local/go/src/runtime/panic.go:884 +0x212\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).join(0xc15b505040?, 0xc2a4164000?, 0x6f80?, 0xc0c26bc000?, 0xc2d0519a40, {0xc15936a6e0, 0x1, 0x1})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:242 +0x71a\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).initOutputs(0xc15b505040, {0x2bf41c0, 0xc78cfe4680})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:109 +0x2a5\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).Series.func1()\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:78 +0x2e\nsync.(*Once).doSlow(0x10?, 0x7fb6b4921c58?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).Series(0xc15b505040?, {0x2bf41c0?, 0xc78cfe4680?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:78 +0x7e\ngithub.com/thanos-community/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables(0xc05d9ec680, {0x2bf41c0?, 0xc78cfe4680?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/aggregate/hashaggregate.go:200 +0x51\ngithub.com/thanos-community/promql-engine/execution/aggregate.(*aggregate).initializeTables(0xc05d9ec680, {0x2bf41c0, 0xc78cfe4680})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/aggregate/hashaggregate.go:168 +0x45\ngithub.com/thanos-community/promql-engine/execution/aggregate.(*aggregate).Series.func1()\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/aggregate/hashaggregate.go:91 +0x2e\nsync.(*Once).doSlow(0x248cb80?, 0xc15b512b01?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/thanos-community/promql-engine/execution/aggregate.(*aggregate).Series(0xc05d9ec680?, {0x2bf41c0?, 0xc78cfe4680?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/aggregate/hashaggregate.go:91 +0x7e\ngithub.com/thanos-community/promql-engine/execution/exchange.(*concurrencyOperator).Series(0xc7457e7fc0?, {0x2bf41c0?, 0xc78cfe4680?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/exchange/concurrent.go:41 +0x2c\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).initOutputs(0xc15b50af00, {0x2bf41c0, 0xc78cfe4680})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:88 +0x52\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).Series.func1()\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:78 +0x2e\nsync.(*Once).doSlow(0x2031d8?, 0x2031d8?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).Series(0xc15b50af00?, {0x2bf41c0?, 0xc78cfe4680?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:78 +0x7e\ngithub.com/thanos-community/promql-engine/execution/binary.(*scalarOperator).loadSeries(0xc05deb7a40, {0x2bf41c0?, 0xc78cfe4680?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/scalar.go:148 +0x4f\ngithub.com/thanos-community/promql-engine/execution/binary.(*scalarOperator).Series.func1()\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/scalar.go:78 +0x2e\nsync.(*Once).doSlow(0x23352a0?, 0xc78cfe4601?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/thanos-community/promql-engine/execution/binary.(*scalarOperator).Series(0xc05deb7a40?, {0x2bf41c0?, 0xc78cfe4680?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/scalar.go:78 +0x76\ngithub.com/thanos-community/promql-engine/engine.(*compatibilityQuery).Exec(0xc15b514370, {0x2bf4268, 0xc57fb5b110})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/engine/engine.go:204 +0x1c2\ngithub.com/thanos-io/thanos/pkg/api/query.(*QueryAPI).query(0xc000a89a40, 0xc6fe7da900)\n\t/app/pkg/api/query/v1.go:441 +0x92b\ngithub.com/thanos-io/thanos/pkg/api.GetInstr.func1.1({0x2bee450, 0xc1593742c0}, 0xc76394e480?)\n\t/app/pkg/api/api.go:211 +0x50\nnet/http.HandlerFunc.ServeHTTP(0xc0ca598360?, {0x2bee450?, 0xc1593742c0?}, 0x2bce434?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/logging.(*HTTPServerMiddleware).HTTPMiddleware.func1({0x2be99e0?, 0xc05deb78f0}, 0xc6fe7da900)\n\t/app/pkg/logging/http.go:73 +0x2d8\nnet/http.HandlerFunc.ServeHTTP(0x2bf4268?, {0x2be99e0?, 0xc05deb78f0?}, 0x2bceca8?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/server/http/middleware.RequestID.func1({0x2be99e0, 0xc05deb78f0}, 0xc6fe7da800)\n\t/app/pkg/server/http/middleware/request_id.go:40 +0x542\nnet/http.HandlerFunc.ServeHTTP(0x2184f60?, {0x2be99e0?, 0xc05deb78f0?}, 0x4?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1({0x2beddc0, 0xc1593742a0}, 0x490001?)\n\t/go/pkg/mod/github.com/!n!y!times/gziphandler@v1.1.1/gzip.go:338 +0x26f\nnet/http.HandlerFunc.ServeHTTP(0xc0028d8a90?, {0x2beddc0?, 0xc1593742a0?}, 0xc050032b40?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/extprom/http.httpInstrumentationHandler.func1({0x7fb68d310e38?, 0xc05de99400}, 0xc6fe7da800)\n\t/app/pkg/extprom/http/instrument_server.go:75 +0x10b\nnet/http.HandlerFunc.ServeHTTP(0x7fb68d310e38?, {0x7fb68d310e38?, 0xc05de99400?}, 0xc57fb5ae70?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1({0x7fb68d310e38?, 0xc05de993b0?}, 0xc6fe7da800)\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:288 +0xc5\nnet/http.HandlerFunc.ServeHTTP(0x7fb68d310e38?, {0x7fb68d310e38?, 0xc05de993b0?}, 0x0?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x7fb68d310e38?, 0xc05de99360?}, 0xc6fe7da800)\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:146 +0xb8\nnet/http.HandlerFunc.ServeHTTP(0x22c9b80?, {0x7fb68d310e38?, 0xc05de99360?}, 0x6?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/extprom/http.instrumentHandlerInFlight.func1({0x7fb68d310e38, 0xc05de99360}, 0xc6fe7da800)\n\t/app/pkg/extprom/http/instrument_server.go:162 +0x169\nnet/http.HandlerFunc.ServeHTTP(0x2bf1390?, {0x7fb68d310e38?, 0xc05de99360?}, 0xc146575698?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1({0x2bf1390?, 0xc0082c4000?}, 0xc6fe7da800)\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:238 +0xc5\nnet/http.HandlerFunc.ServeHTTP(0x2bf4268?, {0x2bf1390?, 0xc0082c4000?}, 0x417d240?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/tracing.HTTPMiddleware.func1({0x2bf1390, 0xc0082c4000}, 0xc6fe7da700)\n\t/app/pkg/tracing/http.go:62 +0x9a2\ngithub.com/prometheus/common/route.(*Router).handle.func1({0x2bf1390, 0xc0082c4000}, 0xc6fe7da600, {0x0, 0x0, 0x478d4e?})\n\t/go/pkg/mod/github.com/prometheus/common@v0.37.1/route/route.go:83 +0x18d\ngithub.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc000b8a360, {0x2bf1390, 0xc0082c4000}, 0xc6fe7da600)\n\t/go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0x81c\ngithub.com/prometheus/common/route.(*Router).ServeHTTP(0xc0082c4000?, {0x2bf1390?, 0xc0082c4000?}, 0xc0ca598300?)\n\t/go/pkg/mod/github.com/prometheus/common@v0.37.1/route/route.go:126 +0x26\nnet/http.(*ServeMux).ServeHTTP(0x0?, {0x2bf1390, 0xc0082c4000}, 0xc6fe7da600)\n\t/usr/local/go/src/net/http/server.go:2487 +0x149\nnet/http.serverHandler.ServeHTTP({0x2be7c00?}, {0x2bf1390, 0xc0082c4000}, 0xc6fe7da600)\n\t/usr/local/go/src/net/http/server.go:2947 +0x30c\nnet/http.(*conn).serve(0xc01a259180, {0x2bf4268, 0xc0009044b0})\n\t/usr/local/go/src/net/http/server.go:1991 +0x607\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:3102 +0x4db\n","ts":"2022-12-26T14:05:51.818271135Z"}

goroutine and memory graphs query

query frontend

Anything else we need to know: Query response time

same issue with store component pprof goroutine dump

goroutine profile: total 19168
9927 @ 0x43cd56 0x44dcfe 0x44dcd5 0x46a225 0x478725 0x18a8cf3 0x18a8ccf 0x18aa546 0x197c79a 0x196c455 0x196c05d 0x19915f9 0x19910e9 0x46e341
#   0x46a224    sync.runtime_SemacquireMutex+0x24                               /usr/local/go/src/runtime/sema.go:77
#   0x478724    sync.(*Mutex).lockSlow+0x164                                    /usr/local/go/src/sync/mutex.go:171
#   0x18a8cf2   sync.(*Mutex).Lock+0xf2                                     /usr/local/go/src/sync/mutex.go:90
#   0x18a8cce   github.com/thanos-io/thanos/pkg/store/cache.(*InMemoryIndexCache).get+0xce          /app/pkg/store/cache/inmemory.go:198
#   0x18aa545   github.com/thanos-io/thanos/pkg/store/cache.(*InMemoryIndexCache).FetchMultiSeries+0x125    /app/pkg/store/cache/inmemory.go:325
#   0x197c799   github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).PreloadSeries+0x1f9          /app/pkg/store/bucket.go:2472
#   0x196c454   github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).nextBatch+0x154          /app/pkg/store/bucket.go:962
#   0x196c05c   github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).Recv+0x5c                /app/pkg/store/bucket.go:922
#   0x19915f8   github.com/thanos-io/thanos/pkg/store.newLazyRespSet.func1.3+0x3f8              /app/pkg/store/proxy_heap.go:424
#   0x19910e8   github.com/thanos-io/thanos/pkg/store.newLazyRespSet.func1+0x228                /app/pkg/store/proxy_heap.go:479

8839 @ 0x43cd56 0x44dcfe 0x44dcd5 0x46a225 0x478725 0x18a8cf3 0x18a8ccf 0x18aa0fd 0x197ac3c 0x197953b 0x196be3a 0x19714b9 0xa553c4 0x46e341
#   0x46a224    sync.runtime_SemacquireMutex+0x24                               /usr/local/go/src/runtime/sema.go:77
#   0x478724    sync.(*Mutex).lockSlow+0x164                                    /usr/local/go/src/sync/mutex.go:171
#   0x18a8cf2   sync.(*Mutex).Lock+0xf2                                     /usr/local/go/src/sync/mutex.go:90
#   0x18a8cce   github.com/thanos-io/thanos/pkg/store/cache.(*InMemoryIndexCache).get+0xce          /app/pkg/store/cache/inmemory.go:198
#   0x18aa0fc   github.com/thanos-io/thanos/pkg/store/cache.(*InMemoryIndexCache).FetchMultiPostings+0x17c  /app/pkg/store/cache/inmemory.go:302
#   0x197ac3b   github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings+0x23b          /app/pkg/store/bucket.go:2243
#   0x197953a   github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings+0x77a       /app/pkg/store/bucket.go:2094
#   0x196be39   github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings+0x79          /app/pkg/store/bucket.go:899
#   0x19714b8   github.com/thanos-io/thanos/pkg/store.(*BucketStore).Series.func2+0x378             /app/pkg/store/bucket.go:1233
#   0xa553c3    golang.org/x/sync/errgroup.(*Group).Go.func1+0x63                       /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75

103 @ 0x43cd56 0x44cc3c 0x7bb395 0x46e341
#   0x7bb394    net/http.(*persistConn).writeLoop+0xf4  /usr/local/go/src/net/http/transport.go:2392

100 @ 0x43cd56 0x435557 0x468829 0x4df112 0x4e047a 0x4e0468 0x587549 0x59b245 0x6eaafd 0x51df78 0x6eace5 0x6e8136 0x6edd8f 0x6edd90 0x7b88ee 0x56253f 0x56269d 0x7b970c 0x46e341
#   0x468828    internal/poll.runtime_pollWait+0x88     /usr/local/go/src/runtime/netpoll.go:305
#   0x4df111    internal/poll.(*pollDesc).wait+0x31     /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#   0x4e0479    internal/poll.(*pollDesc).waitRead+0x259    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#   0x4e0467    internal/poll.(*FD).Read+0x247          /usr/local/go/src/internal/poll/fd_unix.go:167
#   0x587548    net.(*netFD).Read+0x28              /usr/local/go/src/net/fd_posix.go:55
#   0x59b244    net.(*conn).Read+0x44               /usr/local/go/src/net/net.go:183
#   0x6eaafc    crypto/tls.(*atLeastReader).Read+0x3c       /usr/local/go/src/crypto/tls/conn.go:787
#   0x51df77    bytes.(*Buffer).ReadFrom+0x97           /usr/local/go/src/bytes/buffer.go:202
#   0x6eace4    crypto/tls.(*Conn).readFromUntil+0xe4       /usr/local/go/src/crypto/tls/conn.go:809
#   0x6e8135    crypto/tls.(*Conn).readRecordOrCCS+0x115    /usr/local/go/src/crypto/tls/conn.go:616
#   0x6edd8e    crypto/tls.(*Conn).readRecord+0x16e     /usr/local/go/src/crypto/tls/conn.go:582
#   0x6edd8f    crypto/tls.(*Conn).Read+0x16f           /usr/local/go/src/crypto/tls/conn.go:1287
#   0x7b88ed    net/http.(*persistConn).Read+0x4d       /usr/local/go/src/net/http/transport.go:1929
#   0x56253e    bufio.(*Reader).fill+0xfe           /usr/local/go/src/bufio/bufio.go:106
#   0x56269c    bufio.(*Reader).Peek+0x5c           /usr/local/go/src/bufio/bufio.go:144
#   0x7b970b    net/http.(*persistConn).readLoop+0x1ab      /usr/local/go/src/net/http/transport.go:2093

32 @ 0x43cd56 0x44cc3c 0x14f6477 0x14f6af3 0x14f694c 0x1971a82 0x14a69a3 0x196e8bd 0x1440130 0x1362d4c 0x146755a 0x1362d4c 0x14a505b 0x146755a 0x1362d4c 0x146755a 0x1dac9a9 0x146755a 0x1db16de 0x146755a 0x14673fe 0xecff7b 0xed178a 0xecacf8 0x46e341
#   0x14f6476   github.com/prometheus/prometheus/util/gate.(*Gate).Start+0x76                           /go/pkg/mod/github.com/prometheus/prometheus@v0.40.7/util/gate/gate.go:33
#   0x14f6af2   github.com/thanos-io/thanos/pkg/gate.(*instrumentedInFlightGate).Start+0x32                 /app/pkg/gate/gate.go:149
#   0x14f694b   github.com/thanos-io/thanos/pkg/gate.(*instrumentedDurationGate).Start+0xab                 /app/pkg/gate/gate.go:125
#   0x1971a81   github.com/thanos-io/thanos/pkg/store.(*BucketStore).Series.func1+0x61                      /app/pkg/store/bucket.go:1147
#   0x14a69a2   github.com/thanos-io/thanos/pkg/tracing.DoInSpan+0xa2                               /app/pkg/tracing/tracing.go:95
#   0x196e8bc   github.com/thanos-io/thanos/pkg/store.(*BucketStore).Series+0xfc                        /app/pkg/store/bucket.go:1146
#   0x144012f   github.com/thanos-io/thanos/pkg/store/storepb._Store_Series_Handler+0xcf                    /app/pkg/store/storepb/rpc.pb.go:1081
#   0x1362d4b   github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.StreamServerInterceptor.func1+0x2ab        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35
#   0x1467559   github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1+0x39                /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#   0x1362d4b   github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.StreamServerInterceptor.func1+0x2ab        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35
#   0x14a505a   github.com/thanos-io/thanos/pkg/tracing.StreamServerInterceptor.func1+0x19a                 /app/pkg/tracing/grpc.go:42
#   0x1467559   github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1+0x39                /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#   0x1362d4b   github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.StreamServerInterceptor.func1+0x2ab        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35
#   0x1467559   github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1+0x39                /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#   0x1dac9a8   github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1+0x108       /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121
#   0x1467559   github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1+0x39                /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#   0x1db16dd   github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.StreamServerInterceptor.func1+0x9d    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/interceptors.go:45
#   0x1467559   github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1+0x39                /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#   0x14673fd   github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1+0xbd                    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:60
#   0xecff7a    google.golang.org/grpc.(*Server).processStreamingRPC+0xf1a                          /go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1548
#   0xed1789    google.golang.org/grpc.(*Server).handleStream+0x9e9                             /go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1623
#   0xecacf7    google.golang.org/grpc.(*Server).serveStreams.func1.2+0x97                          /go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921

30 @ 0x43cd56 0x44dcfe 0x44dcd5 0x46a105 0x47a1b2 0xa55227 0x197e798 0x196c8af 0x196c05d 0x19915f9 0x19910e9 0x46e341
#   0x46a104    sync.runtime_Semacquire+0x24                            /usr/local/go/src/runtime/sema.go:62
#   0x47a1b1    sync.(*WaitGroup).Wait+0x51                         /usr/local/go/src/sync/waitgroup.go:139
#   0xa55226    golang.org/x/sync/errgroup.(*Group).Wait+0x26                   /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:53
#   0x197e797   github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).load+0x197       /app/pkg/store/bucket.go:2786
#   0x196c8ae   github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).nextBatch+0x5ae  /app/pkg/store/bucket.go:1015
#   0x196c05c   github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).Recv+0x5c        /app/pkg/store/bucket.go:922
#   0x19915f8   github.com/thanos-io/thanos/pkg/store.newLazyRespSet.func1.3+0x3f8      /app/pkg/store/proxy_heap.go:424
#   0x19910e8   github.com/thanos-io/thanos/pkg/store.newLazyRespSet.func1+0x228        /app/pkg/store/proxy_heap.go:479

27 @ 0x43cd56 0x406d5b 0x406858 0x12272f3 0x1227705 0x13156bd 0x1315ae8 0xd1683d 0xd18cc3 0x19789b0 0x197eff3 0x197eddf 0xa553c4 0x46e341
#   0x12272f2   github.com/minio/minio-go/v7.(*Object).doGetRequest+0x112           /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:312
#   0x1227704   github.com/minio/minio-go/v7.(*Object).Read+0x244               /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:386
#   0x13156bc   github.com/thanos-io/objstore/providers/s3.(*Bucket).getRange+0x1fc     /go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221205132204-5aafc0079f06/providers/s3/s3.go:442
#   0x1315ae7   github.com/thanos-io/objstore/providers/s3.(*Bucket).GetRange+0x27      /go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221205132204-5aafc0079f06/providers/s3/s3.go:459
#   0xd1683c    github.com/thanos-io/objstore.(*metricBucket).GetRange+0xbc         /go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221205132204-5aafc0079f06/objstore.go:526
#   0xd18cc2    github.com/thanos-io/objstore.TracingBucket.GetRange+0x242          /go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221205132204-5aafc0079f06/tracing.go:50
#   0x19789af   github.com/thanos-io/thanos/pkg/store.(*bucketBlock).chunkRangeReader+0xaf  /app/pkg/store/bucket.go:1974
#   0x197eff2   github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).loadChunks+0x132 /app/pkg/store/bucket.go:2798
#   0x197edde   github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).load.func3+0xfe  /app/pkg/store/bucket.go:2782
#   0xa553c3    golang.org/x/sync/errgroup.(*Group).Go.func1+0x63               /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75

25 @ 0x43cd56 0x44cc3c 0x7bc254 0x7afd9a 0x7967f9 0x7534d7 0x752d5b 0x7550dc 0x1256993 0x1256988 0x1257712 0x1229a9c 0x1226250 0x46e341
#   0x7bc253    net/http.(*persistConn).roundTrip+0x973             /usr/local/go/src/net/http/transport.go:2620
#   0x7afd99    net/http.(*Transport).roundTrip+0x7b9               /usr/local/go/src/net/http/transport.go:595
#   0x7967f8    net/http.(*Transport).RoundTrip+0x18                /usr/local/go/src/net/http/roundtrip.go:17
#   0x7534d6    net/http.send+0x5f6                     /usr/local/go/src/net/http/client.go:251
#   0x752d5a    net/http.(*Client).send+0x9a                    /usr/local/go/src/net/http/client.go:175
#   0x7550db    net/http.(*Client).do+0x8fb                 /usr/local/go/src/net/http/client.go:715
#   0x1256992   net/http.(*Client).Do+0xb2                  /usr/local/go/src/net/http/client.go:581
#   0x1256987   github.com/minio/minio-go/v7.(*Client).do+0xa7          /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api.go:505
#   0x1257711   github.com/minio/minio-go/v7.(*Client).executeMethod+0x951  /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api.go:621
#   0x1229a9b   github.com/minio/minio-go/v7.(*Client).getObject+0x37b      /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:666
#   0x122624f   github.com/minio/minio-go/v7.(*Client).GetObject.func1+0x30f    /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:93

24 @ 0x43cd56 0x44cc3c 0x12611aa 0x46e341
#   0x12611a9   github.com/minio/minio-go/v7.(*Client).newRetryTimer.func2+0x1c9    /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/retry.go:82

20 @ 0x43cd56 0x44dcfe 0x44dcd5 0x46a105 0x47a1b2 0xa55227 0x19706a6 0x14a69a3 0x196f405 0x1440130 0x1362d4c 0x146755a 0x1362d4c 0x14a505b 0x146755a 0x1362d4c 0x146755a 0x1dac9a9 0x146755a 0x1db16de 0x146755a 0x14673fe 0xecff7b 0xed178a 0xecacf8 0x46e341
#   0x46a104    sync.runtime_Semacquire+0x24                                            /usr/local/go/src/runtime/sema.go:62
#   0x47a1b1    sync.(*WaitGroup).Wait+0x51                                         /usr/local/go/src/sync/waitgroup.go:139
#   0xa55226    golang.org/x/sync/errgroup.(*Group).Wait+0x26                                   /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:53
#   0x19706a5   github.com/thanos-io/thanos/pkg/store.(*BucketStore).Series.func4+0x25                      /app/pkg/store/bucket.go:1299
#   0x14a69a2   github.com/thanos-io/thanos/pkg/tracing.DoInSpan+0xa2                               /app/pkg/tracing/tracing.go:95
#   0x196f404   github.com/thanos-io/thanos/pkg/store.(*BucketStore).Series+0xc44                       /app/pkg/store/bucket.go:1298
#   0x144012f   github.com/thanos-io/thanos/pkg/store/storepb._Store_Series_Handler+0xcf                    /app/pkg/store/storepb/rpc.pb.go:1081
#   0x1362d4b   github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.StreamServerInterceptor.func1+0x2ab        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35
#   0x1467559   github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1+0x39                /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#   0x1362d4b   github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.StreamServerInterceptor.func1+0x2ab        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35
#   0x14a505a   github.com/thanos-io/thanos/pkg/tracing.StreamServerInterceptor.func1+0x19a                 /app/pkg/tracing/grpc.go:42
#   0x1467559   github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1+0x39                /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#   0x1362d4b   github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.StreamServerInterceptor.func1+0x2ab        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35
#   0x1467559   github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1+0x39                /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#   0x1dac9a8   github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1+0x108       /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121
#   0x1467559   github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1+0x39                /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#   0x1db16dd   github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.StreamServerInterceptor.func1+0x9d    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/interceptors.go:45
#   0x1467559   github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1+0x39                /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51
#   0x14673fd   github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1+0xbd                    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:60
#   0xecff7a    google.golang.org/grpc.(*Server).processStreamingRPC+0xf1a                          /go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1548
#   0xed1789    google.golang.org/grpc.(*Server).handleStream+0x9e9                             /go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1623
#   0xecacf7    google.golang.org/grpc.(*Server).serveStreams.func1.2+0x97                          /go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921

3 @ 0x43cd56 0x435557 0x468829 0x4df112 0x4e047a 0x4e0468 0x587549 0x59b245 0x562b1b 0x4d8f3a 0xa886ae 0xa88668 0xa88f15 0xe9d494 0xeca942 0xeca106 0x46e341
#   0x468828    internal/poll.runtime_pollWait+0x88                     /usr/local/go/src/runtime/netpoll.go:305
#   0x4df111    internal/poll.(*pollDesc).wait+0x31                     /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#   0x4e0479    internal/poll.(*pollDesc).waitRead+0x259                    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#   0x4e0467    internal/poll.(*FD).Read+0x247                          /usr/local/go/src/internal/poll/fd_unix.go:167
#   0x587548    net.(*netFD).Read+0x28                              /usr/local/go/src/net/fd_posix.go:55
#   0x59b244    net.(*conn).Read+0x44                               /usr/local/go/src/net/net.go:183
#   0x562b1a    bufio.(*Reader).Read+0x1ba                          /usr/local/go/src/bufio/bufio.go:237
#   0x4d8f39    io.ReadAtLeast+0x99                             /usr/local/go/src/io/io.go:332
#   0xa886ad    io.ReadFull+0x6d                                /usr/local/go/src/io/io.go:351
#   0xa88667    golang.org/x/net/http2.readFrameHeader+0x27                 /go/pkg/mod/golang.org/x/net@v0.4.0/http2/frame.go:237
#   0xa88f14    golang.org/x/net/http2.(*Framer).ReadFrame+0x94                 /go/pkg/mod/golang.org/x/net@v0.4.0/http2/frame.go:498
#   0xe9d493    google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams+0x173    /go/pkg/mod/google.golang.org/grpc@v1.45.0/internal/transport/http2_server.go:605
#   0xeca941    google.golang.org/grpc.(*Server).serveStreams+0x141             /go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:905
#   0xeca105    google.golang.org/grpc.(*Server).handleRawConn.func1+0x45           /go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:847

3 @ 0x43cd56 0x44cc3c 0x1261125 0x46e341
#   0x1261124   github.com/minio/minio-go/v7.(*Client).newRetryTimer.func2+0x144    /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/retry.go:76

3 @ 0x43cd56 0x44cc3c 0x7ba2e5 0x46e341
#   0x7ba2e4    net/http.(*persistConn).readLoop+0xd84  /usr/local/go/src/net/http/transport.go:2213

3 @ 0x43cd56 0x44cc3c 0xe835d5 0xe83ce5 0xe9a68e 0x46e341
#   0xe835d4    google.golang.org/grpc/internal/transport.(*controlBuffer).get+0x114    /go/pkg/mod/google.golang.org/grpc@v1.45.0/internal/transport/controlbuf.go:407
#   0xe83ce4    google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x84   /go/pkg/mod/google.golang.org/grpc@v1.45.0/internal/transport/controlbuf.go:534
#   0xe9a68d    google.golang.org/grpc/internal/transport.NewServerTransport.func2+0xcd /go/pkg/mod/google.golang.org/grpc@v1.45.0/internal/transport/http2_server.go:326

3 @ 0x43cd56 0x44cc3c 0xea1853 0x46e341
#   0xea1852    google.golang.org/grpc/internal/transport.(*http2Server).keepalive+0x232    /go/pkg/mod/google.golang.org/grpc@v1.45.0/internal/transport/http2_server.go:1125

2 @ 0x43cd56 0x406d5b 0x406858 0x12272f3 0x1227705 0xd177f2 0xd19dd2 0x562b1b 0x4d8f3a 0x197f65d 0x197f63d 0x197eddf 0xa553c4 0x46e341
#   0x12272f2   github.com/minio/minio-go/v7.(*Object).doGetRequest+0x112           /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:312
#   0x1227704   github.com/minio/minio-go/v7.(*Object).Read+0x244               /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:386
#   0xd177f1    github.com/thanos-io/objstore.(*timingReadCloser).Read+0x31         /go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221205132204-5aafc0079f06/objstore.go:650
#   0xd19dd1    github.com/thanos-io/objstore.(*tracingReadCloser).Read+0x31            /go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221205132204-5aafc0079f06/tracing.go:138
#   0x562b1a    bufio.(*Reader).Read+0x1ba                          /usr/local/go/src/bufio/bufio.go:237
#   0x4d8f39    io.ReadAtLeast+0x99                             /usr/local/go/src/io/io.go:332
#   0x197f65c   io.ReadFull+0x79c                               /usr/local/go/src/io/io.go:351
#   0x197f63c   github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).loadChunks+0x77c /app/pkg/store/bucket.go:2856
#   0x197edde   github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).load.func3+0xfe  /app/pkg/store/bucket.go:2782
#   0xa553c3    golang.org/x/sync/errgroup.(*Group).Go.func1+0x63               /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75

2 @ 0x43cd56 0x435557 0x468829 0x4df112 0x4e047a 0x4e0468 0x587549 0x59b245 0x6eaafd 0x51df78 0x6eace5 0x6e8b48 0x6edd8f 0x6edd90 0x7b88ee 0x562a66 0x4d9625 0x7ad3fc 0x7ad2c5 0x7bcfc2 0x12643bd 0x1226ad0 0x46e341
#   0x468828    internal/poll.runtime_pollWait+0x88             /usr/local/go/src/runtime/netpoll.go:305
#   0x4df111    internal/poll.(*pollDesc).wait+0x31             /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#   0x4e0479    internal/poll.(*pollDesc).waitRead+0x259            /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#   0x4e0467    internal/poll.(*FD).Read+0x247                  /usr/local/go/src/internal/poll/fd_unix.go:167
#   0x587548    net.(*netFD).Read+0x28                      /usr/local/go/src/net/fd_posix.go:55
#   0x59b244    net.(*conn).Read+0x44                       /usr/local/go/src/net/net.go:183
#   0x6eaafc    crypto/tls.(*atLeastReader).Read+0x3c               /usr/local/go/src/crypto/tls/conn.go:787
#   0x51df77    bytes.(*Buffer).ReadFrom+0x97                   /usr/local/go/src/bytes/buffer.go:202
#   0x6eace4    crypto/tls.(*Conn).readFromUntil+0xe4               /usr/local/go/src/crypto/tls/conn.go:809
#   0x6e8b47    crypto/tls.(*Conn).readRecordOrCCS+0xb27            /usr/local/go/src/crypto/tls/conn.go:661
#   0x6edd8e    crypto/tls.(*Conn).readRecord+0x16e             /usr/local/go/src/crypto/tls/conn.go:582
#   0x6edd8f    crypto/tls.(*Conn).Read+0x16f                   /usr/local/go/src/crypto/tls/conn.go:1287
#   0x7b88ed    net/http.(*persistConn).Read+0x4d               /usr/local/go/src/net/http/transport.go:1929
#   0x562a65    bufio.(*Reader).Read+0x105                  /usr/local/go/src/bufio/bufio.go:223
#   0x4d9624    io.(*LimitedReader).Read+0x44                   /usr/local/go/src/io/io.go:477
#   0x7ad3fb    net/http.(*body).readLocked+0x3b                /usr/local/go/src/net/http/transfer.go:846
#   0x7ad2c4    net/http.(*body).Read+0x124                 /usr/local/go/src/net/http/transfer.go:838
#   0x7bcfc1    net/http.(*bodyEOFSignal).Read+0x141                /usr/local/go/src/net/http/transport.go:2774
#   0x12643bc   github.com/minio/minio-go/v7.glob..func2+0xbc           /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/utils.go:401
#   0x1226acf   github.com/minio/minio-go/v7.(*Client).GetObject.func1+0xb8f    /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:210

2 @ 0x43cd56 0x435557 0x468829 0x4df112 0x4e047a 0x4e0468 0x587549 0x59b245 0x798331 0x56253f 0x56312f 0x563387 0x722ff9 0x7936b9 0x7936ba 0x7998ca 0x79de05 0x46e341
#   0x468828    internal/poll.runtime_pollWait+0x88     /usr/local/go/src/runtime/netpoll.go:305
#   0x4df111    internal/poll.(*pollDesc).wait+0x31     /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#   0x4e0479    internal/poll.(*pollDesc).waitRead+0x259    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#   0x4e0467    internal/poll.(*FD).Read+0x247          /usr/local/go/src/internal/poll/fd_unix.go:167
#   0x587548    net.(*netFD).Read+0x28              /usr/local/go/src/net/fd_posix.go:55
#   0x59b244    net.(*conn).Read+0x44               /usr/local/go/src/net/net.go:183
#   0x798330    net/http.(*connReader).Read+0x170       /usr/local/go/src/net/http/server.go:786
#   0x56253e    bufio.(*Reader).fill+0xfe           /usr/local/go/src/bufio/bufio.go:106
#   0x56312e    bufio.(*Reader).ReadSlice+0x2e          /usr/local/go/src/bufio/bufio.go:372
#   0x563386    bufio.(*Reader).ReadLine+0x26           /usr/local/go/src/bufio/bufio.go:401
#   0x722ff8    net/textproto.(*Reader).readLineSlice+0x98  /usr/local/go/src/net/textproto/reader.go:56
#   0x7936b8    net/textproto.(*Reader).ReadLine+0x78       /usr/local/go/src/net/textproto/reader.go:37
#   0x7936b9    net/http.readRequest+0x79           /usr/local/go/src/net/http/request.go:1036
#   0x7998c9    net/http.(*conn).readRequest+0x249      /usr/local/go/src/net/http/server.go:994
#   0x79de04    net/http.(*conn).serve+0x344            /usr/local/go/src/net/http/server.go:1916

1 @ 0x1225f41 0x46e341
#   0x1225f40   github.com/minio/minio-go/v7.(*Client).GetObject.func1+0x0  /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:64

1 @ 0x40c474 0x46a86f 0x61c039 0x46e341
#   0x46a86e    os/signal.signal_recv+0x2e  /usr/local/go/src/runtime/sigqueue.go:152
#   0x61c038    os/signal.loop+0x18     /usr/local/go/src/os/signal/signal_unix.go:23

1 @ 0x431bf6 0x468305 0xf43395 0xf431ad 0xf4010b 0x1381785 0x138231e 0x79f5ef 0x7a0f89 0x13a553b 0x7a29ec 0x79e0c7 0x46e341
#   0x468304    runtime/pprof.runtime_goroutineProfileWithLabels+0x24   /usr/local/go/src/runtime/mprof.go:846
#   0xf43394    runtime/pprof.writeRuntimeProfile+0xb4          /usr/local/go/src/runtime/pprof/pprof.go:723
#   0xf431ac    runtime/pprof.writeGoroutine+0x4c           /usr/local/go/src/runtime/pprof/pprof.go:683
#   0xf4010a    runtime/pprof.(*Profile).WriteTo+0x14a          /usr/local/go/src/runtime/pprof/pprof.go:330
#   0x1381784   net/http/pprof.handler.ServeHTTP+0x4a4          /usr/local/go/src/net/http/pprof/pprof.go:253
#   0x138231d   net/http/pprof.Index+0x13d              /usr/local/go/src/net/http/pprof/pprof.go:371
#   0x79f5ee    net/http.HandlerFunc.ServeHTTP+0x2e         /usr/local/go/src/net/http/server.go:2109
#   0x7a0f88    net/http.(*ServeMux).ServeHTTP+0x148            /usr/local/go/src/net/http/server.go:2487
#   0x13a553a   golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP+0x59a   /go/pkg/mod/golang.org/x/net@v0.4.0/http2/h2c/h2c.go:125
#   0x7a29eb    net/http.serverHandler.ServeHTTP+0x30b          /usr/local/go/src/net/http/server.go:2947
#   0x79e0c6    net/http.(*conn).serve+0x606                /usr/local/go/src/net/http/server.go:1991

1 @ 0x43b47f 0x46f99b 0x196c260 0x196c1de 0x19915f9 0x19910e9 0x46e341
#   0x196c25f   github.com/thanos-io/thanos/pkg/store/storepb.NewSeriesResponse+0x25f   /app/pkg/store/storepb/custom.go:41
#   0x196c1dd   github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).Recv+0x1dd   /app/pkg/store/bucket.go:937
#   0x19915f8   github.com/thanos-io/thanos/pkg/store.newLazyRespSet.func1.3+0x3f8  /app/pkg/store/proxy_heap.go:424
#   0x19910e8   github.com/thanos-io/thanos/pkg/store.newLazyRespSet.func1+0x228    /app/pkg/store/proxy_heap.go:479

1 @ 0x43cd56 0x406d5b 0x406858 0x1f9c3d3 0x61c4cf 0x46e341
#   0x1f9c3d2   main.main.func2+0x32                /app/cmd/thanos/main.go:119
#   0x61c4ce    github.com/oklog/run.(*Group).Run.func1+0x2e    /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38

1 @ 0x43cd56 0x406d5b 0x406858 0x61c23c 0x1f9bde7 0x43c992 0x46e341
#   0x61c23b    github.com/oklog/run.(*Group).Run+0x7b  /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:43
#   0x1f9bde6   main.main+0x1726            /app/cmd/thanos/main.go:159
#   0x43c991    runtime.main+0x211          /usr/local/go/src/runtime/proc.go:250

1 @ 0x43cd56 0x435557 0x468829 0x4df112 0x4e047a 0x4e0468 0x587549 0x59b245 0x797e5f 0x46e341
#   0x468828    internal/poll.runtime_pollWait+0x88     /usr/local/go/src/runtime/netpoll.go:305
#   0x4df111    internal/poll.(*pollDesc).wait+0x31     /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#   0x4e0479    internal/poll.(*pollDesc).waitRead+0x259    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#   0x4e0467    internal/poll.(*FD).Read+0x247          /usr/local/go/src/internal/poll/fd_unix.go:167
#   0x587548    net.(*netFD).Read+0x28              /usr/local/go/src/net/fd_posix.go:55
#   0x59b244    net.(*conn).Read+0x44               /usr/local/go/src/net/net.go:183
#   0x797e5e    net/http.(*connReader).backgroundRead+0x3e  /usr/local/go/src/net/http/server.go:678

1 @ 0x43cd56 0x435557 0x468829 0x4df112 0x4e4394 0x4e4381 0x589655 0x5a45a8 0x5a377d 0x7a2f45 0x13a395f 0x13a30b1 0xa553c4 0x46e341
#   0x468828    internal/poll.runtime_pollWait+0x88                 /usr/local/go/src/runtime/netpoll.go:305
#   0x4df111    internal/poll.(*pollDesc).wait+0x31                 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#   0x4e4393    internal/poll.(*pollDesc).waitRead+0x233                /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#   0x4e4380    internal/poll.(*FD).Accept+0x220                    /usr/local/go/src/internal/poll/fd_unix.go:614
#   0x589654    net.(*netFD).accept+0x34                        /usr/local/go/src/net/fd_unix.go:172
#   0x5a45a7    net.(*TCPListener).accept+0x27                      /usr/local/go/src/net/tcpsock_posix.go:142
#   0x5a377c    net.(*TCPListener).Accept+0x3c                      /usr/local/go/src/net/tcpsock.go:288
#   0x7a2f44    net/http.(*Server).Serve+0x384                      /usr/local/go/src/net/http/server.go:3070
#   0x13a395e   github.com/prometheus/exporter-toolkit/web.Serve+0x35e          /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.8.2/web/tls_config.go:236
#   0x13a30b0   github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1+0x30 /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.8.2/web/tls_config.go:195
#   0xa553c3    golang.org/x/sync/errgroup.(*Group).Go.func1+0x63           /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75

1 @ 0x43cd56 0x435557 0x468829 0x4df112 0x4e4394 0x4e4381 0x589655 0x5a45a8 0x5a377d 0xec9742 0x1db6d3e 0x1fce0d4 0x61c4cf 0x46e341
#   0x468828    internal/poll.runtime_pollWait+0x88                     /usr/local/go/src/runtime/netpoll.go:305
#   0x4df111    internal/poll.(*pollDesc).wait+0x31                     /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#   0x4e4393    internal/poll.(*pollDesc).waitRead+0x233                    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#   0x4e4380    internal/poll.(*FD).Accept+0x220                        /usr/local/go/src/internal/poll/fd_unix.go:614
#   0x589654    net.(*netFD).accept+0x34                            /usr/local/go/src/net/fd_unix.go:172
#   0x5a45a7    net.(*TCPListener).accept+0x27                          /usr/local/go/src/net/tcpsock_posix.go:142
#   0x5a377c    net.(*TCPListener).Accept+0x3c                          /usr/local/go/src/net/tcpsock.go:288
#   0xec9741    google.golang.org/grpc.(*Server).Serve+0x361                    /go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:779
#   0x1db6d3d   github.com/thanos-io/thanos/pkg/server/grpc.(*Server).ListenAndServe+0x19d  /app/pkg/server/grpc/grpc.go:132
#   0x1fce0d3   main.runStore.func7+0x53                            /app/cmd/thanos/store.go:442
#   0x61c4ce    github.com/oklog/run.(*Group).Run.func1+0x2e                    /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38

1 @ 0x43cd56 0x44cc3c 0x12272b6 0x1227705 0x13156bd 0x1315ae8 0xd1683d 0xd18cc3 0x19789b0 0x197eff3 0x197eddf 0xa553c4 0x46e341
#   0x12272b5   github.com/minio/minio-go/v7.(*Object).doGetRequest+0xd5            /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:306
#   0x1227704   github.com/minio/minio-go/v7.(*Object).Read+0x244               /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:386
#   0x13156bc   github.com/thanos-io/objstore/providers/s3.(*Bucket).getRange+0x1fc     /go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221205132204-5aafc0079f06/providers/s3/s3.go:442
#   0x1315ae7   github.com/thanos-io/objstore/providers/s3.(*Bucket).GetRange+0x27      /go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221205132204-5aafc0079f06/providers/s3/s3.go:459
#   0xd1683c    github.com/thanos-io/objstore.(*metricBucket).GetRange+0xbc         /go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221205132204-5aafc0079f06/objstore.go:526
#   0xd18cc2    github.com/thanos-io/objstore.TracingBucket.GetRange+0x242          /go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221205132204-5aafc0079f06/tracing.go:50
#   0x19789af   github.com/thanos-io/thanos/pkg/store.(*bucketBlock).chunkRangeReader+0xaf  /app/pkg/store/bucket.go:1974
#   0x197eff2   github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).loadChunks+0x132 /app/pkg/store/bucket.go:2798
#   0x197edde   github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).load.func3+0xfe  /app/pkg/store/bucket.go:2782
#   0xa553c3    golang.org/x/sync/errgroup.(*Group).Go.func1+0x63               /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75

1 @ 0x43cd56 0x44cc3c 0x14a9459 0x1fce98e 0x61c4cf 0x46e341
#   0x14a9458   github.com/thanos-io/thanos/pkg/runutil.Repeat+0xd8 /app/pkg/runutil/runutil.go:77
#   0x1fce98d   main.runStore.func3+0x42d               /app/cmd/thanos/store.go:391
#   0x61c4ce    github.com/oklog/run.(*Group).Run.func1+0x2e        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38

1 @ 0x43cd56 0x44cc3c 0x1f9c8aa 0x1f9c1e5 0x61c4cf 0x46e341
#   0x1f9c8a9   main.interrupt+0x109                /app/cmd/thanos/main.go:170
#   0x1f9c1e4   main.main.func4+0x24                /app/cmd/thanos/main.go:143
#   0x61c4ce    github.com/oklog/run.(*Group).Run.func1+0x2e    /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38

1 @ 0x43cd56 0x44cc3c 0x1f9cb7e 0x1f9c169 0x61c4cf 0x46e341
#   0x1f9cb7d   main.reload+0xfd                /app/cmd/thanos/main.go:183
#   0x1f9c168   main.main.func6+0x28                /app/cmd/thanos/main.go:153
#   0x61c4ce    github.com/oklog/run.(*Group).Run.func1+0x2e    /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38

1 @ 0x43cd56 0x44cc3c 0xdbacb9 0x46e341
#   0xdbacb8    github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1+0xb8    /go/pkg/mod/github.com/baidubce/bce-sdk-go@v0.9.111/util/log/logger.go:375

1 @ 0x43cd56 0x44cc3c 0xf549ed 0x46e341
#   0xf549ec    go.opencensus.io/stats/view.(*worker).start+0xac    /go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276

1 @ 0x43cd56 0x44dcfe 0x44dcd5 0x46a105 0x47a1b2 0xa55227 0x13a2f07 0x13a3368 0x14603a7 0x1fcf235 0x61c4cf 0x46e341
#   0x46a104    sync.runtime_Semacquire+0x24                            /usr/local/go/src/runtime/sema.go:62
#   0x47a1b1    sync.(*WaitGroup).Wait+0x51                         /usr/local/go/src/sync/waitgroup.go:139
#   0xa55226    golang.org/x/sync/errgroup.(*Group).Wait+0x26                   /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:53
#   0x13a2f06   github.com/prometheus/exporter-toolkit/web.ServeMultiple+0x66           /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.8.2/web/tls_config.go:198
#   0x13a3367   github.com/prometheus/exporter-toolkit/web.ListenAndServe+0x287         /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.8.2/web/tls_config.go:226
#   0x14603a6   github.com/thanos-io/thanos/pkg/server/http.(*Server).ListenAndServe+0x226  /app/pkg/server/http/http.go:85
#   0x1fcf234   main.runStore.func1+0x34                            /app/cmd/thanos/store.go:253
#   0x61c4ce    github.com/oklog/run.(*Group).Run.func1+0x2e                    /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38

1 @ 0x452537 0x197d9ac 0x197e8cf 0x196c8af 0x196c05d 0x19915f9 0x19910e9 0x46e341
#   0x197d9ab   github.com/thanos-io/thanos/pkg/store.gapBasedPartitioner.Partition+0x12b   /app/pkg/store/bucket.go:2596
#   0x197e8ce   github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).load+0x2ce       /app/pkg/store/bucket.go:2767
#   0x196c8ae   github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).nextBatch+0x5ae  /app/pkg/store/bucket.go:1015
#   0x196c05c   github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).Recv+0x5c        /app/pkg/store/bucket.go:922
#   0x19915f8   github.com/thanos-io/thanos/pkg/store.newLazyRespSet.func1.3+0x3f8      /app/pkg/store/proxy_heap.go:424
#   0x19910e8   github.com/thanos-io/thanos/pkg/store.newLazyRespSet.func1+0x228        /app/pkg/store/proxy_heap.go:479

1 @ 0x4d29c5 0x14f0f9d 0x14f0b05 0x197b586 0x197953b 0x196be3a 0x19714b9 0xa553c4 0x46e341
#   0x4d29c4    sort.Search+0x84                                    /usr/local/go/src/sort/search.go:58
#   0x14f0f9c   github.com/thanos-io/thanos/pkg/block/indexheader.(*BinaryReader).postingsOffset+0x2dc  /app/pkg/block/indexheader/binary_reader.go:716
#   0x14f0b04   github.com/thanos-io/thanos/pkg/block/indexheader.(*BinaryReader).PostingsOffset+0x44   /app/pkg/block/indexheader/binary_reader.go:657
#   0x197b585   github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings+0xb85      /app/pkg/store/bucket.go:2286
#   0x197953a   github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings+0x77a   /app/pkg/store/bucket.go:2094
#   0x196be39   github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings+0x79      /app/pkg/store/bucket.go:899
#   0x19714b8   github.com/thanos-io/thanos/pkg/store.(*BucketStore).Series.func2+0x378         /app/pkg/store/bucket.go:1233
#   0xa553c3    golang.org/x/sync/errgroup.(*Group).Go.func1+0x63                   /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75

1 @ 0x4d5044 0x4d56e5 0x4d303d 0x11d5305 0x11d527b 0x11d5545 0x11d6865 0x12590b3 0x1258fdc 0x125758a 0x1229a9c 0x1226250 0x46e341
#   0x4d5043    sort.insertionSort+0x63                         /usr/local/go/src/sort/zsortinterface.go:13
#   0x4d56e4    sort.pdqsort+0x344                          /usr/local/go/src/sort/zsortinterface.go:73
#   0x4d303c    sort.Sort+0x5c                              /usr/local/go/src/sort/sort.go:48
#   0x11d5304   sort.Strings+0x244                          /usr/local/go/src/sort/sort.go:164
#   0x11d527a   github.com/minio/minio-go/v7/pkg/signer.getSignedHeaders+0x1ba      /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/pkg/signer/request-signature-v4.go:171
#   0x11d5544   github.com/minio/minio-go/v7/pkg/signer.getCanonicalRequest+0x164   /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/pkg/signer/request-signature-v4.go:192
#   0x11d6864   github.com/minio/minio-go/v7/pkg/signer.signV4+0x524            /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/pkg/signer/request-signature-v4.go:305
#   0x12590b2   github.com/minio/minio-go/v7/pkg/signer.SignV4Trailer+0xed2     /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/pkg/signer/request-signature-v4.go:350
#   0x1258fdb   github.com/minio/minio-go/v7.(*Client).newRequest+0xdfb         /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api.go:865
#   0x1257589   github.com/minio/minio-go/v7.(*Client).executeMethod+0x7c9      /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api.go:610
#   0x1229a9b   github.com/minio/minio-go/v7.(*Client).getObject+0x37b          /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:666
#   0x122624f   github.com/minio/minio-go/v7.(*Client).GetObject.func1+0x30f        /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:93

1 @ 0x54d9b4 0x1152b4a 0x1152ae7 0x1152a3c 0x1259878 0x1259770 0x12584da 0x125758a 0x1229a9c 0x1226250 0x46e341
#   0x54d9b3    type..eq.net/url.URL+0x1f3                          <autogenerated>:1
#   0x1152b49   github.com/minio/minio-go/v7/pkg/s3utils.IsAmazonFIPSGovCloudEndpoint+0x1a9 /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/pkg/s3utils/utils.go:195
#   0x1152ae6   github.com/minio/minio-go/v7/pkg/s3utils.IsAmazonGovCloudEndpoint+0x146     /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/pkg/s3utils/utils.go:189
#   0x1152a3b   github.com/minio/minio-go/v7/pkg/s3utils.GetRegionFromURL+0x9b          /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/pkg/s3utils/utils.go:124
#   0x1259877   github.com/minio/minio-go/v7/pkg/s3utils.IsAmazonEndpoint+0x197         /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/pkg/s3utils/utils.go:180
#   0x125976f   github.com/minio/minio-go/v7.(*Client).makeTargetURL+0x8f           /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api.go:884
#   0x12584d9   github.com/minio/minio-go/v7.(*Client).newRequest+0x2f9             /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api.go:743
#   0x1257589   github.com/minio/minio-go/v7.(*Client).executeMethod+0x7c9          /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api.go:610
#   0x1229a9b   github.com/minio/minio-go/v7.(*Client).getObject+0x37b              /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:666
#   0x122624f   github.com/minio/minio-go/v7.(*Client).GetObject.func1+0x30f            /go/pkg/mod/github.com/minio/minio-go/v7@v7.0.45/api-get-object.go:93

thanos-io / thanos

Possible goroutines leak for query and query-frontend #6002