search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.12k stars 2.1k forks source link

receive: periodic huge spikes in latency when blocks are cut #7913

Open verejoel opened 1 day ago

verejoel commented 1 day ago

Thanos, Prometheus and Golang version used: 0.36.1

Object Storage Provider: Azure

What happened: Recently we migrated from hard-tenancy to soft-tenancy. This means we went from having a single tenant per receive statefultset, to having around 14 tenants in one large statefulset. Since migrating, we observe huge spikes in latency on the routing receivers to the receivers:

grafik

This manifests in a large increase in 503 errors, leading to clients needing to retry requests, and a noticeable drop in throughput for a period of about 3-4 minutes:

grafik grafik

The times in the graphs are UTC. The spikes at 07:00, 09:00, 11:00 are when blocks are cut to persistent disk. This leads to an ingestion degradation of around 5 minutes. It looks like we hit some gRPC timeouts of 5s (assumedly between routing receives and receives) and I haven't found a way to configure this or otherwise increase it.

I am not sure what happens at 08:00 and 12:00 (though the effects seem to be much smaller, note the graphs are logarithmic).

What you expected to happen: as before: no disruption to ingestion when blocks are being cut

How to reproduce it (as minimally and precisely as possible): I have only seen this issue at production loads. I will detail our setup below as best as I can.

Environment: kubernetes, AKS

Routing receives 10 replicas 4 vCPU (no limits) 16/32 GiB Memory req/lim

      - args:
        - receive
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --label=ingester_replica="$(NAME)"
        - --log.level=debug
        - --receive.hashrings-file=/etc/hashring/hashrings.json
        - --receive.hashrings-algorithm=ketama
        - --auto-gomemlimit.ratio=0.9
        - --enable-auto-gomemlimit
        - --receive.limits-config-file=/etc/limits/limits.yaml
        - --receive.replication-factor=3
        - --remote-write.address=0.0.0.0:10908

Receives 10 replicas 10 vCPU (no limits, no throttling) 128/256 GiB Memory req/lim

      - args:
        - receive
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --label=ingester_replica="$(NAME)"
        - --log.level=debug
        - --receive.default-tenant-id=unknown
        - --receive.local-endpoint=$(NAME).$(NAMESPACE).svc.cluster.local:10901
        - --remote-write.address=0.0.0.0:10908
        - --tsdb.path=/data/tsdb
        - --tsdb.retention=6h
        - --auto-gomemlimit.ratio=0.9
        - --enable-auto-gomemlimit
        - --tsdb.too-far-in-future.time-window=10m
        - --tsdb.out-of-order.time-window=30m
        - --store.limits.request-series=500000
        - --objstore.config-file=/etc/secrets/objstoreconfig.yaml

Hashring controller

      - args:
        - --configmap-name=thanos-hashring-controller-base
        - --configmap-generated-name=thanos-hashring-controller-generated
        - --file-name=hashrings.json
        - --allow-only-ready-replicas=true
        - --allow-dynamic-scaling=false
        - --namespace=thanos
        - --statefulset-label=controller.receive.thanos.io=thanos-hashring-controller

We are not using Istio or any other proxy in-between routing receivers and receivers.

Full logs to relevant components: too big, will post in a separate comment.

Anything else we need to know: I can provide profiles, or anything else as needed.

Each receive has about 26 million active head series. I have tried scaling out to more, smaller, receivers (up to 60 replicas) but still saw the same issue of high latency, as well as finding that this had a negative impact on query performance.

verejoel commented 1 day ago

Logs from receiver during block cutting:

Logs

``` ❯ kt logs thanos-ingester-8 -f | grep -v proxy.go ts=2024-11-15T13:00:53.030165124Z caller=db.go:1347 level=info component=receive component=multi-tsdb tenant=rule msg="compact ooo head resulted in no blocks" duration=4.872µs ts=2024-11-15T13:00:56.140310278Z caller=head.go:1317 level=info component=receive component=multi-tsdb tenant=bandwidthcontrol msg="WAL checkpoint complete" first=162 last=168 duration=14.1756462s ts=2024-11-15T13:01:04.310902936Z caller=db.go:1347 level=info component=receive component=multi-tsdb tenant=bandwidthcontrol msg="compact ooo head resulted in no blocks" duration=3.04µs ts=2024-11-15T13:01:09.509857452Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:01:09.509881717Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:01:09.509902868Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:01:09.509908497Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:01:09.509918373Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:01:09.509924293Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:01:09.509931982Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:01:09.509937556Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:01:09.509943703Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:01:09.509948536Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:01:09.509956756Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:01:09.509963204Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:01:09.509972962Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:01:09.50998321Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:01:09.51028047Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYHK2G0YDKWXV2N2A1QGEN/meta.json ts=2024-11-15T13:01:09.512968753Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=rule msg="upload new block" id=01JCQYHK2G0YDKWXV2N2A1QGEN ts=2024-11-15T13:01:10.339230517Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHK2G0YDKWXV2N2A1QGEN/chunks/000001 ts=2024-11-15T13:01:10.855650325Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="write block" mint=1731664800003 maxt=1731672000000 ulid=01JCQYHFQ8ZTRJ0GMPRD8GSZ65 duration=1m8.191040913s ooo=false ts=2024-11-15T13:01:10.960356105Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=rule msg="uploaded file" from=/data/tsdb/rule/thanos/upload/01JCQYHK2G0YDKWXV2N2A1QGEN/chunks/000001 dst=01JCQYHK2G0YDKWXV2N2A1QGEN/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:01:10.960451382Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHK2G0YDKWXV2N2A1QGEN/index ts=2024-11-15T13:01:11.941811365Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=rule msg="uploaded file" from=/data/tsdb/rule/thanos/upload/01JCQYHK2G0YDKWXV2N2A1QGEN/index dst=01JCQYHK2G0YDKWXV2N2A1QGEN/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:01:11.941874204Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHK2G0YDKWXV2N2A1QGEN/meta.json ts=2024-11-15T13:01:14.152661162Z caller=head.go:1355 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="Head GC completed" caller=truncateMemory duration=3.283187177s ts=2024-11-15T13:01:19.080665705Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.08073013Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.080740562Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.080796649Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.080803345Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.080827144Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.080818572Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.080868556Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.080924823Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.080997967Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.081073742Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.081083467Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.081142577Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.081148093Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.081177078Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.081444525Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.081594132Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.081724929Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.081745152Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.082164992Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.082192883Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.082281356Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.082390191Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.082730754Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.085520648Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.087015796Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.088470429Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.090559536Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:19.158564936Z caller=checkpoint.go:101 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="Creating checkpoint" from_segment=469 to_segment=490 mint=1731672000000 ts=2024-11-15T13:01:26.08127342Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=1 elapsed=16.571389617s ts=2024-11-15T13:01:37.392059918Z caller=head.go:1317 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="WAL checkpoint complete" first=469 last=490 duration=18.233636681s ts=2024-11-15T13:01:39.5098782Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:01:39.509930272Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:01:39.50994556Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:01:39.50995647Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:01:39.509964258Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:01:39.509973449Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:01:39.509982728Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:01:39.509995045Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:01:39.510012142Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:01:39.510020234Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:01:39.510033412Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:01:39.510045738Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:01:39.510059292Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:01:39.510073724Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:01:39.510907957Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYHFQ8ZTRJ0GMPRD8GSZ65/meta.json ts=2024-11-15T13:01:39.513301155Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="upload new block" id=01JCQYHFQ8ZTRJ0GMPRD8GSZ65 ts=2024-11-15T13:01:39.51933602Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHFQ8ZTRJ0GMPRD8GSZ65/chunks/000001 ts=2024-11-15T13:01:41.425203488Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=centralinfrastructure msg="uploaded file" from=/data/tsdb/centralinfrastructure/thanos/upload/01JCQYHFQ8ZTRJ0GMPRD8GSZ65/chunks/000001 dst=01JCQYHFQ8ZTRJ0GMPRD8GSZ65/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:01:41.425300946Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHFQ8ZTRJ0GMPRD8GSZ65/index ts=2024-11-15T13:01:43.23620884Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=centralinfrastructure msg="uploaded file" from=/data/tsdb/centralinfrastructure/thanos/upload/01JCQYHFQ8ZTRJ0GMPRD8GSZ65/index dst=01JCQYHFQ8ZTRJ0GMPRD8GSZ65/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:01:43.236295085Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHFQ8ZTRJ0GMPRD8GSZ65/meta.json ts=2024-11-15T13:01:43.828316473Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=1 elapsed=4.31838584s ts=2024-11-15T13:01:53.0285465Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.029665766Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.028778245Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.028944875Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.029048588Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.029110378Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.029217644Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.029432811Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.030127412Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.030281737Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.030434603Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.0305173Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.031176202Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.031954218Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.03217123Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.03236436Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.032508813Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.032586404Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.033623267Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.03370533Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.033763925Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.033821283Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.033932762Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.034834113Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.034903718Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.034961109Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.035062149Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.035889626Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.036128984Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.036187445Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.036412066Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.037229123Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.037332098Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:53.037490123Z caller=handler.go:1011 level=debug component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded" ts=2024-11-15T13:01:57.770856496Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=wan msg="write block" mint=1731664800001 maxt=1731672000000 ulid=01JCQYHDYB42G3867JPZX6WC4W duration=1m56.927212257s ooo=false ts=2024-11-15T13:01:58.161855024Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=wan msg="Deleting obsolete block" block=01JCQ9Y8ANXZ49Z5A3WRQAYCE5 ts=2024-11-15T13:01:58.228511537Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=wan msg="Deleting obsolete block" block=01JCQA2TS587424ZW63HQWQWX9 ts=2024-11-15T13:01:58.42380206Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=wan msg="Looked up series not found" ts=2024-11-15T13:01:58.438234778Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=wan msg="Looked up series not found" ts=2024-11-15T13:02:01.994733735Z caller=head.go:1355 level=info component=receive component=multi-tsdb tenant=wan msg="Head GC completed" caller=truncateMemory duration=3.766109988s ts=2024-11-15T13:02:02.176427285Z caller=checkpoint.go:101 level=info component=receive component=multi-tsdb tenant=wan msg="Creating checkpoint" from_segment=424 to_segment=441 mint=1731672000000 ts=2024-11-15T13:02:09.509820005Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:02:09.509850856Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:02:09.509863123Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:02:09.509868963Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:02:09.50987393Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:02:09.509879396Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:02:09.509884617Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:02:09.509889347Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:02:09.509904921Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:02:09.509914066Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:02:09.509919936Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:02:09.509929968Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:02:09.50993732Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:02:09.509943331Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:02:09.510140066Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYHDYB42G3867JPZX6WC4W/meta.json ts=2024-11-15T13:02:09.51255552Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=wan msg="upload new block" id=01JCQYHDYB42G3867JPZX6WC4W ts=2024-11-15T13:02:09.518232628Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHDYB42G3867JPZX6WC4W/chunks/000001 ts=2024-11-15T13:02:11.893288879Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=wan msg="uploaded file" from=/data/tsdb/wan/thanos/upload/01JCQYHDYB42G3867JPZX6WC4W/chunks/000001 dst=01JCQYHDYB42G3867JPZX6WC4W/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:02:11.893409907Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHDYB42G3867JPZX6WC4W/chunks/000002 ts=2024-11-15T13:02:11.903181646Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="write block" mint=1731664800000 maxt=1731672000000 ulid=01JCQYMMPCKVD2AGP31K0FN3X9 duration=25.842556502s ooo=true ts=2024-11-15T13:02:12.784072761Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=wan msg="uploaded file" from=/data/tsdb/wan/thanos/upload/01JCQYHDYB42G3867JPZX6WC4W/chunks/000002 dst=01JCQYHDYB42G3867JPZX6WC4W/chunks/000002 bucket="tracing: thanos-blocks" ts=2024-11-15T13:02:12.784160558Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHDYB42G3867JPZX6WC4W/index ts=2024-11-15T13:02:13.559112048Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="write block" mint=1731672000000 maxt=1731679200000 ulid=01JCQYNDXZTF7VPJ6A0ABHX5M3 duration=1.655884085s ooo=true ts=2024-11-15T13:02:13.559160129Z caller=db.go:1354 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="out-of-order compaction completed" duration=27.498536271s ulids="[01JCQYMMPCKVD2AGP31K0FN3X9 01JCQYNDXZTF7VPJ6A0ABHX5M3]" ts=2024-11-15T13:02:13.753344722Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="Deleting obsolete block" block=01JCQGT1RG6JRNFZEC9TM1X0CS ts=2024-11-15T13:02:13.759602328Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="Deleting obsolete block" block=01JCQGX72H7CVJBPTYDMKQ9Q1R ts=2024-11-15T13:02:13.765742014Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="Deleting obsolete block" block=01JCQA15QV0GHVHTN565JWAZTV ts=2024-11-15T13:02:15.038796889Z caller=head.go:1355 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="Head GC completed" caller=truncateOOO duration=1.27289717s ts=2024-11-15T13:02:16.046707677Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=wan msg="uploaded file" from=/data/tsdb/wan/thanos/upload/01JCQYHDYB42G3867JPZX6WC4W/index dst=01JCQYHDYB42G3867JPZX6WC4W/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:02:16.046774951Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHDYB42G3867JPZX6WC4W/meta.json ts=2024-11-15T13:02:16.057072403Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=1 elapsed=6.547221995s ts=2024-11-15T13:02:23.79623817Z caller=head.go:1317 level=info component=receive component=multi-tsdb tenant=wan msg="WAL checkpoint complete" first=424 last=441 duration=21.61995278s ts=2024-11-15T13:02:39.510558113Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:02:39.510583804Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:02:39.510596657Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:02:39.510603659Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:02:39.510608902Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:02:39.51061626Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:02:39.510624724Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:02:39.510630462Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:02:39.510639561Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:02:39.510644272Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:02:39.510650095Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:02:39.510658868Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:02:39.510663894Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:02:39.510668694Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:02:39.511142731Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYMMPCKVD2AGP31K0FN3X9/meta.json ts=2024-11-15T13:02:39.513909675Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="upload new block" id=01JCQYMMPCKVD2AGP31K0FN3X9 ts=2024-11-15T13:02:39.518736642Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYMMPCKVD2AGP31K0FN3X9/chunks/000001 ts=2024-11-15T13:02:39.602004923Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=centralinfrastructure msg="uploaded file" from=/data/tsdb/centralinfrastructure/thanos/upload/01JCQYMMPCKVD2AGP31K0FN3X9/chunks/000001 dst=01JCQYMMPCKVD2AGP31K0FN3X9/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:02:39.602103622Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYMMPCKVD2AGP31K0FN3X9/index ts=2024-11-15T13:02:40.161892735Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=centralinfrastructure msg="uploaded file" from=/data/tsdb/centralinfrastructure/thanos/upload/01JCQYMMPCKVD2AGP31K0FN3X9/index dst=01JCQYMMPCKVD2AGP31K0FN3X9/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:02:40.161958401Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYMMPCKVD2AGP31K0FN3X9/meta.json ts=2024-11-15T13:02:40.167050205Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYNDXZTF7VPJ6A0ABHX5M3/meta.json ts=2024-11-15T13:02:40.171437323Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=centralinfrastructure msg="upload new block" id=01JCQYNDXZTF7VPJ6A0ABHX5M3 ts=2024-11-15T13:02:40.175681455Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYNDXZTF7VPJ6A0ABHX5M3/chunks/000001 ts=2024-11-15T13:02:40.182422077Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=centralinfrastructure msg="uploaded file" from=/data/tsdb/centralinfrastructure/thanos/upload/01JCQYNDXZTF7VPJ6A0ABHX5M3/chunks/000001 dst=01JCQYNDXZTF7VPJ6A0ABHX5M3/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:02:40.182515958Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYNDXZTF7VPJ6A0ABHX5M3/index ts=2024-11-15T13:02:40.227123341Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=centralinfrastructure msg="uploaded file" from=/data/tsdb/centralinfrastructure/thanos/upload/01JCQYNDXZTF7VPJ6A0ABHX5M3/index dst=01JCQYNDXZTF7VPJ6A0ABHX5M3/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:02:40.227190418Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYNDXZTF7VPJ6A0ABHX5M3/meta.json ts=2024-11-15T13:02:40.236429351Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=2 elapsed=725.845553ms ts=2024-11-15T13:02:45.794283833Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=wan msg="write block" mint=1731664800000 maxt=1731672000000 ulid=01JCQYP8WV73N7B6RV9V5BAXEJ duration=6.278604284s ooo=true ts=2024-11-15T13:02:45.794340764Z caller=db.go:1354 level=info component=receive component=multi-tsdb tenant=wan msg="out-of-order compaction completed" duration=6.278661756s ulids=[01JCQYP8WV73N7B6RV9V5BAXEJ] ts=2024-11-15T13:02:45.825899749Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=proxy msg="write block" mint=1731664800004 maxt=1731672000000 ulid=01JCQYHDSEKDPNNYVCAN1FG5DN duration=2m45.139440013s ooo=false ts=2024-11-15T13:02:47.296035801Z caller=head.go:1355 level=info component=receive component=multi-tsdb tenant=wan msg="Head GC completed" caller=truncateOOO duration=1.475243727s ts=2024-11-15T13:02:51.711782108Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=proxy msg="Looked up series not found" ts=2024-11-15T13:02:51.71183487Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=proxy msg="Looked up series not found" ts=2024-11-15T13:02:51.71445793Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=proxy msg="Looked up series not found" ts=2024-11-15T13:02:51.714492043Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=proxy msg="Looked up series not found" ts=2024-11-15T13:02:51.798477406Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=proxy msg="Looked up series not found" ts=2024-11-15T13:02:51.798510858Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=proxy msg="Looked up series not found" ts=2024-11-15T13:02:51.800492121Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=proxy msg="Looked up series not found" ts=2024-11-15T13:02:51.800516457Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=proxy msg="Looked up series not found" ts=2024-11-15T13:02:54.369720634Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=mdr msg="write block" mint=1731664818258 maxt=1731672000000 ulid=01JCQYPQATQKFWMRRE086RS94T duration=71.032543ms ooo=false ts=2024-11-15T13:02:54.372876086Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=mdr msg="Deleting obsolete block" block=01JCQA1TGYFD6XN4S2Q3B07SWR ts=2024-11-15T13:02:54.374732608Z caller=head.go:1355 level=info component=receive component=multi-tsdb tenant=mdr msg="Head GC completed" caller=truncateMemory duration=1.818167ms ts=2024-11-15T13:02:54.375284522Z caller=db.go:1347 level=info component=receive component=multi-tsdb tenant=mdr msg="compact ooo head resulted in no blocks" duration=1.51µs ts=2024-11-15T13:02:57.98661807Z caller=head.go:1355 level=info component=receive component=multi-tsdb tenant=proxy msg="Head GC completed" caller=truncateMemory duration=12.154054028s ts=2024-11-15T13:02:58.12697815Z caller=checkpoint.go:101 level=info component=receive component=multi-tsdb tenant=proxy msg="Creating checkpoint" from_segment=461 to_segment=484 mint=1731672000000 ts=2024-11-15T13:03:09.509856673Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:03:09.510151506Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:03:09.510263496Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:03:09.510350358Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:03:09.510437909Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:03:09.510538948Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:03:09.510595535Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:03:09.51067082Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:03:09.510675186Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYPQATQKFWMRRE086RS94T/meta.json ts=2024-11-15T13:03:09.510696026Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:03:09.510812854Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYHDSEKDPNNYVCAN1FG5DN/meta.json ts=2024-11-15T13:03:09.510814242Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:03:09.511037732Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYP8WV73N7B6RV9V5BAXEJ/meta.json ts=2024-11-15T13:03:09.5110434Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:03:09.511264227Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:03:09.511281896Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:03:09.511294483Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:03:09.512911458Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=mdr msg="upload new block" id=01JCQYPQATQKFWMRRE086RS94T ts=2024-11-15T13:03:09.513080551Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=proxy msg="upload new block" id=01JCQYHDSEKDPNNYVCAN1FG5DN ts=2024-11-15T13:03:09.513658055Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=wan msg="upload new block" id=01JCQYP8WV73N7B6RV9V5BAXEJ ts=2024-11-15T13:03:09.524096348Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYPQATQKFWMRRE086RS94T/chunks/000001 ts=2024-11-15T13:03:09.524529331Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHDSEKDPNNYVCAN1FG5DN/chunks/000001 ts=2024-11-15T13:03:09.524805253Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYP8WV73N7B6RV9V5BAXEJ/chunks/000001 ts=2024-11-15T13:03:09.537214548Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=mdr msg="uploaded file" from=/data/tsdb/mdr/thanos/upload/01JCQYPQATQKFWMRRE086RS94T/chunks/000001 dst=01JCQYPQATQKFWMRRE086RS94T/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:03:09.537453829Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYPQATQKFWMRRE086RS94T/index ts=2024-11-15T13:03:09.539372733Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=wan msg="uploaded file" from=/data/tsdb/wan/thanos/upload/01JCQYP8WV73N7B6RV9V5BAXEJ/chunks/000001 dst=01JCQYP8WV73N7B6RV9V5BAXEJ/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:03:09.539593072Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYP8WV73N7B6RV9V5BAXEJ/index ts=2024-11-15T13:03:09.555811637Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=mdr msg="uploaded file" from=/data/tsdb/mdr/thanos/upload/01JCQYPQATQKFWMRRE086RS94T/index dst=01JCQYPQATQKFWMRRE086RS94T/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:03:09.555898519Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYPQATQKFWMRRE086RS94T/meta.json ts=2024-11-15T13:03:09.822573935Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=wan msg="uploaded file" from=/data/tsdb/wan/thanos/upload/01JCQYP8WV73N7B6RV9V5BAXEJ/index dst=01JCQYP8WV73N7B6RV9V5BAXEJ/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:03:09.822656032Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYP8WV73N7B6RV9V5BAXEJ/meta.json ts=2024-11-15T13:03:12.282546823Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=proxy msg="uploaded file" from=/data/tsdb/proxy/thanos/upload/01JCQYHDSEKDPNNYVCAN1FG5DN/chunks/000001 dst=01JCQYHDSEKDPNNYVCAN1FG5DN/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:03:12.282632549Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHDSEKDPNNYVCAN1FG5DN/chunks/000002 ts=2024-11-15T13:03:13.69967663Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=proxy msg="uploaded file" from=/data/tsdb/proxy/thanos/upload/01JCQYHDSEKDPNNYVCAN1FG5DN/chunks/000002 dst=01JCQYHDSEKDPNNYVCAN1FG5DN/chunks/000002 bucket="tracing: thanos-blocks" ts=2024-11-15T13:03:13.69975682Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHDSEKDPNNYVCAN1FG5DN/index ts=2024-11-15T13:03:17.826794547Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=proxy msg="uploaded file" from=/data/tsdb/proxy/thanos/upload/01JCQYHDSEKDPNNYVCAN1FG5DN/index dst=01JCQYHDSEKDPNNYVCAN1FG5DN/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:03:17.82684091Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHDSEKDPNNYVCAN1FG5DN/meta.json ts=2024-11-15T13:03:18.028129694Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=3 elapsed=8.518006549s ts=2024-11-15T13:03:22.517709267Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=firewall msg="write block" mint=1731664800013 maxt=1731672000000 ulid=01JCQYHF72XDHEHV64RQZ4NRAB duration=3m20.371029257s ooo=false ts=2024-11-15T13:03:22.530255716Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=firewall msg="Looked up series not found" ts=2024-11-15T13:03:22.530258011Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=firewall msg="Looked up series not found" ts=2024-11-15T13:03:22.535651989Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=firewall msg="Looked up series not found" ts=2024-11-15T13:03:22.535676975Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=firewall msg="Looked up series not found" ts=2024-11-15T13:03:22.537730078Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=firewall msg="Looked up series not found" ts=2024-11-15T13:03:22.537907315Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=firewall msg="Looked up series not found" ts=2024-11-15T13:03:22.538092226Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=firewall msg="Looked up series not found" ts=2024-11-15T13:03:22.538279296Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=firewall msg="Looked up series not found" ts=2024-11-15T13:03:25.593860173Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=firewall msg="Looked up series not found" ts=2024-11-15T13:03:36.787182482Z caller=head.go:1355 level=info component=receive component=multi-tsdb tenant=firewall msg="Head GC completed" caller=truncateMemory duration=14.261164561s ts=2024-11-15T13:03:37.088089599Z caller=checkpoint.go:101 level=info component=receive component=multi-tsdb tenant=firewall msg="Creating checkpoint" from_segment=747 to_segment=779 mint=1731672000000 ts=2024-11-15T13:03:39.510107705Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:03:39.51013312Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:03:39.510142903Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:03:39.510148882Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:03:39.510154282Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:03:39.510159661Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:03:39.510164671Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:03:39.510169927Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:03:39.510175372Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:03:39.510180681Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:03:39.510187729Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:03:39.510196783Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:03:39.510207114Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:03:39.510212915Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:03:39.51059432Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYHF72XDHEHV64RQZ4NRAB/meta.json ts=2024-11-15T13:03:39.513778897Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=firewall msg="upload new block" id=01JCQYHF72XDHEHV64RQZ4NRAB ts=2024-11-15T13:03:43.823438313Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHF72XDHEHV64RQZ4NRAB/chunks/000001 ts=2024-11-15T13:03:44.013974906Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=sdp msg="write block" mint=1731664800002 maxt=1731672000000 ulid=01JCQYHDQ8VNP1SXAZ7Q1SZPRH duration=3m43.397631659s ooo=false ts=2024-11-15T13:03:45.215141018Z caller=head.go:1317 level=info component=receive component=multi-tsdb tenant=proxy msg="WAL checkpoint complete" first=461 last=484 duration=47.088278695s ts=2024-11-15T13:03:45.528136072Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=sdp msg="Looked up series not found" ts=2024-11-15T13:03:45.528634488Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=sdp msg="Looked up series not found" ts=2024-11-15T13:03:45.529006094Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=sdp msg="Looked up series not found" ts=2024-11-15T13:03:45.529997333Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=sdp msg="Looked up series not found" ts=2024-11-15T13:03:45.532275299Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=sdp msg="Looked up series not found" ts=2024-11-15T13:03:45.532638906Z caller=head_read.go:135 level=debug component=receive component=multi-tsdb tenant=sdp msg="Looked up series not found" ts=2024-11-15T13:03:46.290260499Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=firewall msg="uploaded file" from=/data/tsdb/firewall/thanos/upload/01JCQYHF72XDHEHV64RQZ4NRAB/chunks/000001 dst=01JCQYHF72XDHEHV64RQZ4NRAB/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:03:46.290360545Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHF72XDHEHV64RQZ4NRAB/chunks/000002 ts=2024-11-15T13:03:48.542153799Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=firewall msg="uploaded file" from=/data/tsdb/firewall/thanos/upload/01JCQYHF72XDHEHV64RQZ4NRAB/chunks/000002 dst=01JCQYHF72XDHEHV64RQZ4NRAB/chunks/000002 bucket="tracing: thanos-blocks" ts=2024-11-15T13:03:48.542276642Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHF72XDHEHV64RQZ4NRAB/index ts=2024-11-15T13:03:54.674569969Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=firewall msg="uploaded file" from=/data/tsdb/firewall/thanos/upload/01JCQYHF72XDHEHV64RQZ4NRAB/index dst=01JCQYHF72XDHEHV64RQZ4NRAB/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:03:54.674619948Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHF72XDHEHV64RQZ4NRAB/meta.json ts=2024-11-15T13:03:54.684160266Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=1 elapsed=15.174024723s ts=2024-11-15T13:03:56.614915935Z caller=head.go:1355 level=info component=receive component=multi-tsdb tenant=sdp msg="Head GC completed" caller=truncateMemory duration=12.536888579s ts=2024-11-15T13:03:57.094582566Z caller=checkpoint.go:101 level=info component=receive component=multi-tsdb tenant=sdp msg="Creating checkpoint" from_segment=996 to_segment=1038 mint=1731672000000 ts=2024-11-15T13:04:09.509809961Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:04:09.509838789Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:04:09.509855099Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:04:09.509863947Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:04:09.509871205Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:04:09.509881082Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:04:09.509892405Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:04:09.509900294Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:04:09.509907665Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:04:09.509919718Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:04:09.509927263Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:04:09.509934684Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:04:09.509943938Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:04:09.509949327Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:04:09.510331366Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYHDQ8VNP1SXAZ7Q1SZPRH/meta.json ts=2024-11-15T13:04:09.513743662Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=sdp msg="upload new block" id=01JCQYHDQ8VNP1SXAZ7Q1SZPRH ts=2024-11-15T13:04:09.69137273Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHDQ8VNP1SXAZ7Q1SZPRH/chunks/000001 ts=2024-11-15T13:04:12.279773422Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=sdp msg="uploaded file" from=/data/tsdb/sdp/thanos/upload/01JCQYHDQ8VNP1SXAZ7Q1SZPRH/chunks/000001 dst=01JCQYHDQ8VNP1SXAZ7Q1SZPRH/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:04:12.279858527Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHDQ8VNP1SXAZ7Q1SZPRH/chunks/000002 ts=2024-11-15T13:04:14.519112511Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=sdp msg="uploaded file" from=/data/tsdb/sdp/thanos/upload/01JCQYHDQ8VNP1SXAZ7Q1SZPRH/chunks/000002 dst=01JCQYHDQ8VNP1SXAZ7Q1SZPRH/chunks/000002 bucket="tracing: thanos-blocks" ts=2024-11-15T13:04:14.519238502Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHDQ8VNP1SXAZ7Q1SZPRH/chunks/000003 ts=2024-11-15T13:04:14.75464567Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=sdp msg="uploaded file" from=/data/tsdb/sdp/thanos/upload/01JCQYHDQ8VNP1SXAZ7Q1SZPRH/chunks/000003 dst=01JCQYHDQ8VNP1SXAZ7Q1SZPRH/chunks/000003 bucket="tracing: thanos-blocks" ts=2024-11-15T13:04:14.754726485Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHDQ8VNP1SXAZ7Q1SZPRH/index ts=2024-11-15T13:04:20.414080575Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=proxy msg="write block" mint=1731664800000 maxt=1731672000000 ulid=01JCQYSARQ6CH91NGBC1ZYA58Y duration=678.42272ms ooo=true ts=2024-11-15T13:04:21.155228759Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=sdp msg="uploaded file" from=/data/tsdb/sdp/thanos/upload/01JCQYHDQ8VNP1SXAZ7Q1SZPRH/index dst=01JCQYHDQ8VNP1SXAZ7Q1SZPRH/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:04:21.155367339Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYHDQ8VNP1SXAZ7Q1SZPRH/meta.json ts=2024-11-15T13:04:21.177197688Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=1 elapsed=11.667359192s ts=2024-11-15T13:04:21.216303682Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=proxy msg="write block" mint=1731672000000 maxt=1731679200000 ulid=01JCQYSBDYZPC18C5Z67X3B4MB duration=802.136206ms ooo=true ts=2024-11-15T13:04:21.216371431Z caller=db.go:1354 level=info component=receive component=multi-tsdb tenant=proxy msg="out-of-order compaction completed" duration=1.480700271s ulids="[01JCQYSARQ6CH91NGBC1ZYA58Y 01JCQYSBDYZPC18C5Z67X3B4MB]" ts=2024-11-15T13:04:21.228781566Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=proxy msg="Deleting obsolete block" block=01JCQA47JZ76B02VBJSXP9T9YD ts=2024-11-15T13:04:21.770249492Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=proxy msg="Deleting obsolete block" block=01JCQGSZS030JDCWKB2CJZ334M ts=2024-11-15T13:04:21.786341119Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=proxy msg="Deleting obsolete block" block=01JCQH19AVT1WWYZR492PC9XAT ts=2024-11-15T13:04:25.263014541Z caller=head.go:1355 level=info component=receive component=multi-tsdb tenant=proxy msg="Head GC completed" caller=truncateOOO duration=3.474435927s ts=2024-11-15T13:04:29.030830902Z caller=head.go:1317 level=info component=receive component=multi-tsdb tenant=firewall msg="WAL checkpoint complete" first=747 last=779 duration=51.942916496s ts=2024-11-15T13:04:39.509814574Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:04:39.509845768Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:04:39.509865023Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:04:39.5098759Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:04:39.509890811Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:04:39.509900018Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:04:39.509914928Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:04:39.509926444Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:04:39.509940129Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:04:39.509948236Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:04:39.509962012Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:04:39.509970403Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:04:39.50998415Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:04:39.509993637Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:04:39.510442385Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYSARQ6CH91NGBC1ZYA58Y/meta.json ts=2024-11-15T13:04:39.513082975Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=proxy msg="upload new block" id=01JCQYSARQ6CH91NGBC1ZYA58Y ts=2024-11-15T13:04:39.587725947Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYSARQ6CH91NGBC1ZYA58Y/chunks/000001 ts=2024-11-15T13:04:39.594364311Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=proxy msg="uploaded file" from=/data/tsdb/proxy/thanos/upload/01JCQYSARQ6CH91NGBC1ZYA58Y/chunks/000001 dst=01JCQYSARQ6CH91NGBC1ZYA58Y/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:04:39.594436615Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYSARQ6CH91NGBC1ZYA58Y/index ts=2024-11-15T13:04:39.637178993Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=proxy msg="uploaded file" from=/data/tsdb/proxy/thanos/upload/01JCQYSARQ6CH91NGBC1ZYA58Y/index dst=01JCQYSARQ6CH91NGBC1ZYA58Y/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:04:39.637232982Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYSARQ6CH91NGBC1ZYA58Y/meta.json ts=2024-11-15T13:04:39.642327451Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYSBDYZPC18C5Z67X3B4MB/meta.json ts=2024-11-15T13:04:39.644321797Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=proxy msg="upload new block" id=01JCQYSBDYZPC18C5Z67X3B4MB ts=2024-11-15T13:04:39.64801369Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYSBDYZPC18C5Z67X3B4MB/chunks/000001 ts=2024-11-15T13:04:39.662490589Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=proxy msg="uploaded file" from=/data/tsdb/proxy/thanos/upload/01JCQYSBDYZPC18C5Z67X3B4MB/chunks/000001 dst=01JCQYSBDYZPC18C5Z67X3B4MB/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:04:39.662580841Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYSBDYZPC18C5Z67X3B4MB/index ts=2024-11-15T13:04:39.714933604Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=proxy msg="uploaded file" from=/data/tsdb/proxy/thanos/upload/01JCQYSBDYZPC18C5Z67X3B4MB/index dst=01JCQYSBDYZPC18C5Z67X3B4MB/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:04:39.71497211Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYSBDYZPC18C5Z67X3B4MB/meta.json ts=2024-11-15T13:04:39.723728046Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=2 elapsed=213.882665ms ts=2024-11-15T13:04:57.331788355Z caller=head.go:1317 level=info component=receive component=multi-tsdb tenant=sdp msg="WAL checkpoint complete" first=996 last=1038 duration=1m0.237472019s ts=2024-11-15T13:05:07.752138816Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=firewall msg="write block" mint=1731664800000 maxt=1731672000000 ulid=01JCQYTSA8VCF1T1QAAPZ89KSQ duration=351.298286ms ooo=true ts=2024-11-15T13:05:09.509883766Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:05:09.509918448Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:05:09.509928191Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:05:09.50993458Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:05:09.509940336Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:05:09.509946586Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:05:09.509951828Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:05:09.509957128Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:05:09.50996704Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:05:09.50997323Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:05:09.509981386Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:05:09.509987371Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:05:09.509995312Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:05:09.510003782Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:05:09.510456898Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYTSA8VCF1T1QAAPZ89KSQ/meta.json ts=2024-11-15T13:05:09.51346033Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=firewall msg="upload new block" id=01JCQYTSA8VCF1T1QAAPZ89KSQ ts=2024-11-15T13:05:09.517266343Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYTSA8VCF1T1QAAPZ89KSQ/chunks/000001 ts=2024-11-15T13:05:09.523456002Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=firewall msg="uploaded file" from=/data/tsdb/firewall/thanos/upload/01JCQYTSA8VCF1T1QAAPZ89KSQ/chunks/000001 dst=01JCQYTSA8VCF1T1QAAPZ89KSQ/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:05:09.523528098Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYTSA8VCF1T1QAAPZ89KSQ/index ts=2024-11-15T13:05:09.556621651Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=firewall msg="uploaded file" from=/data/tsdb/firewall/thanos/upload/01JCQYTSA8VCF1T1QAAPZ89KSQ/index dst=01JCQYTSA8VCF1T1QAAPZ89KSQ/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:05:09.556666549Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYTSA8VCF1T1QAAPZ89KSQ/meta.json ts=2024-11-15T13:05:09.565110786Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=1 elapsed=55.192592ms ts=2024-11-15T13:05:09.922389467Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=firewall msg="write block" mint=1731672000000 maxt=1731679200000 ulid=01JCQYTSN815458D8W04HKJ38T duration=2.170201245s ooo=true ts=2024-11-15T13:05:09.922436699Z caller=db.go:1354 level=info component=receive component=multi-tsdb tenant=firewall msg="out-of-order compaction completed" duration=2.521596349s ulids="[01JCQYTSA8VCF1T1QAAPZ89KSQ 01JCQYTSN815458D8W04HKJ38T]" ts=2024-11-15T13:05:09.930178105Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=firewall msg="Deleting obsolete block" block=01JCQH309GMTRZ26V5TSPRJQHB ts=2024-11-15T13:05:09.93630394Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=firewall msg="Deleting obsolete block" block=01JCQA6PGFB4FS0FQAY1TRVGC6 ts=2024-11-15T13:05:10.382397141Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=firewall msg="Deleting obsolete block" block=01JCQGSZDAZ53001RR0W7X8HWA ts=2024-11-15T13:05:14.649375887Z caller=head.go:1355 level=info component=receive component=multi-tsdb tenant=firewall msg="Head GC completed" caller=truncateOOO duration=4.266635107s ts=2024-11-15T13:05:39.510355802Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:05:39.510408023Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:05:39.51043715Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:05:39.51044653Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:05:39.510457903Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:05:39.510467157Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:05:39.510483916Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:05:39.510495123Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:05:39.510508881Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:05:39.510517801Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:05:39.510530271Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:05:39.510536855Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:05:39.510549737Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:05:39.510569538Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:05:39.511116329Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYTSN815458D8W04HKJ38T/meta.json ts=2024-11-15T13:05:39.51437815Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=firewall msg="upload new block" id=01JCQYTSN815458D8W04HKJ38T ts=2024-11-15T13:05:39.519688397Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYTSN815458D8W04HKJ38T/chunks/000001 ts=2024-11-15T13:05:39.559810251Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=firewall msg="uploaded file" from=/data/tsdb/firewall/thanos/upload/01JCQYTSN815458D8W04HKJ38T/chunks/000001 dst=01JCQYTSN815458D8W04HKJ38T/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:05:39.559929958Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYTSN815458D8W04HKJ38T/index ts=2024-11-15T13:05:39.697383463Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=firewall msg="uploaded file" from=/data/tsdb/firewall/thanos/upload/01JCQYTSN815458D8W04HKJ38T/index dst=01JCQYTSN815458D8W04HKJ38T/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:05:39.69743017Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYTSN815458D8W04HKJ38T/meta.json ts=2024-11-15T13:05:39.707453904Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=1 elapsed=197.043482ms ts=2024-11-15T13:05:42.590187341Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=sdp msg="write block" mint=1731664800000 maxt=1731672000000 ulid=01JCQYVSGYE69YNHT7ZDZK63WY duration=2.207692189s ooo=true ts=2024-11-15T13:05:46.699810943Z caller=compact.go:580 level=info component=receive component=multi-tsdb tenant=sdp msg="write block" mint=1731672000000 maxt=1731679200000 ulid=01JCQYVVNYZH04A92XNYGKBRPH duration=4.10958597s ooo=true ts=2024-11-15T13:05:46.69985772Z caller=db.go:1354 level=info component=receive component=multi-tsdb tenant=sdp msg="out-of-order compaction completed" duration=6.317368775s ulids="[01JCQYVSGYE69YNHT7ZDZK63WY 01JCQYVVNYZH04A92XNYGKBRPH]" ts=2024-11-15T13:05:46.723385115Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=sdp msg="Deleting obsolete block" block=01JCQA87JXNM54E9EWK3RTVBFV ts=2024-11-15T13:05:47.290770365Z caller=db.go:1716 level=info component=receive component=multi-tsdb tenant=sdp msg="Deleting obsolete block" block=01JCQGSZT8KJ3KJVYRZM6B990B ts=2024-11-15T13:05:53.637458325Z caller=head.go:1355 level=info component=receive component=multi-tsdb tenant=sdp msg="Head GC completed" caller=truncateOOO duration=6.346197414s ts=2024-11-15T13:06:09.509985222Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:06:09.510018245Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:06:09.510032131Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:06:09.510037593Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:06:09.510042945Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:06:09.510048766Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:06:09.510056821Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:06:09.51006341Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:06:09.510068884Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:06:09.510075921Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:06:09.51008539Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:06:09.510101497Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:06:09.51011007Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:06:09.510115266Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:06:09.510453537Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYVSGYE69YNHT7ZDZK63WY/meta.json ts=2024-11-15T13:06:09.513123354Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=sdp msg="upload new block" id=01JCQYVSGYE69YNHT7ZDZK63WY ts=2024-11-15T13:06:09.521681147Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYVSGYE69YNHT7ZDZK63WY/chunks/000001 ts=2024-11-15T13:06:09.526768654Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=sdp msg="uploaded file" from=/data/tsdb/sdp/thanos/upload/01JCQYVSGYE69YNHT7ZDZK63WY/chunks/000001 dst=01JCQYVSGYE69YNHT7ZDZK63WY/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:06:09.526834773Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYVSGYE69YNHT7ZDZK63WY/index ts=2024-11-15T13:06:09.586309108Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=sdp msg="uploaded file" from=/data/tsdb/sdp/thanos/upload/01JCQYVSGYE69YNHT7ZDZK63WY/index dst=01JCQYVSGYE69YNHT7ZDZK63WY/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:06:09.586353801Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYVSGYE69YNHT7ZDZK63WY/meta.json ts=2024-11-15T13:06:09.592512936Z caller=azure.go:301 level=debug component=receive msg="checking if blob exists" blob=01JCQYVVNYZH04A92XNYGKBRPH/meta.json ts=2024-11-15T13:06:09.595497033Z caller=shipper.go:372 level=info component=receive component=multi-tsdb tenant=sdp msg="upload new block" id=01JCQYVVNYZH04A92XNYGKBRPH ts=2024-11-15T13:06:09.599025109Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYVVNYZH04A92XNYGKBRPH/chunks/000001 ts=2024-11-15T13:06:09.623031609Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=sdp msg="uploaded file" from=/data/tsdb/sdp/thanos/upload/01JCQYVVNYZH04A92XNYGKBRPH/chunks/000001 dst=01JCQYVVNYZH04A92XNYGKBRPH/chunks/000001 bucket="tracing: thanos-blocks" ts=2024-11-15T13:06:09.623137168Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYVVNYZH04A92XNYGKBRPH/index ts=2024-11-15T13:06:09.697870443Z caller=objstore.go:291 level=debug component=receive component=multi-tsdb tenant=sdp msg="uploaded file" from=/data/tsdb/sdp/thanos/upload/01JCQYVVNYZH04A92XNYGKBRPH/index dst=01JCQYVVNYZH04A92XNYGKBRPH/index bucket="tracing: thanos-blocks" ts=2024-11-15T13:06:09.697912119Z caller=azure.go:314 level=debug component=receive msg="uploading blob" blob=01JCQYVVNYZH04A92XNYGKBRPH/meta.json ts=2024-11-15T13:06:09.707345644Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=2 elapsed=197.326763ms ts=2024-11-15T13:06:39.509907552Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:06:39.509935235Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:06:39.509949099Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:06:39.509957349Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:06:39.509963098Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:06:39.509969998Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:06:39.509975836Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:06:39.509983948Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:06:39.509989431Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:06:39.509998474Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:06:39.510003411Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:06:39.510009783Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:06:39.510016113Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:06:39.510025192Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:06:39.519098184Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=0 elapsed=9.164536ms ts=2024-11-15T13:07:09.509893356Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:07:09.509928086Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:07:09.509941257Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:07:09.509951759Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:07:09.509958153Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:07:09.509962971Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:07:09.509969131Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:07:09.509974741Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:07:09.509980816Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:07:09.509989182Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:07:09.509995709Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:07:09.510006466Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:07:09.510015206Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:07:09.510024748Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:07:09.522705555Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=0 elapsed=12.774735ms ts=2024-11-15T13:07:39.50980852Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:07:39.509833256Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:07:39.509842762Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:07:39.509848292Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:07:39.50985302Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:07:39.50985929Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:07:39.509864092Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:07:39.509868494Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:07:39.509875574Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:07:39.509880465Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:07:39.509887797Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:07:39.509892577Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:07:39.509899601Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:07:39.50990454Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:07:39.519172659Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=0 elapsed=9.340746ms ts=2024-11-15T13:08:09.510128752Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:08:09.510202818Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:08:09.510217891Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:08:09.510229716Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:08:09.510240115Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:08:09.510247758Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:08:09.510264234Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:08:09.510308164Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:08:09.510335145Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:08:09.510353795Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:08:09.510369707Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:08:09.510391578Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:08:09.510407113Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:08:09.510521415Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:08:09.52500339Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=0 elapsed=14.821041ms ts=2024-11-15T13:08:39.509808465Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:08:39.509832655Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:08:39.509849296Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:08:39.509858062Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:08:39.509863837Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:08:39.509869795Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:08:39.50988719Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:08:39.50989384Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:08:39.509903772Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:08:39.509918838Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:08:39.509928711Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:08:39.509937704Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:08:39.509949436Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:08:39.509959548Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:08:39.517564498Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=0 elapsed=7.732316ms ts=2024-11-15T13:09:09.509959472Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:09:09.510000441Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:09:09.510014724Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:09:09.510022918Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:09:09.510030835Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:09:09.510035987Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:09:09.510041115Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:09:09.510047356Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:09:09.51005305Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:09:09.510062839Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:09:09.510071087Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:09:09.510080661Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:09:09.510087563Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:09:09.510092601Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:09:09.517208983Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=0 elapsed=7.206266ms ts=2024-11-15T13:09:39.509820091Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:09:39.509851498Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:09:39.509864511Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:09:39.509869602Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:09:39.509874106Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:09:39.509878684Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:09:39.509884035Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:09:39.509889129Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:09:39.509897515Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:09:39.509905439Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:09:39.509915526Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:09:39.509920903Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:09:39.509929165Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:09:39.509933954Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:09:39.517214813Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=0 elapsed=7.361ms ts=2024-11-15T13:10:09.510006732Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:10:09.510038297Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:10:09.510050868Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:10:09.510057492Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:10:09.510062811Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:10:09.510068797Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:10:09.510073946Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:10:09.510078742Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:10:09.510083114Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:10:09.510088013Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:10:09.510093651Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:10:09.510103338Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:10:09.510107999Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:10:09.510117187Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:10:09.517283516Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=0 elapsed=7.244421ms ts=2024-11-15T13:10:39.509819453Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:10:39.509849616Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:10:39.509869068Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:10:39.509882593Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:10:39.509906323Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:10:39.509918177Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:10:39.509934063Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:10:39.509942828Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:10:39.509965803Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:10:39.509974352Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:10:39.509982191Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:10:39.509995563Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:10:39.51000379Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:10:39.510020559Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:10:39.516919558Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=0 elapsed=7.07124ms ts=2024-11-15T13:11:09.510625231Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:11:09.51066225Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:11:09.510680426Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:11:09.510688185Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:11:09.510695304Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:11:09.510700578Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:11:09.510706156Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:11:09.510715619Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:11:09.510724734Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:11:09.510732098Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:11:09.510740513Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:11:09.510745768Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:11:09.510751803Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:11:09.510760704Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:11:09.517557842Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=0 elapsed=6.898597ms ts=2024-11-15T13:11:39.509964458Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:11:39.509987606Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:11:39.509998137Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:11:39.510004126Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:11:39.510010509Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:11:39.510017091Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:11:39.510023339Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:11:39.510031961Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:11:39.510037369Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:11:39.510043927Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:11:39.510049768Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:11:39.510062774Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:11:39.51007369Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:11:39.510081944Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:11:39.516265568Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=0 elapsed=6.278439ms ts=2024-11-15T13:12:09.509872501Z caller=receive.go:671 level=debug component=receive component=uploader msg="upload phase starting" ts=2024-11-15T13:12:09.509904455Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=cloudinfrastructure ts=2024-11-15T13:12:09.509921561Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=centralinfrastructure ts=2024-11-15T13:12:09.509927362Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=bandwidthcontrol ts=2024-11-15T13:12:09.509932439Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=nsm ts=2024-11-15T13:12:09.509937409Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=firewall ts=2024-11-15T13:12:09.509942629Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=probing ts=2024-11-15T13:12:09.509947884Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=sdp ts=2024-11-15T13:12:09.509952161Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=wan ts=2024-11-15T13:12:09.509956573Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=ztna ts=2024-11-15T13:12:09.509964715Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mail ts=2024-11-15T13:12:09.509972902Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=rule ts=2024-11-15T13:12:09.509978491Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=proxy ts=2024-11-15T13:12:09.509986879Z caller=multitsdb.go:559 level=debug component=receive component=multi-tsdb msg="uploading block for tenant" tenant=mdr ts=2024-11-15T13:12:09.516595947Z caller=receive.go:679 level=debug component=receive component=uploader msg="upload phase done" uploaded=0 elapsed=6.690026ms ts=2024-11-15T13:12:19.431905741Z caller=writer.go:163 level=debug component=receive component=receive-writer tenant=nsm msg="Sample is too old" ```

dosubot[bot] commented 1 day ago

I found a similar discussion regarding autoscaling best practices with Thanos in a router-ingestor configuration, which might be relevant to your issue. The discussion is still open and can be found here [1].

Regarding your issue with latency spikes during block cutting, here are a few suggestions that might help:

  1. gRPC Timeout Configuration: You mentioned gRPC timeouts of 5 seconds. You can configure gRPC timeouts in Thanos using parameters like grpc-server-max-connection-age, grpc-grace-period, and endpoint.info-timeout. Adjusting these settings might help manage the connection stability during block cutting [2][3][4].

  2. Resource Management: Ensure that your CPU and memory allocations are sufficient for the load. Thanos recommends matching CPU resources to the --compact.concurrency setting and suggests a memory limit of around 10GB for medium-sized buckets to handle compaction smoothly [5][6].

  3. Handling High Ingestion Rates: Consider tuning the remote write protocol and implementing series and samples limits to manage payload sizes. Deploying multiple receive instances and using the --receive.replication-factor flag can help ensure data duplication and durability [7].

These adjustments might help mitigate the latency spikes and improve the overall performance of your Thanos setup.

To continue talking to Dosu, mention @dosu.

GiedriusS commented 1 day ago

https://github.com/prometheus/prometheus/pull/15374 should help, I think. I have been a bit busy recently so I didn't have time to dig into it 100%