search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.14k stars 2.1k forks source link

Longer loading times for index headers with Thanos Receive compared to Thanos Sidecar #7663

Open roth-wine opened 3 months ago

roth-wine commented 3 months ago

Thanos, Prometheus and Golang version used:

Thanos: v0.36.1
Prometheus: v2.53.1

Object Storage Provider: Ceph Cluster

What happened: We noticed that our Thanos Stores, which uses an Objectstorage as storage backend, has significant longer loading times per block/index when the are generated from the Thanos Receiver compared to Thanos Sidecar. We then checked the two buckets, which both have a similar size and the blocks are also similar in size. For Tenant 1 we have 3112 index files with an average of 348MB. For tenant 2 we have 2993 index files with an average of 309MB per file. But this does not explain why Tenant1 takes on average 2-3 times as long to load per Index Header compared to Tenant2.

What you expected to happen: Regardless of which data source the blocks are written from, according to the documentation they should make no difference to how the Thanos Store generates the index headers. With similarly large buckets and a similarly large number of blocks to be loaded, almost the same speeds should be achieved when generating the index header/block.

How to reproduce it (as minimally and precisely as possible): Compares a bucket which was filled with Thanos Receive data and one which was filled with Thanos Sidecar data. Then start a store for the receive bucket and a store for the sidecar bucket. You should now be able to recognise the differences.

Full logs to relevant components:

Receiver Blocks Sample Output from `thanos tools bucket inspect` ``` | ULID | FROM | UNTIL | RANGE | UNTIL-DOWN | #SERIES | #SAMPLES | #CHUNKS | COMP-LEVEL | COMP-FAILED | LABELS | RESOLUTION | SOURCE | |----------------------------|----------------------|----------------------|---------------|----------------|-----------|---------------|------------|------------|-------------|--------------------------------------------------------------------------------------------------------------|------------|-----------| | 01J5ADJ0YDNACNJDHHZQ8569R9 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,390,745 | 6,632,884,192 | 53,817,971 | 4 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-1,tenant_id=tenant1-storage | 0s | compactor | | 01J5AE1S6B4B3SQT2N2P90RFZE | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,470,793 | 6,822,144,336 | 54,662,046 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-20,tenant_id=tenant1-storage | 0s | compactor | | 01J5AENP6H3APP8HMAZ4YB1Q69 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,523,468 | 7,296,872,037 | 58,401,675 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-19,tenant_id=tenant1-storage | 0s | compactor | | 01J5AF6JVAACJHSJKDPD0PWT9S | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,473,601 | 6,940,360,938 | 55,575,188 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-22,tenant_id=tenant1-storage | 0s | compactor | | 01J5AFKCBDFXHGASHFXX0X53DN | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,472,729 | 6,954,088,819 | 55,707,100 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-16,tenant_id=tenant1-storage | 0s | compactor | | 01J5AG38NNJH1N6TYF2V1F9XE3 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,499,885 | 6,926,953,022 | 55,516,517 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-0,tenant_id=tenant1-storage | 0s | compactor | | 01J5AGFCSBEHH2T4Z4W0GM2575 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,391,613 | 6,727,877,419 | 54,527,334 | 4 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-15,tenant_id=tenant1-storage | 0s | compactor | | 01J5AGWMQJX829DA1KFV7EQKSY | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,296,077 | 6,450,519,360 | 53,433,198 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-23,tenant_id=tenant1-storage | 0s | compactor | | 01J5AJ0C392CNMDJH9BEZFY3J4 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,483,324 | 6,940,353,514 | 55,577,142 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-6,tenant_id=tenant1-storage | 0s | compactor | | 01J5AJW46CSWWCMF31R32P5JKN | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,520,486 | 7,222,286,006 | 57,850,518 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-5,tenant_id=tenant1-storage | 0s | compactor | | 01J5AKJJ831QYYFX0JRCZHDER9 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,513,029 | 7,125,389,232 | 57,103,805 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-18,tenant_id=tenant1-storage | 0s | compactor | | 01J5AM79Y8R9R7GCKWWVZHEH99 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,447,667 | 6,800,683,513 | 54,452,452 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-13,tenant_id=tenant1-storage | 0s | compactor | | 01J5ANEJ1KQ0NR41AG4PEZSFP7 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,507,697 | 7,077,556,168 | 56,662,247 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-8,tenant_id=tenant1-storage | 0s | compactor | | 01J5APA35Y4C5R39ASY2T8GYGV | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,510,407 | 7,045,648,983 | 56,417,914 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-4,tenant_id=tenant1-storage | 0s | compactor | | 01J5APY3B0XS4YQ16Q2EBWR9VK | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,426,950 | 6,718,865,003 | 53,820,753 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-7,tenant_id=tenant1-storage | 0s | compactor | | 01J5AQF2AC5FRZM83AMB7FA4FB | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,479,027 | 6,856,283,653 | 54,953,937 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-10,tenant_id=tenant1-storage | 0s | compactor | | 01J5AQZME2KQ702WQZ9WHA9343 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,399,707 | 6,645,819,614 | 53,204,040 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-17,tenant_id=tenant1-storage | 0s | compactor | | 01J5ARWPSASD1V3D59NCVC4CSM | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,550,251 | 7,261,090,323 | 58,164,387 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-14,tenant_id=tenant1-storage | 0s | compactor | | 01J5ASDK72HZGNM4MBA4J5FSCG | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,515,668 | 7,131,415,720 | 57,106,606 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-2,tenant_id=tenant1-storage | 0s | compactor | | 01J5AT1BGWF7355BJ1KAYDRKZ1 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,476,254 | 7,070,666,245 | 56,617,297 | 3 | false | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-3,tenant_id=tenant1-storage | 0s | compactor | ```
Sidecar Blocks Sample Output from `thanos tools bucket inspect` ``` | ULID | FROM | UNTIL | RANGE | UNTIL-DOWN | #SERIES | #SAMPLES | #CHUNKS | COMP-LEVEL | COMP-FAILED | LABELS | RESOLUTION | SOURCE | |----------------------------|----------------------|----------------------|---------------|---------------|-----------|---------------|------------|------------|-------------|----------------------------------------------------------------------------------------------------------|------------|-----------| | 01J5AAF5HPF2PFW5ABW85SF0WT | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,167,658 | 5,209,825,975 | 46,609,953 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-10,prometheus_replica=prometheus-sidecar-cnodes-10-0,region=fsn1 | 0s | compactor | | 01J5AAXV5ATB0HMGVJR6W1FZ2V | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | -8h0m0s | 1,089,725 | 5,059,691,020 | 46,056,643 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-1,prometheus_replica=prometheus-sidecar-cnodes-1-0,region=fsn1 | 0s | compactor | | 01J5AB7V3H3J0E5FR1M8GA00GP | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | 192h0m0s | 1,167,658 | 624,417,950 | 4,694,514 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-10,prometheus_replica=prometheus-sidecar-cnodes-10-0,region=fsn1 | 5m0s | compactor | | 01J5ABQ09FP3JD4QK5F6C3JAX0 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s | 192h0m0s | 1,089,725 | 609,356,419 | 4,519,046 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-1,prometheus_replica=prometheus-sidecar-cnodes-1-0,region=fsn1 | 5m0s | compactor | | 01J5AN61W5608MDYQDY4Q2NYWV | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.951s | -7h59m59.951s | 1,466,663 | 6,651,517,461 | 59,629,508 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-11,prometheus_replica=prometheus-sidecar-cnodes-11-0,region=fsn1 | 0s | compactor | | 01J5ANN2BX9XNR4M29YXWYBVDB | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.998s | -7h59m59.998s | 1,582,493 | 7,864,742,275 | 69,268,695 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-22,prometheus_replica=prometheus-sidecar-cnodes-22-0,region=fsn1 | 0s | compactor | | 01J5APAJ59PFEAZTAMACFPNV98 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.992s | -7h59m59.992s | 1,333,182 | 6,461,683,230 | 57,603,142 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-15,prometheus_replica=prometheus-sidecar-cnodes-15-0,region=fsn1 | 0s | compactor | | 01J5AQ1NPR8M6B4NGFMV12EHM3 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.998s | -7h59m59.998s | 1,613,122 | 7,945,622,683 | 70,102,189 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-19,prometheus_replica=prometheus-sidecar-cnodes-19-0,region=fsn1 | 0s | compactor | | 01J5AQQ0N1EXJ6X50G1YZSA4AE | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.978s | -7h59m59.978s | 1,330,199 | 6,189,685,122 | 55,563,851 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-12,prometheus_replica=prometheus-sidecar-cnodes-12-0,region=fsn1 | 0s | compactor | | 01J5AR5R0KP9FA2KDZP5SBDWVX | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.975s | -7h59m59.975s | 1,308,462 | 6,319,618,992 | 56,376,946 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-2,prometheus_replica=prometheus-sidecar-cnodes-2-0,region=fsn1 | 0s | compactor | | 01J5ARHMGKAK7JR90VGPDKGQDR | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.993s | -7h59m59.993s | 1,032,887 | 4,753,217,152 | 43,427,265 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-9,prometheus_replica=prometheus-sidecar-cnodes-9-0,region=fsn1 | 0s | compactor | | 01J5ARVN1S460E8G55ZC1VH0J5 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.952s | -7h59m59.952s | 879,900 | 4,269,481,051 | 37,960,012 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-24,prometheus_replica=prometheus-sidecar-cnodes-24-0,region=fsn1 | 0s | compactor | | 01J5ASA4NHPFQP41M52CN32SKH | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.993s | -7h59m59.993s | 1,423,181 | 7,007,498,147 | 62,191,611 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-14,prometheus_replica=prometheus-sidecar-cnodes-14-0,region=fsn1 | 0s | compactor | | 01J5ASTXDVACMQ8X28Z2WP07PE | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.999s | -7h59m59.999s | 1,319,988 | 6,271,030,141 | 56,402,900 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-3,prometheus_replica=prometheus-sidecar-cnodes-3-0,region=fsn1 | 0s | compactor | | 01J5ATBB4N72TSMSB6K3WA8HCM | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.991s | -7h59m59.991s | 1,306,804 | 6,213,659,211 | 55,874,472 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-5,prometheus_replica=prometheus-sidecar-cnodes-5-0,region=fsn1 | 0s | compactor | | 01J5ATSCJM8J7N0P2FC3H69QQV | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.998s | -7h59m59.998s | 1,430,156 | 6,383,481,921 | 57,123,055 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-17,prometheus_replica=prometheus-sidecar-cnodes-17-0,region=fsn1 | 0s | compactor | | 01J5AVAHJNP1AGX8CETWPDCNHM | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.992s | -7h59m59.992s | 1,230,953 | 5,984,898,253 | 53,706,705 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-13,prometheus_replica=prometheus-sidecar-cnodes-13-0,region=fsn1 | 0s | compactor | | 01J5AVPZ5XTCBP0091RV7ABH67 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.998s | -7h59m59.998s | 1,293,051 | 5,843,361,729 | 52,638,926 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-4,prometheus_replica=prometheus-sidecar-cnodes-4-0,region=fsn1 | 0s | compactor | | 01J5AW2NH655C3GZYTHDHPD84N | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.995s | -7h59m59.995s | 1,455,648 | 6,593,190,912 | 58,906,486 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-18,prometheus_replica=prometheus-sidecar-cnodes-18-0,region=fsn1 | 0s | compactor | | 01J5AWG07X66J4D30T82505G1V | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.985s | -7h59m59.985s | 1,073,459 | 4,956,575,766 | 45,101,988 | 3 | false | monitor=cnodes,prom=prom-tenant2-cnodes-8,prometheus_replica=prometheus-sidecar-cnodes-8-0,region=fsn1 | 0s | compactor | ```
thanos_bucket_store_blocks_loaded sum (thanos_bucket_store_blocks_loaded{region="fsn1", pod="store-test-0") by (namespace, region) ``` {namespace="monitoring-tenant1-storage", region="fsn1"} 2497 {namespace="scrape-tenant2-cnodes", region="fsn1"} 2491 ```
thanos_bucket_store_indexheader_load_duration_seconds_sum ``` Green: Tenant1 Blue: Tenant2 ``` ![image](https://github.com/user-attachments/assets/81e00cff-5e7f-411d-8dff-027689205f73)
thanos_objstore_bucket_operation_transferred_bytes_sum ``` Green: Tenant1 Blue: Tenant2 ``` ![image](https://github.com/user-attachments/assets/0defb2b8-9376-463f-837f-7d5a59f62638)
yeya24 commented 3 months ago

Is this always reproducible or you discovered this issue the first time?

For Tenant 1 we have 3112 index files with an average of 348MB

Have you checked the largest index file size?

roth-wine commented 2 months ago

Hey, yeah this is always reproducible even with new buckets. Currently we scrape and receive the same data for one tenant and also for this tenant this behavior is reproducible.

The largest index file is about 800MB.