GiedriusS
commented
2 months ago Is thanos_compact_iterations_total more than 0? :thinking:
Open vincent-olivert-riera opened 2 months ago
GiedriusS
commented
2 months ago Is thanos_compact_iterations_total more than 0? :thinking:
vincent-olivert-riera
commented
2 months ago Is
thanos_compact_iterations_totalmore than 0? 🤔
Yes, it is constantly growing.
This is how thanos_compact_todo_compactions compares with thanos_compact_iterations_total:
douglascamata
commented
2 months ago @vincent-olivert-riera can you show us some information about the level 4 blocks you mentioned? What's their duration?
vincent-olivert-riera
commented
2 months ago @vincent-olivert-riera can you show us some information about the level 4 blocks you mentioned? What's their duration?
Sure.
douglascamata
commented
2 months ago @vincent-olivert-riera if you grep your Compactor's log with block IDs of the blocks that didn't get compacted, do you see anything that stands out? If possible, maybe increase the Compactor's log level to generate more logs (then revert it, otherwise logs might be too spammy). 🤔
vincent-olivert-riera
commented
2 months ago @douglascamata , I haven't increased the Compactor's log level yet, but this is what the Compactor is doing (in a loop):
Apr 19, 2024 @ 20:29:37.120{"caller":"compact.go:1478","level":"info","msg":"compaction iterations done","ts":"2024-04-19T11:29:29.342094884Z"}
Apr 19, 2024 @ 20:29:37.120{"caller":"compact.go:457","level":"info","msg":"downsampling was explicitly disabled","ts":"2024-04-19T11:29:29.342370667Z"}
Apr 19, 2024 @ 20:27:42.421{"cached":346,"caller":"fetcher.go:487","component":"block.BaseFetcher","duration":"8.78195813s","duration_ms":8781,"level":"info","msg":"successfully synchronized block metadata","partial":0,"returned":346,"ts":"2024-04-19T11:26:37.988636543Z"}
Apr 19, 2024 @ 20:27:42.421{"caller":"fetcher.go:317","component":"block.BaseFetcher","concurrency":32,"level":"debug","msg":"fetching meta data","ts":"2024-04-19T11:27:29.206720687Z"}
Apr 19, 2024 @ 20:27:42.421{"cached":346,"caller":"fetcher.go:487","component":"block.BaseFetcher","duration":"7.076132358s","duration_ms":7076,"level":"info","msg":"successfully synchronized block metadata","partial":0,"returned":346,"ts":"2024-04-19T11:27:36.282764217Z"}
Apr 19, 2024 @ 20:26:36.158{"caller":"fetcher.go:317","component":"block.BaseFetcher","concurrency":32,"level":"debug","msg":"fetching meta data","ts":"2024-04-19T11:25:29.206734945Z"}
Apr 19, 2024 @ 20:26:36.158{"cached":346,"caller":"fetcher.go:487","component":"block.BaseFetcher","duration":"7.2963137s","duration_ms":7296,"level":"info","msg":"successfully synchronized block metadata","partial":0,"returned":346,"ts":"2024-04-19T11:25:36.502939791Z"}
Apr 19, 2024 @ 20:26:36.158{"caller":"fetcher.go:317","component":"block.BaseFetcher","concurrency":32,"level":"debug","msg":"fetching meta data","ts":"2024-04-19T11:26:29.20683454Z"}
Apr 19, 2024 @ 20:25:37.421{"caller":"compact.go:1414","level":"info","msg":"start sync of metas","ts":"2024-04-19T11:24:22.419242154Z"}
Apr 19, 2024 @ 20:25:37.421{"caller":"fetcher.go:317","component":"block.BaseFetcher","concurrency":32,"level":"debug","msg":"fetching meta data","ts":"2024-04-19T11:24:22.419842845Z"}
Apr 19, 2024 @ 20:25:37.421{"cached":346,"caller":"fetcher.go:487","component":"block.BaseFetcher","duration":"5.786435988s","duration_ms":5786,"level":"info","msg":"successfully synchronized block metadata","partial":0,"returned":174,"ts":"2024-04-19T11:24:28.206118667Z"}
Apr 19, 2024 @ 20:25:37.421{"caller":"compact.go:1419","level":"info","msg":"start of GC","ts":"2024-04-19T11:24:28.20786563Z"}
Apr 19, 2024 @ 20:25:37.421{"caller":"compact.go:1442","level":"info","msg":"start of compactions","ts":"2024-04-19T11:24:28.208735693Z"}I have search for all the block IDs, but Kibana does not return anything at all.
~I will try to increase the log level and see what happens.~ The log level is debug.
Thanos, Prometheus and Golang version used:
Thanos: 0.32.4 Golang: go1.20.8
Prometheus: 2.45.0 goVersion: go1.20.5
Object Storage Provider:
Openstack S3 compatible
What happened:
I have a Thanos compactor with the following metrics:
It is tracking a bucket where almost all blocks have been compacted up to level-4. However, there are some level-1 blocks that are not compacted, and I was expecting them to be compacted into a level-2 block. I have made this animated gif to show it more clearly:
None of those blocks has been marked as no-compaction, so they should be compacted.
These are the
meta.jsonfor each one of them:01HT1G02DF2W21A1KTHDVPX0BR
``` { "ulid": "01HT1G02DF2W21A1KTHDVPX0BR", "minTime": 1711584000246, "maxTime": 1711591200000, "stats": { "numSamples": 2492646, "numSeries": 5196, "numChunks": 20775 }, "compaction": { "level": 1, "sources": [ "01HT1G02DF2W21A1KTHDVPX0BR" ] }, "version": 1, "thanos": { "labels": { "cluster_name": "alpha", "cluster_node": "prometheus004", "datasource": "alpha-002" }, "downsample": { "resolution": 0 }, "source": "sidecar", "segment_files": [ "000001" ], "files": [ { "rel_path": "chunks/000001", "size_bytes": 3964613 }, { "rel_path": "index", "size_bytes": 646029 }, { "rel_path": "meta.json" } ], "index_stats": {} } } ```01HT1PVSMCNYF8ZSDW53123NJX
``` { "ulid": "01HT1PVSMCNYF8ZSDW53123NJX", "minTime": 1711591200246, "maxTime": 1711598400000, "stats": { "numSamples": 2492640, "numSeries": 5193, "numChunks": 20772 }, "compaction": { "level": 1, "sources": [ "01HT1PVSMCNYF8ZSDW53123NJX" ] }, "version": 1, "thanos": { "labels": { "cluster_name": "alpha", "cluster_node": "prometheus004", "datasource": "alpha-002" }, "downsample": { "resolution": 0 }, "source": "sidecar", "segment_files": [ "000001" ], "files": [ { "rel_path": "chunks/000001", "size_bytes": 3957077 }, { "rel_path": "index", "size_bytes": 644900 }, { "rel_path": "meta.json" } ], "index_stats": {} } } ```01HT1XQGXB5CHQB21YT5DNXFC8
``` { "ulid": "01HT1XQGXB5CHQB21YT5DNXFC8", "minTime": 1711598400246, "maxTime": 1711605600000, "stats": { "numSamples": 2492640, "numSeries": 5193, "numChunks": 20772 }, "compaction": { "level": 1, "sources": [ "01HT1XQGXB5CHQB21YT5DNXFC8" ] }, "version": 1, "thanos": { "labels": { "cluster_name": "alpha", "cluster_node": "prometheus004", "datasource": "alpha-002" }, "downsample": { "resolution": 0 }, "source": "sidecar", "segment_files": [ "000001" ], "files": [ { "rel_path": "chunks/000001", "size_bytes": 3969637 }, { "rel_path": "index", "size_bytes": 645540 }, { "rel_path": "meta.json" } ], "index_stats": {} } } ```01HT24K86QTXJ1HV2NW252DAEV
``` { "ulid": "01HT24K86QTXJ1HV2NW252DAEV", "minTime": 1711605600246, "maxTime": 1711612800000, "stats": { "numSamples": 2492646, "numSeries": 5196, "numChunks": 20775 }, "compaction": { "level": 1, "sources": [ "01HT24K86QTXJ1HV2NW252DAEV" ] }, "version": 1, "thanos": { "labels": { "cluster_name": "alpha", "cluster_node": "prometheus004", "datasource": "alpha-002" }, "downsample": { "resolution": 0 }, "source": "sidecar", "segment_files": [ "000001" ], "files": [ { "rel_path": "chunks/000001", "size_bytes": 3981026 }, { "rel_path": "index", "size_bytes": 645293 }, { "rel_path": "meta.json" } ], "index_stats": {} } } ```This is the command line that I'm using:
Contents of /etc/thanos/objstore.yml
``` type: S3 config: bucket: "thanos-alpha" endpoint: "redacted" access_key: "redacted" insecure: false signature_version2: false secret_key: "redacted" list_objects_version: "v1" http_config: idle_conn_timeout: 60s ```Contents of /etc/thanos/relabel_config.yml
``` - action: keep regex: "alpha-002" source_labels: - datasource ```What could be the reason for this behavior?