search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.83k stars 3.55k forks source link

Compactions causing iowait #24063

Open ahsanali opened 1 year ago

ahsanali commented 1 year ago

Our Influx db starts 8 compactions at a time which uses up all the memory and causes cpu iowait. The iowait is not induced by the disk and we do see swapping on the instance when this starts. We are aware of the series cardinality issues on our database, is it caused by this ? if not then is there a way to make the compactions less intense. So far I have tried to use max-concurrent-compactions and compact-throughput-* but to no avail. Can someone please advise what we are doing wrong here ? Meanwhile I will try to upgrade to the latest minor version.

Linux 3.16.0-5-amd64 x86_64 Influx version: v1.8.6 block.txt diagnostics.txt goroutine.txt iostat.csv profiles.tar.gz shards.txt stats.txt vars.txt heap.txt.zip

logs.csv config.txt

Screen Shot 2023-01-25 at 3 10 17 PM Screen Shot 2023-01-25 at 3 07 38 PM
ahsanali commented 1 year ago

Upgraded to v1.8.10 still the same issue unfortunately :(

ahsanali commented 1 year ago

FYI the compaction is stuck on shard 3712. Below is the RP


> show RETENTION POLICIES
name    duration shardGroupDuration replicaN default
----    -------- ------------------ -------- -------
autogen 504h0m0s 168h0m0s           1        true
~```