[DocDB] Frequent Paused Compactions and High CPU Usage on one node; Observed many SST Files Not Compacted

[shamanthchandra-yb]

Jira Link: DB-12659

Description

A potential bug has been observed in the YugabyteDB cluster where SST files are not being compacted as expected, and node n2 is experiencing frequent paused compactions along with high CPU usage. Please find slack thread in JIRA description.

Setup Details:

• Replication Factor (RF): 5
• Nodes: 7 (8 cores c5.2xlarge machines)
• YCQL Workload: Cassandra Batch Time Series on 4 keyspaces, with 5 write threads and 1 read thread each
• This is observed on 2.14 but also observed in another experiment after upgrading it to 2.20.6

Configuration:

• For #proj-ttl-expiry setup
• rocksdb_max_file_size_for_compaction: 300 MB
• TTL: 6 hours

Observations:

1. Node n2 (who was also a master leader) showed high CPU usage, reaching up to 97%, with compaction threads piling up. Disk utilisation was around 50% on each disk (confirmed via iostat), while other nodes maintained around 75% CPU usage.
2. We suspected a bug in the priority pool. We should certainly see why we mark tasks as non active and never review them.
3. Need to find RCA for why there are so many paused compactions. Latest update from slack, by @arybochkin :

We restarted n2 to get the fresh set of logs. It helped with number of SST files — all old files have been deleted by ‘universal deletion compaction’, which again confirms the files had been hold by some paused compactions, and that lead to a huge reduction of SST files and write rejections stoped as expected. I’m monitoring the situation to get more pointers from the fresh logs, seems the situation is going to repeat, as I already see a couple of dozens of paused compactions.

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• [X] I confirm this issue does not contain any sensitive information.

[rthallamko3]

Per @ttyusupov "Compactions tasks priorities are frequently changed because they are based on number of SST files in current RocksDB state. And that is causing pausing and transferring control to tasks with higher priority again and again. That was increasing the effect of going out of stable state, because that makes node to work slowly on almost all 150-250 background compactions switching between them instead of completing them one by one."

The following tserver gflags changes helped as it avoided frequent priority changes to compactions tasks and therefore pausing/resuming of compactions.

compaction_priority_step_size=10
compaction_priority_start_bound=20

[rthallamko3]

As part of the analysis by @ttyusupov , We identified two items:

• [x] #24540 : Avoid adding new compaction task for the same RocksDB instance while we already have a pending one. That was the way compaction logic worked in original RocksDB but wasn’t implemented for compaction tasks with priority thread pool implementation. This logic allows for one large (major) and one small (minor) compaction to be scheduled for a tablet.

• [ ] #24541 (In review) - Determine input files at the compaction start time instead of the time compaction was scheduled. Original RocksDB compaction queue stored pointers to ColumnFamilyData instead of compactions and decision about which files to include in compaction was made at compaction start. However it was changed in Yugabyte DB as part of D2700 (in 2017) to support small/large compaction queues. The plan is to go back to the RockDB stratgey by adding rocksdb_determine_compaction_input_at_start flag which switches RocksDB to put pointer to ColumnFamilyData and presumed compaction size (small/large) into priority thread pool CompactionTask object. Once compaction task is actually started, we pick the appropriate compaction and its input files. Delaying picking files should help with better compaction decision and potentially avoid repeated work.

The two ideas together should help with better compaction strategies, preventing too many compactions in the queue etc.