search

apache / druid

Apache Druid: a high performance real-time analytics database.
https://druid.apache.org/
Apache License 2.0
13.37k stars 3.67k forks source link

Coordinator is not calulating compaction taskSlots correctly #16694

Open sachidananda007 opened 1 month ago

sachidananda007 commented 1 month ago

Druid is on 25.0.0 running on kubernetes.

I see one issue in auto-compaction for coordinator when it calculates the totalSlots - It doesn't take into account the middle-manager-tiers. We have 2 tiers.

for example:

  1. compact_tier(10 MMs with 4 slots each)
  2. default_tier(50 MMs with 10 slots each) So It ignores tiers and shows maxCompactionslots to 54 which is [0.1*(40+500)]

In this log it shows wrong "Found [32] available task slots for compaction out of [54] max compaction task capacity"

sachidananda007 commented 1 month ago

slack thread for this issue https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1719990602054439

kfaraz commented 1 month ago

Possible duplicate of #15847 .