Open m-ghazanfar opened 7 months ago
My overlord dynamic config looks like this,
We worked around this by explicitly setting the maxCompactionTaskSlots
to the total available worker slots on the compaction tier.(Actually, we set it a little higher than that to improve middle manager utilisation)
See Update capacity for compaction tasks
Example,
curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config/compaction/taskslots?max=600"
Description
The getCompactionTaskCapacity function which is used to ascertain that the Druid cluster has enough task slots before the coordinator schedules additional compaction tasks doesn't take into consideration overlord dynamic config. The overlord dynamic config can prevent compaction tasks from running on specific categories of workers. This way, the compaction task capacity is incorrectly overestimated.
For example, I have two worker categories,
compaction-category
with a total of600
task slots and anotheringestion-category
with2000
slots(high number because of multiple ingestion task replicas).Using the overlord dynamic config,
compaction-category
is configured to run the following task types,ingestion-category
is configured to run,Now, getCompactionTaskCapacity would return
2600
as the total capacity, which is inaccurate since only600
slots are actually available for compaction tasks. While this might not pose a problem in a healthy cluster, it becomes critical during compaction task failures. The oversight leads to the coordinator creating excessive compaction tasks, resulting in contention on compaction slots and slowing down all compaction tasks. This creates a feedback loop where the increasing number of compaction tasks exacerbates contention, ultimately overwhelming the overlord with too many tasks to handleAffected Version
Saw this on Druid 25. Is also present in master.