Closed ddanielr closed 2 months ago
This ticket should be added to the elasticity project.
This could be specified by a property or it could be something that a compaction planner plugin could set by calling SPI methods.
To keep compactors busy a max queue size that is greater than the number of compactors would be needed. Something to consider w/ this property. The ideal max queue is probably a multiple of the number of compactors in the group. For example if there are 100 compactors in a group then may want the max queue size to be 4x100 to ensure work is buffered when they ask for it.
Closed by #4767
Each CompactionJobPriorityQueue needs to have a independently configurable queue maxSize for resource management.
https://github.com/apache/accumulo/blob/09ceaeddbfbc02f960fa9e5b1fdbfa79be345fe1/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobQueues.java#L104-L106
Multiple queues can belong to a given Compaction service. However, each queue is only defined by the executorId (queue name) as the identifying reference and not a corresponding service.
Because of this, #3547 will probably need to be resolved first before this ticket can be worked.
This issue is required to test rejected job metrics in #3551. To avoid being blocked I've gone ahead and created a temporary property called
MANAGER_COMPACTION_SERVICE_PRIORITY_QUEUE_SIZE
which is included in #3551https://github.com/apache/accumulo/pull/3551/files#diff-6c2d215eebb1a6157a2b7d95f49496834bd91a150c8aedd97cca6b01aedc05cbR377-R380
This property is used for all priorityQueues created by the Manager. So it does not fulfill the scope of this issue, just allows for the metric queue generation work to be closed out.
I've linked this ticket in the code and marked the property as
Experimental