apache / accumulo

Apache Accumulo
https://accumulo.apache.org
Apache License 2.0
1.07k stars 445 forks source link

Add configurable Property to set a CompactionJobPriorityQueue's maxSize #3635

Closed ddanielr closed 2 months ago

ddanielr commented 1 year ago

Each CompactionJobPriorityQueue needs to have a independently configurable queue maxSize for resource management.

https://github.com/apache/accumulo/blob/09ceaeddbfbc02f960fa9e5b1fdbfa79be345fe1/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobQueues.java#L104-L106

Multiple queues can belong to a given Compaction service. However, each queue is only defined by the executorId (queue name) as the identifying reference and not a corresponding service.

Because of this, #3547 will probably need to be resolved first before this ticket can be worked.

This issue is required to test rejected job metrics in #3551. To avoid being blocked I've gone ahead and created a temporary property called MANAGER_COMPACTION_SERVICE_PRIORITY_QUEUE_SIZE which is included in #3551

https://github.com/apache/accumulo/pull/3551/files#diff-6c2d215eebb1a6157a2b7d95f49496834bd91a150c8aedd97cca6b01aedc05cbR377-R380

This property is used for all priorityQueues created by the Manager. So it does not fulfill the scope of this issue, just allows for the metric queue generation work to be closed out.

I've linked this ticket in the code and marked the property as Experimental

ddanielr commented 1 year ago

This ticket should be added to the elasticity project.

keith-turner commented 1 year ago

This could be specified by a property or it could be something that a compaction planner plugin could set by calling SPI methods.

keith-turner commented 3 months ago

To keep compactors busy a max queue size that is greater than the number of compactors would be needed. Something to consider w/ this property. The ideal max queue is probably a multiple of the number of compactors in the group. For example if there are 100 compactors in a group then may want the max queue size to be 4x100 to ensure work is buffered when they ask for it.

dlmarion commented 2 months ago

Closed by #4767