[Ingest Pipelines] Load-Balancing on Coordinating Nodes

Description

Issue: Coordinating nodes with an ingest role directly process ingest pipelines, leading to potential CPU pressure imbalances when bulk requests are unevenly distributed across nodes.

Details: In scenarios where coordinating nodes are also ingest nodes, an uneven distribution of bulk requests can result in skewed processing loads for ingest pipelines. This skew can cause significant CPU pressure on certain nodes, leading to potential backlogs and degraded performance.

Code Reference: Relevant code:

TransportBulkAction.java for bulk request routing logic.

                if (clusterService.localNode().isIngestNode()) {
                    processBulkIndexIngestRequest(task, bulkRequest, executor, metadata, l);
                } else {
                    ingestForwarder.forwardIngestRequest(bulkAction, bulkRequest, l);
                }

Impact: In high-throughput environments, this imbalance can lead to node saturation, impacting both ingest throughput and search performance due to resource contention.

Query: Is there an existing strategy for more evenly distributing ingest workload across nodes? Could we consider a feature similar to Adaptive Replica Selection for ingest operations?

Contribution: I am open to contributing to the discussion and development of this feature to enhance Elasticsearch's resilience and efficiency.

Thank you for your attention to this matter.

elastic / elasticsearch

[Ingest Pipelines] Load-Balancing on Coordinating Nodes #108846

Description