Open fulmicoton opened 3 months ago
What about one pipeline per indexer, and split the indexing pipeline into small tasks to increase the parallelism.
Then the desired_num_pipelines
could be used to control the split pruning.
And the indexer could make pipelines from different indexes more balanced on one node(based on things like task units for indexing-pipeline)
Given one index, we attempt to spread the load on as few indexer as possible to have better split pruning.
The current logic has some pitfalls:
These are not critical (we are always below the load we target), but they seem to be confusing users.