quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
https://quickwit.io
Other
6.99k stars 291 forks source link

Improving placement #4731

Open fulmicoton opened 3 months ago

fulmicoton commented 3 months ago

Given one index, we attempt to spread the load on as few indexer as possible to have better split pruning.

The current logic has some pitfalls:

These are not critical (we are always below the load we target), but they seem to be confusing users.

dojiong commented 3 months ago

What about one pipeline per indexer, and split the indexing pipeline into small tasks to increase the parallelism.

Then the desired_num_pipelines could be used to control the split pruning.

And the indexer could make pipelines from different indexes more balanced on one node(based on things like task units for indexing-pipeline)