search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.89k stars 1.84k forks source link

[Feature Request] Background Tasks #12727

Open linuxpi opened 8 months ago

linuxpi commented 8 months ago

Is your feature request related to a problem? Please describe

With Offline fleet, data nodes and Offline Fleet nodes itself can submit background tasks to Offline Fleet. At any point, the no of Tasks submitted might be too much for available nodes in the Offline Fleet to distribute amongst themselves.

Even if we do try to assign a task to a particular node right after its submitted, the node may or may not have resources at that time to start the task and would need to put it into a “Queue”. Apart from that, if the node goes down, and this Queue is not persisted in Remote, all those tasks in Queue are lost.

Describe the solution you'd like

To support various background tasks to be run on dedicated/separate fleet of nodes, we need to explore and propose ideas around how to model tasks and submit for execution of the dedicated fleet

RFC - https://github.com/opensearch-project/OpenSearch/issues/12361

Sub items

Related component

Storage

Describe alternatives you've considered

No response

Additional context

No response

linuxpi commented 7 months ago

Design Details outlined in https://github.com/opensearch-project/OpenSearch/issues/13554

getsaurabh02 commented 3 months ago

@linuxpi do we need to move this to 2.17?