Core: Can the ray core's scheduling mechanism support customized extensions?

[EnjianGong]

Description

Most of our company's business is some AP offline tasks, so the resource utilization rate of the entire ray cluster is very low, and there are a lot of idle resource fragments in time and space. Therefore, I look forward to using DQN to learn the resource wave model to predict resource occupancy for a period of time in the future, and then share these idle resources to reduce costs. Therefore, we want to customize Ray core's scheduling mechanism to adapt to workload fluctuations

Use case

Workload scenario: There are a lot of idle resources in space and time, and the resource occupation has a strong periodicity

[rkooo567]

cc @jjyao

[EnjianGong]

Is there a solution to this issue @clarng

[rkooo567]

Currently this is not possible

[clarng]

Do you have more details of the proposal? Depending on what it is perhaps it is possible to modularize the code base

[jjyao]

@EnjianGong I'd like to know more about the scheduling mechanism you want. Could you elaborate.

[EnjianGong]

Currently, there is only one simple idea: Train a resource prediction model to be used in the scheduler's inference process.

• Observation: CPU/MEM/IO load status
• Reward: resource utilization rate, task satisfaction rate

There are several challenges:

1. How to coordinate the rule-based scheduler and the model-based scheduler, and determine their priority relationship.
2. Over time, the optimal scheduling will experience distribution drift.

[jjyao]

So Ray scheduler is basically a function that maps from input task + cluster status to the output node. You are saying training a ML model to be that function?

[EnjianGong]

Yes, use this function to obtain a resource prediction