[Ray Core] A RayJob that is in a pending state should be cancelable.

[xiangp152]

Description

A RayJob that is in a pending state and is preparing the runtime env should be cancellable immediately.

Use case

If you have submitted a Ray job that requires a long time to prepare the runtime env and you realize too late that it will take a long time to execute, you may need to cancel the task and find a better way to run it. If the task can be cancelled immediately, you can re-run it right away with the new approach. This can save you time and prevent unnecessary delays in your project. Therefore, it is important to have a cancellation mechanism in place that can handle such situations efficiently and without causing any issues with the runtime env.

[SongGuyang]

@architkulkarni @Catch-Bull any thought of this?

[architkulkarni]

Duplicate of https://github.com/ray-project/ray/issues/28221

[architkulkarni]

I think this enhancement makes sense and there's nothing controversial about the behavior. There are two parts, some of which might already be done:

• (Core) when an actor is killed, the runtime env installation should be interrupted and canceled.
• (Job) when ray job stop is called, the supervisor actor should be killed even if it the runtime env is still being set up.

We likely won't be able to prioritize adding this, but would welcome any PRs. cc @akshay-anyscale and cc @jjyao (since we discussed some related things before)