search

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.63k stars 5.71k forks source link

[core][dags] Ray non-compiled DAGs do not respect bind order on actors #47159

Open stephanie-wang opened 2 months ago

stephanie-wang commented 2 months ago

What happened + What you expected to happen

There is no way to define control dependencies between tasks that execute on the same actor in Ray DAGs. This is inconsistent with normal Ray actor tasks, which execute in the order that they are submitted in, and the experimental compiled Ray DAGs, which use the bind order on an actor to define the order of execution.

Versions / Dependencies

3.0dev

Reproduction script

a = Actor.remote()

with InputNode() as inp:
  x = a.foo.bind(inp)
  y = a.bar.bind(inp)
  dag = MultiOutputNode([x, y])

foo and bar can execute in any order. Ideally we should execute them in the order that they are bound in.

Issue Severity

None

stephanie-wang commented 2 months ago

Not really an accelerated DAG issue because it only affects non-compiled DAGs, but it is important for aDAG debuggability that both execution modes produce the same execution order.

cadedillon commented 1 week ago

Hi @stephanie-wang I'd like to work on this issue, thank you!

stephanie-wang commented 1 week ago

Thanks @cadedillon !

clr-li commented 3 days ago

Hi @stephanie-wang I'm interested in working on this, thanks!