ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
34.03k stars 5.78k forks source link

[aDAG][Core] Weird exception if ref is deleted after adag teardown #47687

Open rkooo567 opened 2 months ago

rkooo567 commented 2 months ago

What happened + What you expected to happen

We call ray.get on ref del, but if the dag is already teardown, it raises an weird exception

Versions / Dependencies

master

Reproduction script

import ray
from ray.dag import InputNode, MultiOutputNode

ray.init()

@ray.remote
class A:
    def f(self, inp):
        return 1

a = A.remote()
b = A.remote()
with InputNode() as inp:
    x = b.f.bind(inp)
    y = a.f.bind(inp)
    dag = MultiOutputNode([x,y ])
adag = dag.experimental_compile()
refs = adag.execute(1)
adag.teardown()

Issue Severity

None

ruisearch42 commented 2 months ago

Can you post the stack trace of the exception?

AruparnaMaity commented 1 month ago

Is this issue resolved? Or is someone working on this? @rkooo567 @stephanie-wang

rkooo567 commented 3 weeks ago

@AruparnaMaity feel free to take it!