search

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.7k stars 5.73k forks source link

Class attribute in ray actor cannot be accessed/copied #11959

Open jiangtianli91 opened 3 years ago

jiangtianli91 commented 3 years ago

What is the problem?

Ray actor fails to access class attributes.

Ray version and other system information (Python version, TensorFlow version, OS): ray 1.0.0 Python 3.7.7

Reproduction (REQUIRED)

This code is modified from https://github.com/dask/distributed/issues/4233#issue-740152650

from module import Foo
import ray

@ray.remote
class ray_Foo:
    def __init__(self, Foo):
        self.Foo = Foo

    def show(self):
        return self.Foo.bar

ray.init()
foo = Foo()
foo.append(123)
print(foo.show())

ray_foo = ray_Foo.remote(foo)
print(ray.get(ray_foo.show.remote()))
$ cat module.py
class Foo:
    bar = []

    def append(self, value):
        self.bar.append(value)

    def show(self):
        return self.bar

$ python ray_issue_new.py
[123]
[]

If we cannot run your script, we cannot fix your issue.

rkooo567 commented 3 years ago

This is an interesting issue. It seems like it only happens when we send serialized objects to other processes.

from module import Foo
import ray

@ray.remote
class ray_Foo:
    def __init__(self, Foo):
        self.Foo = Foo

    def show(self):
        return self.Foo.bar

@ray.remote
def f(foo):
    return foo.bar

ray.init()
foo = Foo()
foo.append(123)
print(foo.show())
p = ray.put(foo)
print(ray.get(p).bar)
ray_foo = ray_Foo.remote(foo)
print(ray.get(f.remote(p)))
print(ray.get(ray_foo.show.remote()))

As you can see 

p = ray.put(foo)
print(ray.get(p).bar)

works, but it doesn't work only when objects are sent to other processes.

rkooo567 commented 3 years ago

cc @robertnishihara Have you seen similar issues before?

rkooo567 commented 3 years ago

Also @richardliaw

wuisawesome commented 3 years ago

Since this seems to be a pickling issue, I wonder if we should first try to reproduce the issue with just cloudpickle and forward this to their team.

rkooo567 commented 3 years ago

@wuisawesome That sounds like a good idea. @jiangtianli91 Does this issue block you, or is it something that you can get around? (a.k.a, is it possible to not use class attributes? I assume this can be a blocker if this happens from other libraries that you depend on)

jiangtianli91 commented 3 years ago

@wuisawesome That sounds like a good idea. @jiangtianli91 Does this issue block you, or is it something that you can get around? (a.k.a, is it possible to not use class attributes? I assume this can be a blocker if this happens from other libraries that you depend on)

So far we have hit this issue because our own libraries used class attributes, so we could move those although it does disrupt the organization a bit.

rkooo567 commented 3 years ago

Gotcha. Btw, @suquark Have you seen any similar issue before?

jenny-hm-lee commented 3 years ago

I have this issue.

jenny-hm-lee commented 3 years ago

Have this issues been resolved in latest version of Ray?

jiangtianli91 commented 3 years ago

@rkooo567 @wuisawesome Hello, it seems that cloudpickle has merged a pull request (here) that can solve this issue. I wonder whether it can be implemented here?

mbwmbw1337 commented 2 years ago

We're seeing the same issue. What is the best workaround or have you been converting all class attributes to instances?

mbwmbw1337 commented 2 years ago

Have this issues been resolved in latest version of Ray?

Not resolved as far as I am able to tell.

jiangtianli91 commented 2 years ago

@rkooo567 Hello, I wonder is there any plan to fix this bug?

rkooo567 commented 2 years ago

Hmm, if this is correct https://github.com/ray-project/ray/issues/11959#issuecomment-894888593, using the latest cloudpickle should fix the issue? Have we merged the upstream cloudpickle? cc @suquark

filip-halt commented 2 years ago

I am also having this issue, sending an object to an actor causes it to lose its attributes.

clarkzinzow commented 2 years ago

@suquark Do you know if that upstream fix was merged?

qmihau commented 2 years ago

Hi, I'm having that issue in ray 1.13.0 with cloudpickle 2.1.0. Is there any chance it will be resolved fairly soon?

rkooo567 commented 2 years ago

Let us revisit this soon. Will bump up the priority.

qmihau commented 2 years ago

Thanks! Happy to help test or if you'll need any extra information.

qmihau commented 2 years ago

@rkooo567 Do you have a feel for when that might get into the work pipeline? Definitely don't want to be pushy, but just trying to plan our work accordingly :)

rkooo567 commented 1 year ago

I think it is hard to guarantee the specific timeline for the fix. We are planning to take a look, but there are plenty of other issues, and this is currently not the highest priority.

thomasbbrunner commented 1 year ago

Can confirm that this is still an issue in Ray 2.6.3.