ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.71k stars 5.73k forks source link

Core Multiprocessing - crashes cloudpickling`lru_cache` #31099

Open garrettwrong opened 1 year ago

garrettwrong commented 1 year ago

What happened + What you expected to happen

I recently added lru_cache to some simple grid generating methods in a Python class. This class is later used inside a ray multiprocessing Pool.

Expected behavior: Not crash. Ideally we could ship the lru_cache. I would also be fine with clearing it and losing any optimization when distributed, but it can't crash. This will block us from doing some pretty common optimizations without reimplementing standard library tools.

I believe this is https://github.com/cloudpipe/cloudpickle/issues/178

Thanks

Versions / Dependencies

ray==2.2.0 Python 3.10.6 Linux

But I this is not version dependent...

Reproduction script

performed on osx. 11.6.2

conda create -n rpb python=3.8 pip install ray

from functools import lru_cache
import ray
from ray.util.multiprocessing import Pool

class Frog:
    def __init__(self):
        # We want a local cache per instance
        self.f = lru_cache(maxsize=8)(self.f)

    @staticmethod
    @lru_cache(maxsize=8)
    def g(n):
        print(f"expensive stuff {n}")
        return n

    def f(self, n):
        print(f"more expensive stuff {n} composed with expensive stuff g({n}).")
        return n * Frog.g(n)

    def compute_all_the_things(self):
        print(list(map(self.f, range(3))))

    def compute_all_the_things_par(self):
        ray.init()
        with Pool() as p:
            res = p.map(self.f, range(3))
        ray.shutdown()
        print(list(res))

a = Frog()
a.compute_all_the_things()
a.compute_all_the_things_par()

Issue Severity

Medium: It is a significant difficulty but I can work around it.

garrettwrong commented 2 months ago

Hi, is this fixed? I don't see an associated PR or updates on the referenced cloudpickle issue. Thanks!

jjyao commented 2 months ago

We are blocked by cloudpickle fix.