Core Multiprocessing - crashes cloudpickling`lru_cache`

[garrettwrong]

What happened + What you expected to happen

I recently added lru_cache to some simple grid generating methods in a Python class. This class is later used inside a ray multiprocessing Pool.

Expected behavior: Not crash. Ideally we could ship the lru_cache. I would also be fine with clearing it and losing any optimization when distributed, but it can't crash. This will block us from doing some pretty common optimizations without reimplementing standard library tools.

I believe this is https://github.com/cloudpipe/cloudpickle/issues/178

Thanks

Versions / Dependencies

ray==2.2.0 Python 3.10.6 Linux

But I this is not version dependent...

Reproduction script

performed on osx. 11.6.2

conda create -n rpb python=3.8 pip install ray

from functools import lru_cache
import ray
from ray.util.multiprocessing import Pool

class Frog:
    def __init__(self):
        # We want a local cache per instance
        self.f = lru_cache(maxsize=8)(self.f)

    @staticmethod
    @lru_cache(maxsize=8)
    def g(n):
        print(f"expensive stuff {n}")
        return n

    def f(self, n):
        print(f"more expensive stuff {n} composed with expensive stuff g({n}).")
        return n * Frog.g(n)

    def compute_all_the_things(self):
        print(list(map(self.f, range(3))))

    def compute_all_the_things_par(self):
        ray.init()
        with Pool() as p:
            res = p.map(self.f, range(3))
        ray.shutdown()
        print(list(res))

a = Frog()
a.compute_all_the_things()
a.compute_all_the_things_par()

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[garrettwrong]

Hi, is this fixed? I don't see an associated PR or updates on the referenced cloudpickle issue. Thanks!

[jjyao]

We are blocked by cloudpickle fix.