I recently added lru_cache to some simple grid generating methods in a Python class. This class is later used inside a ray multiprocessing Pool.
Expected behavior: Not crash. Ideally we could ship the lru_cache. I would also be fine with clearing it and losing any optimization when distributed, but it can't crash. This will block us from doing some pretty common optimizations without reimplementing standard library tools.
from functools import lru_cache
import ray
from ray.util.multiprocessing import Pool
class Frog:
def __init__(self):
# We want a local cache per instance
self.f = lru_cache(maxsize=8)(self.f)
@staticmethod
@lru_cache(maxsize=8)
def g(n):
print(f"expensive stuff {n}")
return n
def f(self, n):
print(f"more expensive stuff {n} composed with expensive stuff g({n}).")
return n * Frog.g(n)
def compute_all_the_things(self):
print(list(map(self.f, range(3))))
def compute_all_the_things_par(self):
ray.init()
with Pool() as p:
res = p.map(self.f, range(3))
ray.shutdown()
print(list(res))
a = Frog()
a.compute_all_the_things()
a.compute_all_the_things_par()
Issue Severity
Medium: It is a significant difficulty but I can work around it.
What happened + What you expected to happen
I recently added
lru_cache
to some simple grid generating methods in a Python class. This class is later used inside a ray multiprocessing Pool.Expected behavior: Not crash. Ideally we could ship the
lru_cache
. I would also be fine with clearing it and losing any optimization when distributed, but it can't crash. This will block us from doing some pretty common optimizations without reimplementing standard library tools.I believe this is https://github.com/cloudpipe/cloudpickle/issues/178
Thanks
Versions / Dependencies
ray==2.2.0 Python 3.10.6 Linux
But I this is not version dependent...
Reproduction script
performed on osx. 11.6.2
conda create -n rpb python=3.8 pip install ray
Issue Severity
Medium: It is a significant difficulty but I can work around it.