uqfoundation / mystic

constrained nonlinear optimization for scientific machine learning, UQ, and AI
http://mystic.rtfd.io
Other
467 stars 50 forks source link

cache doesn't sync for in-memory archives across parallel map #210

Open mmckerns opened 1 year ago

mmckerns commented 1 year ago

A mystic.cache fails to cache when used in a parallel map if cached=True, however succeeds if cached=False unless it's a dict_archive. This indicates that the in-memory cache is passed to the child process, but the in-memory cached results are not linked back to the main process. Which, makes sense, but maybe isn't what is expected. Is it reasonable to expect the in-memory cache in the main process to be able to fetch the results from the in-memory caches of the child processes?

Current behavior:

Python 3.8.17 (default, Jun 11 2023, 01:54:00) 
[Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mystic.cache as mc
>>> import mystic.models as mm
>>> model = mc.cached(archive=mc.archive.dict_archive('foo'))(mm.rosen)
>>> model([1,2,3])
201.0
>>> model.__cache__()
dict_archive('foo', {(1, 2, 3): 201.0}, cached=True)
>>> import pathos as pa
>>> pmap = pa.maps.Map(pa.pools.ProcessPool)
>>> pmap(model, [[4,5,6],[7,8,9]])
[48225.0, 470685.0]
>>> model.__cache__()
dict_archive('foo', {(1, 2, 3): 201.0}, cached=True)
>>> 
>>> model = mc.cached(archive=mc.archive.dict_archive('bar', cached=False))(mm.rosen)
>>> model([1,2,3])
201.0
>>> model.__cache__()
dict_archive({(1, 2, 3): 201.0}, cached=False)
>>> pmap(model, [[4,5,6],[7,8,9]])
[48225.0, 470685.0]
>>> model.__cache__()
dict_archive({(1, 2, 3): 201.0}, cached=False)
>>> 
>>> model = mc.cached(archive=mc.archive.dir_archive('baz', cached=False))(mm.rosen)
>>> model([1,2,3])
201.0
>>> model.__cache__()
dir_archive('baz', {(1, 2, 3): 201.0}, cached=False)
>>> pmap(model, [[4,5,6],[7,8,9]])
[48225.0, 470685.0]
>>> model.__cache__()
dir_archive('baz', {(7, 8, 9): 470685.0, (1, 2, 3): 201.0, (4, 5, 6): 48225.0}, cached=False)
>>> 
>>> model = mc.cached(archive=mc.archive.dir_archive('zap'))(mm.rosen)
>>> model([1,2,3])
201.0
>>> model.__cache__()
dir_archive('zap', {(1, 2, 3): 201.0}, cached=True)
>>> pmap(model, [[4,5,6],[7,8,9]])
[48225.0, 470685.0]
>>> model.__cache__()
dir_archive('zap', {(1, 2, 3): 201.0}, cached=True)
>>> 
>>> mmap = pa.maps.Map()
>>> mmap(model, [[4,5,6],[7,8,9]])
[48225.0, 470685.0]
>>> model.__cache__()
dir_archive('zap', {(1, 2, 3): 201.0, (4, 5, 6): 48225.0, (7, 8, 9): 470685.0}, cached=True)
>>> 
>>> model = mc.cached(archive=mc.archive.dict_archive('bar', cached=False))(mm.rosen)
>>> model([1,2,3])
201.0
>>> model.__cache__()
dict_archive({(1, 2, 3): 201.0}, cached=False)
>>> mmap(model, [[4,5,6],[7,8,9]])
[48225.0, 470685.0]
>>> model.__cache__()
dict_archive({(1, 2, 3): 201.0, (4, 5, 6): 48225.0, (7, 8, 9): 470685.0}, cached=False)
>>>