Open prjemian opened 2 months ago
Thanks @strempfer for showing this example!
Was able to reproduce locally by trying to open an older PersistentDict directory.
ls
beamline_id#22 conda_prefix#24 databroker_catalog#23 iconfig#21 instrument_name#27 login_id#26 pid#28 proposal_id#29 scan_id#25 versions#30
ipython
In [1]: from bluesky.utils import PersistentDict
In [2]: %xmode Verbose
Exception reporting mode: Verbose
In [3]: pd = PersistentDict(".")
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
Cell In[3], line 1
----> 1 pd = PersistentDict(".")
PersistentDict = <class 'bluesky.utils.PersistentDict'>
File ~/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/bluesky/utils/__init__.py:775, in PersistentDict.__init__(self=<PersistentDict {}>, directory='.')
773 self._func = zict.Func(self._dump, self._load, self._file)
774 self._cache = {}
--> 775 self.reload()
self = <PersistentDict {}>
777 # Similar to flush() or _do_update(), but without reference to self
778 # to avoid circular reference preventing collection.
779 # NOTE: This still doesn't guarantee call on delete or gc.collect()!
780 # Explicitly call flush() if immediate write to disk required.
781 def finalize(zfile, cache, dump):
File ~/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/bluesky/utils/__init__.py:841, in PersistentDict.reload(self=<PersistentDict {}>)
839 def reload(self):
840 """Force a reload from disk, overwriting current cache"""
--> 841 self._cache = dict(self._func.items())
self._cache = {}
self = <PersistentDict {}>
self._func = <Func: _dump<->_load <File: ., mode="a", 10 elements>>
File ~/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/zict/func.py:74, in <genexpr>(.0=<generator object ItemsView.__iter__>)
73 def items(self) -> Iterator[tuple[KT, VT]]: # type: ignore
---> 74 return ((k, self.load(v)) for k, v in self.d.items())
self.load = <function PersistentDict._load at 0x7f21045d8ae0>
self = <Func: _dump<->_load <File: ., mode="a", 10 elements>>
self.d = <File: ., mode="a", 10 elements>
File <frozen _collections_abc>:861, in __iter__(self=ItemsView(<File: ., mode="a", 10 elements>))
File ~/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/zict/file.py:82, in File.__getitem__(self=<File: ., mode="a", 10 elements>, key='versions#30')
80 raise KeyError(key)
81 fn = os.path.join(self.directory, _safe_key(key))
---> 82 with open(fn, "rb") as fh:
fn = './versions%2330'
83 if self.memmap:
84 return memoryview(mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ))
FileNotFoundError: [Errno 2] No such file or directory: './versions%2330'
The exception is correct, no such file ./versions%2330
exists. The existing file is named ./versions#30
.
A workaround for the situation is described.
The problem occurs for any file ending in #n
(where n is an integer) in the persistent directory. The solution is to clean up the file names in the directory before opening with PersistentDict(directory)
. Clean-up involves stripping the problematic suffix. If a non-suffixed file exists, just delete the suffixed version. Should make log entries for any of these actions.
Failures have been observed from this code block: https://github.com/BCDA-APS/bluesky_training/blob/d2b5f4cfa3bf356c20131f61bf91335430647592/bluesky/instrument/framework/initialize.py#L50-L61
When observed, the directory exists. Is there a mismatch between what is found and what is expected? In all observed cases so far, the problem is resolved by removing the directory defined by https://github.com/BCDA-APS/bluesky_training/blob/d2b5f4cfa3bf356c20131f61bf91335430647592/bluesky/instrument/iconfig.yml#L46
Needs more documentation, such as the exact exception traceback.