dstl / Stone-Soup

A software project to provide the target tracking community with a framework for the development and testing of tracking algorithms.
https://stonesoup.rtfd.io
MIT License
406 stars 134 forks source link

Possible issue with lru cache, _pickle.PicklingError: #748

Closed apiszcz closed 1 year ago

apiszcz commented 1 year ago

This must be my setup, The first time I'm seeing lru_cache issues, I use multiprocessing for a lot of different projects, this is a first time error.

Can't pickle <functools._lru_cache_wrapper object at 0xxxxx: it's not the same object as stonesoup.measures.SquaredMahalanobis._inv_cov

==

  File "C:\ss\lib\python\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\ss\lib\python\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\ss\lib\python\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\ss\lib\python\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\ss\lib\python\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <functools._lru_cache_wrapper object at 0x000001D130788040>: it's not the same object as stonesoup.measures.SquaredMahalanobis._inv_cov
apiszcz commented 1 year ago

Setting the value at line 162 of measures.py to 0 eliminates the error . Perhaps the lru cache approach does not support multiprocessing?

    state_covar_inv_cache_size: int = Property(
        **default=0,**
        doc="Number of covariance matrix inversions to cache. Setting to `0` will disable the "
            "cache, whilst setting to `None` will not limit the size of the cache. Default is "
            "128.")
sdhiscocks commented 1 year ago

I suspect the issue may be caused by the fact that the static method on the instance is replaced by a the lru_cache version on initialisation if state_covar_inv_cache_size is not 0, which some how breaks pickle process as it's wrapping the a different object when unpickling?

Looks like this question and answer on stack overflow has similar issue, overriding the pickling process.

apiszcz commented 1 year ago

The default, 128 is what causes the stack trace. 0 is working, any suggestions or perhaps this parameter needs to be elevated to the tracking initiator parameter?

sdhiscocks commented 1 year ago

You should be able to manually set this to 0 when instantiating the SquaredMahalanobis or Mahalanobis measure for now.

apiszcz commented 1 year ago

Initial tests with Mahalanobis(state_covar_inv_cache_size = 0) are working. Do we need a note stating state_covar_inv_cache_size = 0 is required for multiprocessing?