Numba ReferenceError: underlying object has vanished

[voloddia]

I get this error whenever I run claspy segmentation in Jupyter notebooks after restarting kernel. If I run it again without changing anything, it runs fine. Do you have any idea what might cause this?

Numba version 0.56.4 Reproducible code

np.random.seed(1379)
dataset = "Noise"
cps = np.array([400, 1600, 1000])
time_series = np.concatenate((
    np.random.rand(cps[0]),
    1 + np.random.rand(cps[1]),
    -1 + np.random.rand(cps[2])
))
cps = np.cumsum(cps)[:-1]

clasp = BinaryClaSPSegmentation().fit(time_series)
clasp.plot(heading=dataset, gt_cps=cps)
plt.show()

Error

KeyError                                  Traceback (most recent call last)
c:\ProgramData\Anaconda3\envs\base\lib\site-packages\numba\core\caching.py in save(self, key, data)
    468             # If key already exists, we will overwrite the file
--> 469             data_name = overloads[key]
    470         except KeyError:

KeyError: ((array(float64, 1d, C), int64, int64, ListType[UniTuple(int64 x 2)], ListType[array(int64, 1d, C)], type(CPUDispatcher(<function znormed_euclidean_distance at 0x000001F5E79D3D30>)), type(CPUDispatcher(<function sliding_mean_std at 0x000001F5E79D38B0>))), ('x86_64-pc-windows-msvc', 'tigerlake', '+64bit,+adx,+aes,-amx-bf16,-amx-int8,-amx-tile,+avx,+avx2,-avx512bf16,+avx512bitalg,+avx512bw,+avx512cd,+avx512dq,-avx512er,+avx512f,+avx512ifma,-avx512pf,+avx512vbmi,+avx512vbmi2,+avx512vl,+avx512vnni,+avx512vp2intersect,+avx512vpopcntdq,+bmi,+bmi2,-cldemote,+clflushopt,+clwb,-clzero,+cmov,+cx16,+cx8,-enqcmd,+f16c,+fma,-fma4,+fsgsbase,+fxsr,+gfni,+invpcid,-lwp,+lzcnt,+mmx,+movbe,+movdir64b,+movdiri,-mwaitx,+pclmul,-pconfig,-pku,+popcnt,-prefetchwt1,+prfchw,-ptwrite,+rdpid,+rdrnd,+rdseed,-rtm,+sahf,-serialize,-sgx,+sha,+shstk,+sse,+sse2,+sse3,+sse4.1,+sse4.2,-sse4a,+ssse3,-tbm,-tsxldtrk,+vaes,+vpclmulqdq,-waitpkg,-wbnoinvd,-xop,+xsave,+xsavec,+xsaveopt,+xsaves'), ('19597f6ba93d4431633b3bf3a936017a49578d550391ab41686d4ffc7c35fbb2', 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'))

During handling of the above exception, another exception occurred:

ReferenceError                            Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_10548/1821199485.py in <module>
      9 cps = np.cumsum(cps)[:-1]
     10 
---> 11 clasp = BinaryClaSPSegmentation().fit(time_series)
     12 clasp.plot(heading=dataset, gt_cps=cps)
     13 plt.show()

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\claspy\segmentation.py in fit(self, time_series)
    220         if self.n_segments > 1:
    221             prange = 0, time_series.shape[0]
--> 222             clasp = ClaSPEnsemble(
    223                 n_estimators=self.n_estimators,
    224                 window_size=self.window_size,

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\claspy\clasp.py in fit(self, time_series, knn, validation, threshold)
    400 
    401         if knn is None:
--> 402             knn = KSubsequenceNeighbours(
    403                 window_size=self.window_size,
    404                 k_neighbours=self.k_neighbours,

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\claspy\nearest_neighbour.py in fit(self, time_series, temporal_constraints)
    382             if end > start: pranges.append((start, end))
    383 
--> 384         self.distances, self.offsets = _parallel_knn(time_series, self.window_size, self.k_neighbours,
    385                                                      pranges, List(self.temporal_constraints),
    386                                                      self.distance, self.distance_preprocessing)

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\numba\core\dispatcher.py in _compile_for_args(self, *args, **kws)
    485                     e.patch_message('\n'.join((str(e).rstrip(), help_msg)))
    486             # ignore the FULL_TRACEBACKS config, this needs reporting!
--> 487             raise e
    488         finally:
    489             self._types_active_call = []

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\numba\core\dispatcher.py in _compile_for_args(self, *args, **kws)
    418         return_val = None
    419         try:
--> 420             return_val = self.compile(tuple(argtypes))
    421         except errors.ForceLiteralArg as e:
    422             # Received request for compiler re-entry with the list of arguments

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\numba\core\dispatcher.py in compile(self, sig)
    970                         raise e.bind_fold_arguments(folded)
    971                     self.add_overload(cres)
--> 972                 self._cache.save_overload(sig, cres)
    973                 return cres.entry_point
    974 

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\numba\core\caching.py in save_overload(self, sig, data)
    650         """
    651         with self._guard_against_spurious_io_errors():
--> 652             self._save_overload(sig, data)
    653 
    654     def _save_overload(self, sig, data):

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\numba\core\caching.py in _save_overload(self, sig, data)
    660         key = self._index_key(sig, data.codegen)
    661         data = self._impl.reduce(data)
--> 662         self._cache_file.save(key, data)
    663 
    664     @contextlib.contextmanager

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\numba\core\caching.py in save(self, key, data)
    476                     break
    477             overloads[key] = data_name
--> 478             self._save_index(overloads)
    479         self._save_data(data_name, data)
    480 

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\numba\core\caching.py in _save_index(self, overloads)
    520     def _save_index(self, overloads):
    521         data = self._source_stamp, overloads
--> 522         data = self._dump(data)
    523         with self._open_for_write(self._index_path) as f:
    524             pickle.dump(self._version, f, protocol=-1)

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\numba\core\caching.py in _dump(self, obj)
    548 
    549     def _dump(self, obj):
--> 550         return dumps(obj)
    551 
    552     @contextlib.contextmanager

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\numba\core\serialize.py in dumps(obj)
     55     with io.BytesIO() as buf:
     56         p = pickler(buf, protocol=4)
---> 57         p.dump(obj)
     58         pickled = buf.getvalue()
     59 

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\numba\cloudpickle\cloudpickle_fast.py in dump(self, obj)
    566     def dump(self, obj):
    567         try:
--> 568             return Pickler.dump(self, obj)
    569         except RuntimeError as e:
    570             if "recursion" in e.args[0]:

c:\ProgramData\Anaconda3\envs\base\lib\site-packages\numba\core\types\functions.py in __getnewargs__(self)
    484         obj = self()
    485         if obj is None:
--> 486             raise ReferenceError("underlying object has vanished")
    487         return (obj,)
    488 

ReferenceError: underlying object has vanished

[ermshaua]

Hi @voloddia,

thanks for mentioning this. I'll have a look.

Best, Arik

[ermshaua]

Hi @voloddia,

actually, I cannot reproduce your error. I ran

conda create -n claspytest python=3.9 conda activate claspytest pip install claspy pip install notebook pip install numba==0.56.4 jupyter notebook

and then created a notebook with your short example, restarting and running the kernel in all kinds of ways. But it worked, regardless. I also tried other numba versions, which all work fine.

However, maybe I can help you nevertheless: The ReferenceError you posted comes up, because we cache the results of the nearest neighbour computation with numba. I've had this error a few times myself in random scenarios. It sometimes comes up when numba confuses something in the caching (I guess). More importantly, I always fixed it by cleaning the cache. So, please go to your claspy installation you find with

pip show claspy

Then head to the pycache folder and delete it. This hopefully should solve your problem. Please let me know!

Best regards, Arik

[voloddia]

Thank you. After restarting the computer, I also cannot reproduce the error. That object vanished error still happened seemingly randomly in other places a few times. Clearing pycache of claspy worked in those times, so I'll close this.