TypeError: can't pickle weakref objects

[sxia-cb]

I ran UMAP with umap_model = umap.UMAP(n_neighbors=15,n_components=5, low_memory=False, random_state=42).fit(df) and it works, but when run joblib.dump(umap_model, 'test.sav') or umap_model.transform(df1) get errors: TypeError: can't pickle weakref objects

[sxia-cb]

I did some tests and found it is wield: umap_model = umap.UMAP(n_neighbors=15,n_components=5, low_memory=False, random_state=42).fit(df) umap_model.transform(x) I got the error: TypeError: can't pickle weakref objects but continue to do: df1 = df.reset_index(drop = True) or save df to file, then load umap_model = umap.UMAP(n_neighbors=15,n_components=5, low_memory=False, random_state=42).fit(df1) umap_model.transform(x) it works and then ran df again: umap_model = umap.UMAP(n_neighbors=15,n_components=5, low_memory=False, random_state=42).fit(df) umap_model.transform(x) it also works

And It got error when run first time umap_model = umap.UMAP(n_neighbors=15,n_components=5, low_memory=False, random_state=42).fit(df) umap_model.transform(x) but as long as it works, it will not get this error.

Any idea to solve this issue ?

[adilosa]

Hit this error also, it seems to be in PyNNDescent. I'm on the latest commit dd415c

Stacktrace (click to expand full):

```python --------------------------------------------------------------------------- KeyError Traceback (most recent call last) lib/python3.7/site-packages/numba/core/caching.py in save(self, key, data) 486 # If key already exists, we will overwrite the file --> 487 data_name = overloads[key] 488 except KeyError: KeyError: ((array(int32, 1d, C), array(int32, 1d, C), array(float32, 1d, C), array(float32, 2d, C), type(CPUDispatcher()), array(int64, 1d, C), float64), ('x86_64-apple-darwin20.5.0', 'skylake', '+64bit,+adx,+aes,+avx,+avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512er,-avx512f,-avx512ifma,-avx512pf,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vpopcntdq,+bmi,+bmi2,-cldemote,+clflushopt,-clwb,-clzero,+cmov,+cx16,+cx8,-enqcmd,+f16c,+fma,-fma4,+fsgsbase,+fxsr,-gfni,+invpcid,-lwp,+lzcnt,+mmx,+movbe,-movdir64b,-movdiri,-mwaitx,+pclmul,-pconfig,-pku,+popcnt,-prefetchwt1,+prfchw,-ptwrite,-rdpid,+rdrnd,+rdseed,-rtm,+sahf,+sgx,-sha,-shstk,+sse,+sse2,+sse3,+sse4.1,+sse4.2,-sse4a,+ssse3,-tbm,-vaes,-vpclmulqdq,-waitpkg,-wbnoinvd,-xop,+xsave,+xsavec,+xsaveopt,+xsaves'), ('308c49885ad3c35a475c360e21af1359caa88c78eb495fa0f5e8c6676ae5019e', 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855')) During handling of the above exception, another exception occurred: ```

```python --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in ----> 1 nnquery = nnindex.query(data, k=70) lib/python3.7/site-packages/pynndescent/pynndescent_.py in query(self, query_data, k, epsilon) 1564 """ 1565 if not hasattr(self, "_search_graph"): -> 1566 self._init_search_graph() 1567 1568 if not self._is_sparse: lib/python3.7/site-packages/pynndescent/pynndescent_.py in _init_search_graph(self) 1061 self._distance_func, 1062 self.rng_state, -> 1063 self.diversify_prob, 1064 ) 1065 reverse_graph.eliminate_zeros() lib/python3.7/site-packages/numba/core/dispatcher.py in _compile_for_args(self, *args, **kws) 432 e.patch_message('\n'.join((str(e).rstrip(), help_msg))) 433 # ignore the FULL_TRACEBACKS config, this needs reporting! --> 434 raise e 435 436 def inspect_llvm(self, signature=None): lib/python3.7/site-packages/numba/core/dispatcher.py in _compile_for_args(self, *args, **kws) 365 argtypes.append(self.typeof_pyval(a)) 366 try: --> 367 return self.compile(tuple(argtypes)) 368 except errors.ForceLiteralArg as e: 369 # Received request for compiler re-entry with the list of arguments lib/python3.7/site-packages/numba/core/compiler_lock.py in _acquire_compile_lock(*args, **kwargs) 30 def _acquire_compile_lock(*args, **kwargs): 31 with self: ---> 32 return func(*args, **kwargs) 33 return _acquire_compile_lock 34 lib/python3.7/site-packages/numba/core/dispatcher.py in compile(self, sig) 823 raise e.bind_fold_arguments(folded) 824 self.add_overload(cres) --> 825 self._cache.save_overload(sig, cres) 826 return cres.entry_point 827 lib/python3.7/site-packages/numba/core/caching.py in save_overload(self, sig, data) 669 """ 670 with self._guard_against_spurious_io_errors(): --> 671 self._save_overload(sig, data) 672 673 def _save_overload(self, sig, data): lib/python3.7/site-packages/numba/core/caching.py in _save_overload(self, sig, data) 679 key = self._index_key(sig, _get_codegen(data)) 680 data = self._impl.reduce(data) --> 681 self._cache_file.save(key, data) 682 683 @contextlib.contextmanager lib/python3.7/site-packages/numba/core/caching.py in save(self, key, data) 494 break 495 overloads[key] = data_name --> 496 self._save_index(overloads) 497 self._save_data(data_name, data) 498 lib/python3.7/site-packages/numba/core/caching.py in _save_index(self, overloads) 540 def _save_index(self, overloads): 541 data = self._source_stamp, overloads --> 542 data = self._dump(data) 543 with self._open_for_write(self._index_path) as f: 544 pickle.dump(self._version, f, protocol=-1) lib/python3.7/site-packages/numba/core/caching.py in _dump(self, obj) 568 569 def _dump(self, obj): --> 570 return pickle.dumps(obj, protocol=-1) 571 572 @contextlib.contextmanager TypeError: can't pickle weakref objects ```

[sxia-cb]

@adilosa, do you know how to solve this issue?

[lmcinnes]

@adilosa This is an issue with numba interacting with whatever disk is available for caching; I'm not sure there is much I can do because this is deep into numba. Ostensibly you can try setting environment variables for the numba cache directory to point it to more reliable disk. I would definitely reach out to the numba team for help with this however.

[adilosa]

Ah, ok, thanks Leland. For reference, I had this issue on the latest umap commit. Normally I use umap==0.4.6, pynndescent==0.4.8, and numba==0.51.2 – and reinstalling these versions work just fine as usual.

@sxia-cb, perhaps the issue is with a recent numba release, maybe roll back a version and see if it goes away. the related issues also mention it might be a problem with cloud storage

Same stack trace in numba/numba#7279, also referencing lmcinnes/pynndescent#133

[rxjx]

I ran into this problem and found a way around it. Hopefully this might help others. My code was running fine on local machines but resulting in this error when run on Colab Pro. The solution referenced in the numba forums related to env vars did not work when using the default Colab libraries and umap installed via 'pip install umap-learn'. However, the error went away if I updated libraries to these versions and then used the env var suggestions

!pip install tbb --upgrade !pip install numba==0.52.0 !pip install pynndescent==0.5.2 !mkdir /tmp/numba_cache %env NUMBA_CACHE_DIR=/tmp/numba_cache !pip install umap-learn

Note that you have to restart the Colab instance until these new versions 'take'. Ideally check version for each.

Another way to make it work is to just grab the latest versions of these dependencies in which case you don't need the env var change: !pip install --upgrade tbb numba pynndescent umap-learn. Of course you still have to restart Colab to ensure the latest versions are in play.