search

tgbugs / pyontutils

python utilities for working with ontologies
MIT License
19 stars 123 forks source link

joblib 0.12.2 unpickling issues #31

Closed tgbugs closed 4 years ago

tgbugs commented 6 years ago

There seems to have been a change to something that joblib is doing. Possibly related to https://github.com/joblib/joblib/issues/745. When running parcellation.py all pools show the following error.

Process ForkPoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/home/tom/.local/share/virtualenvs/pyontutils-a77fIBQE/lib/python3.6/site-packages/joblib/pool.py", line 149, in get
    return recv()
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 253, in recv
    return _ForkingPickler.loads(buf.getbuffer())
TypeError: __new__() takes 1 positional argument but 2 were given

So far this has been traced to /usr/lib64/python3.6/multiprocessing/connection.py when using backend='multiprocessing' but also occurs for other backends. When saving the pickle to disk and trying to reopen in another interpreter the same error occurs.

tgbugs commented 4 years ago

Resolved by switching to >=joblib-0.14.0.