Error unpickling sharedmem object where inner dimension is 0

[takluyver]

Just ran into this. The array shouldn't have had a zero-length dimension, but failures unpickling seem to cause multiprocessing.Pool() to get stuck waiting for tasks that aren't running.

import pickle
import sharedmem

sha = sharedmem.empty((5, 0), dtype='f4')
b = pickle.dumps(sha)
print(pickle.loads(b))

Running it gives:

$ python3 sharedmem_pickle.py 
/usr/lib64/python3.8/site-packages/numpy/ctypeslib.py:523: RuntimeWarning: A builtin ctypes object gave a PEP3118 format string that does not match its itemsize, so a best-guess will be made of the data type. Newer versions of python may behave correctly.
  return array(obj, copy=False)
Traceback (most recent call last):
  File "sharedmem_pickle.py", line 6, in <module>
    print(pickle.loads(b))
  File "/home/takluyver/.local/lib/python3.8/site-packages/sharedmem/sharedmem.py", line 863, in __unpickle__
    buffer = numpy.ctypeslib.as_array(ra).ravel()
  File "/usr/lib64/python3.8/site-packages/numpy/ctypeslib.py", line 523, in as_array
    return array(obj, copy=False)
  File "/usr/lib64/python3.8/site-packages/numpy/core/_dtype_ctypes.py", line 102, in dtype_from_ctypes_type
    return _from_ctypes_array(t)
  File "/usr/lib64/python3.8/site-packages/numpy/core/_dtype_ctypes.py", line 32, in _from_ctypes_array
    return np.dtype((dtype_from_ctypes_type(t._type_), (t._length_,)))
ValueError: invalid itemsize in generic type tuple

[rainwoodman]

Thanks for the reproducer! The fix only addresses the crash.

The pool stuck issue can be more subtle. Let me know if the error persists.

[takluyver]

Thanks for the quick fix! This seems to solve things for our use case - we get an error in our code instead of an error transferring the task, so it crashes rather than hanging, which is what we expected. :)

If you've got time to do a new release with the fix, that would be very convenient for us, but I don't want to be demanding!

[rainwoodman]

No problem. 0.3.8 is up, thanks to travis auto-deployment.

On Thu, Oct 1, 2020 at 1:50 AM Thomas Kluyver notifications@github.com wrote:

Thanks for the quick fix! This seems to solve things for our use case - we get an error in our code instead of an error transferring the task, so it crashes rather than hanging, which is what we expected. :)

If you've got time to do a new release with the fix, that would be very convenient for us, but I don't want to be demanding!

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/rainwoodman/sharedmem/issues/21#issuecomment-701989577, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBWTDXZYUWUSRE7U3H3KLSIQ7ETANCNFSM4R7LTH5Q .

[takluyver]

Smashing, thanks!