search

ArcetriAdaptiveOptics / arte

Arcetri Random sTuff collEction
MIT License
2 stars 0 forks source link

multiprocessing on Mac OSX #18

Open lbusoni opened 3 years ago

lbusoni commented 3 years ago

Tests are not passing on OSX because of utils.shared_array that uses multiprocessing in a way that is not supported by mac os x

On python 3.8.5 darwin: FAILED test/utils/shared_array_test.py::SharedArrayTest::test_data - AttributeError: Can't pickle local object 'SharedArrayTest.test_data.<locals>.task'

I haven't investigated the details yet

lbusoni commented 3 years ago

SharedArray is actually part of the standard library since 3.8 https://docs.python.org/3/library/multiprocessing.shared_memory.html

alfiopuglisi commented 3 years ago

Uffa. Funzionava tanto bene (tranne che sul mac). I don't think that we have set requirements on the exact Python 3 version, right?

lbusoni commented 3 years ago

It looks like it doesn't work on Windows as well. The issue is related to the way forking is done: with fork() the serialization of pickle works fine. With spawn() (that is the new standard of multiprocess.Process since 3.8 on Unix and it has been the default value since a long tie on MacOs and Windows) the pickle serialization fails.

It is described here for instance https://chrissardegna.com/blog/multiprocessing-changes-python-3-8/