anthonydb / pneumatic

pneumatic is a bulk-upload library for DocumentCloud.
MIT License
23 stars 3 forks source link

Python 3.8 multiprocessing behavior breaks pneumatic on macOS #21

Open anthonydb opened 4 years ago

anthonydb commented 4 years ago

Per https://bugs.python.org/issue33725, on macOS the default behavior of the multiprocessing module is spawn instead of fork, which has been deemed unsafe in newer macOS versions. As a result, the way multiprocessing was implemented in pneumatic no longer works; instead, the uploader loops.

Removed multiprocessing via https://github.com/anthonydb/pneumatic/commit/56654a2b46878978b26f260f7bee0817950a19e7 for now.

The benefit of multiprocessing as originally used is questionable since uploads are bound by network capability. At most, it just queues up files faster. Needs research.