search

dstndstn / astrometry.net

Astrometry.net -- automatic recognition of astronomical images
http://astrometry.net
Other
662 stars 186 forks source link

fix 2 GB limit on multiprocessing / timingpool calls #188

Closed dstndstn closed 4 years ago

dstndstn commented 4 years ago

When sending giant, eg numpy arrays to a mp.map call, we can get

File "/global/homes/d/dstn/astrometry/astrometry/util/timingpool.py", line 375, in timing_handle_tasks put(task) File "/global/homes/d/dstn/astrometry/astrometry/util/timingpool.py", line 146, in send return self.real.send_bytes(s) File "/global/common/software/desi/cori/desiconda/20190311-1.2.7-img/conda/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/global/common/software/desi/cori/desiconda/20190311-1.2.7-img/conda/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes header = struct.pack("!i", n) struct.error: 'i' format requires -2147483648 <= number <= 2147483647

This comes from multiprocessing's connection.py class, _send_bytes where it sends a 4-byte signed int to pass the buffer size via header = struct.pack("!i", n)

dstndstn commented 4 years ago

Python 3.8 has a backwards-compatible fix https://github.com/python/cpython/blob/master/Lib/multiprocessing/connection.py#L395

dstndstn commented 4 years ago

fixed in 45dad121769b833033b4f5a1c63d26d0dc8efa56