Errors in processing image that is larger than 32k by 32k

[cnwangfeng]

Hi,

We got an error when we process the image with 32768x32768.

Running cmd: aegean --table aegean_32K.csv realistic_dipole_element_beams_1p2arcsec_32768_nmoment3_cip.taylor.0.restored.fits

The error messages: py3.7.egg/AegeanTools/pprocess.py:410: UserWarning: Removed channel <AegeanTools.pprocess.Channel object at 0x7f37274c5090> due to exception: [Errno 32] Broken pipe warn("Removed channel %r due to exception: %s" % (channel, exc)) Traceback (most recent call last): File "/software/anaconda3/envs/testsc/bin/aegean", line 4, in import('pkg_resources').run_script('AegeanTools==2.2.5', 'aegean') File "/software/anaconda3/envs/testsc/lib/python3.7/site-packages/pkg_resources/init.py", line 665, in run_script self.require(requires)[0].run_script(script_name, ns) File "/software/anaconda3/envs/testsc/lib/python3.7/site-packages/pkg_resources/init.py", line 1463, in run_script exec(code, namespace, namespace) File "/software/anaconda3/envs/testsc/lib/python3.7/site-packages/AegeanTools-2.2.5-py3.7.egg/EGG-INFO/scripts/aegean", line 347, in progress=options.progress) File "/software/anaconda3/envs/testsc/lib/python3.7/site-packages/AegeanTools-2.2.5-py3.7.egg/AegeanTools/source_finder.py", line 2578, in find_sources_in_image for srcs in queue: File "/software/anaconda3/envs/testsc/lib/python3.7/site-packages/AegeanTools-2.2.5-py3.7.egg/AegeanTools/pprocess.py", line 908, in next self.store() File "/software/anaconda3/envs/testsc/lib/python3.7/site-packages/AegeanTools-2.2.5-py3.7.egg/AegeanTools/pprocess.py", line 406, in store self.store_data(channel) File "/software/anaconda3/envs/testsc/lib/python3.7/site-packages/AegeanTools-2.2.5-py3.7.egg/AegeanTools/pprocess.py", line 888, in store_data data = channel.receive() File "/software/anaconda3/envs/testsc/lib/python3.7/site-packages/AegeanTools-2.2.5-py3.7.egg/AegeanTools/pprocess.py", line 141, in receive obj = self._receive() File "/software/anaconda3/envs/testsc/lib/python3.7/site-packages/AegeanTools-2.2.5-py3.7.egg/AegeanTools/pprocess.py", line 127, in _receive raise obj File "/software/anaconda3/envs/testsc/lib/python3.7/site-packages/AegeanTools-2.2.5-py3.7.egg/AegeanTools/pprocess.py", line 1205, in start callable(channel, *args, **kw) File "/software/anaconda3/envs/testsc/lib/python3.7/site-packages/AegeanTools-2.2.5-py3.7.egg/AegeanTools/pprocess.py", line 957, in call t = channel.receive() File "/software/anaconda3/envs/testsc/lib/python3.7/site-packages/AegeanTools-2.2.5-py3.7.egg/AegeanTools/pprocess.py", line 141, in receive obj = self._receive() File "/software/anaconda3/envs/testsc/lib/python3.7/site-packages/AegeanTools-2.2.5-py3.7.egg/AegeanTools/pprocess.py", line 125, in _receive obj = pickle.load(self.read_pipe) _pickle.UnpicklingError: pickle data was truncated

[PaulHancock]

You are seeing the same error as described in #108. The pprocess library cannot handle the pickling and un-pickling of large objects which means that multi-core support doesn't work on very large images. A current work in progress aims to remove the dependency on pprocess so that we can get around this issue but for now the solutions are:

• use a single core
• divide the image into smaller pieces, source find, and then stitch the catalogues together

Neither of the above are ideal.

[cnwangfeng]

Thanks a lot. Sorry for the duplication. I'll close the comment and follow #108.