search

ijiraq / gemini_processing

scripts and stuff to process gemini data witin the CADC.
1 stars 1 forks source link

Parallelism in DRAGONS #39

Closed Nat1405 closed 3 years ago

Nat1405 commented 3 years ago

For the record, in case anyone is wondering about running Gemini DRAGONS in parallel:

NIRI imaging processing with DRAGONS seems to work fairly well with the standard python multiprocessing library. However, a few steps require SExtractor and seem to be writing to temp folders that disappear in between steps. The relevant error message is something like:

2021-02-25 13:01:18 STATUS   -    PRIMITIVE: detectSources
2021-02-25 13:01:18 STATUS   -    ------------------------
2021-02-25 13:01:19 ERROR    - ERROR - Reduce received an unhandled exception. Aborting ...
2021-02-25 13:01:19 ERROR    - ERROR - Traceback (most recent call last):
2021-02-25 13:01:19 ERROR    - ERROR -   File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/recipe_system/reduction/coreReduce.py", line 229, in runr
2021-02-25 13:01:19 ERROR    - ERROR -     recipe(p)
2021-02-25 13:01:19 ERROR    - ERROR -   File "/Users/comeaun/hia_2020/reductions/niri/tim_davidge/GN-2007B-Q-85/exp_firsttry/myrecipelibrary.py", line 86, in reduce_w_pattern_noise_standard
2021-02-25 13:01:19 ERROR    - ERROR -     p.detectSources(stream='skysub')
2021-02-25 13:01:19 ERROR    - ERROR -   File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/recipe_system/utils/decorators.py", line 155, in gn
2021-02-25 13:01:19 ERROR    - ERROR -     ret_value = fn(pobj, adinputs=adinputs, **dict(config.items()))
2021-02-25 13:01:19 ERROR    - ERROR -   File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/geminidr/core/primitives_photometry.py", line 214, in detectSources
2021-02-25 13:01:19 ERROR    - ERROR -     sex_task.run()
2021-02-25 13:01:19 ERROR    - ERROR -   File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/gempy/gemini/eti/sextractoreti.py", line 106, in run
2021-02-25 13:01:19 ERROR    - ERROR -     self.clean()
2021-02-25 13:01:19 ERROR    - ERROR -   File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/gempy/eti_core/eti.py", line 142, in clean
2021-02-25 13:01:19 ERROR    - ERROR -     fil.clean()
2021-02-25 13:01:19 ERROR    - ERROR -   File "/opt/miniconda2/envs/dragons/lib/python3.6/site-packages/gempy/gemini/eti/sextractoretifile.py", line 75, in clean
2021-02-25 13:01:19 ERROR    - ERROR -     os.remove(self._disk_file)
2021-02-25 13:01:19 ERROR    - ERROR - FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/__/kmtshyr526n5xz4k_x3qgjs80000gq/T/N20080121S0176_skySubtracted_1.fits'

It's probably possible to wrap this in a try/except or get parallelism working with a minimum amount of effort; it's just not quite working perfectly yet out of the box.