MVM: Handle Astrodrizzle createMedian error

[stscijgbot-hstdp]

Issue HLA-609 was created on JIRA by Michele De La Pena:

Processing SkyCell p0081x16y14 encountered a problem in AstroDrizzle causing the processing to fail.  The traceback is below. Since the MVM dataset is comprised of greater than 700 images which take a significant amount of time to process, the output from this processing run is preserved on Freyja in /home/mdelapena/HLA/MVM/PRESERVE.

[INFO    ] reference sky value for image 'hst_skycell-p0081x16y14_wfc3_uvis_f625w_all-iehc23a7_flc.fits' is 74.766845703125
[ERROR   ] ##############################################################################
[ERROR   ] #                                                                            #
[ERROR   ] # ERROR:                                                                     #
[ERROR   ] # AstroDrizzle Version 3.2.0rc1.dev113+g89078f91 encountered a problem!      #
[ERROR   ] # Processing terminated at 22:02:47.14 (29/07/2021).                         #
[ERROR   ] #                                                                            #
[ERROR   ] ##############################################################################
[INFO    ]
[INFO    ]    --------------------          --------------------
[INFO    ]                    Step          Elapsed time
[INFO    ]    --------------------          --------------------
[INFO    ]
[INFO    ]          Initialization          46.0418 sec.
[INFO    ]             Static Mask          13.1141 sec.
[INFO    ]            Subtract Sky          1867.6460 sec.
[INFO    ]        Separate Drizzle          5814.4448 sec.
[INFO    ]           Create Median          0.0000 sec.
[INFO    ]    ====================          ====================
[INFO    ]                   Total          7741.2467 sec.
[INFO    ]
[INFO    ] Trailer file written to:  hst_skycell-p0081x16y14_wfc3_uvis_f625w_all_trl.log
[ERROR   ] An unhandled exception ocurred:
Traceback (most recent call last):
  File "/home/mdelapena/repos/drizzlepac/drizzlepac/util.py", line 230, in wrapper
    result = func(*args, **kwargs)
  File "/home/mdelapena/repos/drizzlepac/drizzlepac/astrodrizzle.py", line 223, in run
    createMedian.createMedian(imgObjList, configobj, procSteps=procSteps)
  File "/home/mdelapena/repos/drizzlepac/drizzlepac/createMedian.py", line 99, in createMedian
    _median(imgObjList, paramDict)
  File "/home/mdelapena/repos/drizzlepac/drizzlepac/createMedian.py", line 360, in _median
    imdrizSectionsList[i, :, :] = w[e1:e2]
  File "/home/mdelapena/miniconda3/envs/caldp_20210721/lib/python3.6/site-packages/stsci/tools/iterfile.py", line 89, in __getitem__
    hdu = self.open()
  File "/home/mdelapena/miniconda3/envs/caldp_20210721/lib/python3.6/site-packages/stsci/tools/iterfile.py", line 66, in open
    self.handle = fits.open(self.fname, mode='readonly')
  File "/home/mdelapena/miniconda3/envs/caldp_20210721/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 164, in fitsopen
    lazy_load_hdus, **kwargs)
  File "/home/mdelapena/miniconda3/envs/caldp_20210721/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 402, in fromfile
    lazy_load_hdus=lazy_load_hdus, **kwargs)
  File "/home/mdelapena/miniconda3/envs/caldp_20210721/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 1051, in _readfrom
    fileobj = _File(fileobj, mode=mode, memmap=memmap, cache=cache)
  File "/home/mdelapena/miniconda3/envs/caldp_20210721/lib/python3.6/site-packages/astropy/utils/decorators.py", line 535, in wrapper
    return function(*args, **kwargs)
  File "/home/mdelapena/miniconda3/envs/caldp_20210721/lib/python3.6/site-packages/astropy/io/fits/file.py", line 175, in __init__
    self._open_filename(fileobj, mode, overwrite)
  File "/home/mdelapena/miniconda3/envs/caldp_20210721/lib/python3.6/site-packages/astropy/io/fits/file.py", line 564, in _open_filename
    self._file = fileobj_open(self.name, IO_FITS_MODES[mode])
  File "/home/mdelapena/miniconda3/envs/caldp_20210721/lib/python3.6/site-packages/astropy/io/fits/util.py", line 397, in fileobj_open
    return open(filename, mode, buffering=0)
FileNotFoundError: [Errno 2] No such file or directory: 'hst_skycell-p0081x16y14_wfc3_uvis_f625w_all-idad20w4_single_sci.fits'*no* further _formatting_ is done here```

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

2021210191247 INFO src=drizzlepac.astrodrizzle- USER INPUT PARAMETERS common to all Processing Steps: 2021210191247 INFO src=drizzlepac.astrodrizzle- build : True 2021210191247 INFO src=drizzlepac.astrodrizzle- coeffs : True 2021210191247 INFO src=drizzlepac.astrodrizzle- context : True 2021210191247 INFO src=drizzlepac.astrodrizzle- crbit : 4096 2021210191247 INFO src=drizzlepac.astrodrizzle- group : 2021210191247 INFO src=drizzlepac.astrodrizzle- in_memory : False

[stscijgbot-hstdp]

Comment by Warren Hack on JIRA:

A review of this datasets products and logs from the results_2021-08-06 test showed this skycell exiting at a different point in the processing. In fact, in the 2021-08-06 test resulted in 7 layers being generated, whereas none were successfully generated in the 2021-07-30 test. This supports the contention that processing ended due to out-of-memory issues or some similar resource issue during processing.

[stsci-hack]

Closed by #1219 (at the least).