NIFS Processing Progress

Nat1405 commented 3 years ago

The goal is to summarize processing of the Gemini NIFS archive.

1933 Science Observations
345 Science Programs

In the creation of basic, uncorrected, unmerged data cubes:

165 programs passed
180 failed
- 51 due to wavelength errors
- 57 crashed
- 59 terminated, but had less than 90% of constituent science data reduced
- 13 AWOL

Nat1405 commented 3 years ago

Here is a slightly newer summary:

|Number of science programs:    | 345  |
|Number of failures:            | 169  |
|Failures by type:              |      |
|Output not found:              | 0    |
|Unsupported wavelength:        | 52   |
|Less than 90 percent:          | 60   |
|Crashed:                       | 57   |
|   - No space left:            | 6    |
|   - IRAF:                     | 18   |
|   - Frame Failed to Download: | 5    |
|   - CADC Registry Error:      | 0    |
|   - UnboundLocal:             | 13   |
|   - IndexError:               | 7    |
|   - IOError:                  | 3    |
|   - QueryError:               | 0    |
|   - OSError:                  | 1    |
|   - ValueError:               | 2    |
|   - Interactive Wave Error:   | 2    |
|   - Unknown:                  | 0    |

Fixes on the go:

UnboundLocal: seems to be a problem with telluric/flux corrections. Turning those off for now.
IndexError: seems due to several things. First, some science/telluric frames still aren't getting enough calibrations and that wasn't being handled. Second, telluric spectrum combining wasn't handling the case of no spectra being provided to it. I might sweep that under the rug until we start tackling telluric corrected cubes. And lastly, a single IndexError was triggered when a frame failed to download.
IRAF Errors: first, task fxcopy is failing somewhere in the cube merging step. Turning off merging for now. Second, task files failing seems due to gemcombine trying to combine too many files together. One of the last lines in an output file before seeing that error is the following, and much longer. Third, imcombine errors are possibly due to too many files being provided to imcombine. I'm not completely sure of this though. This is in the cube merging step so I'm turning it off for now. Lastly, there are one or two failures each of the tasks telluric, imgets and imcopy. Not sure what caused these last three.
```
GEMCOMBINE: input = xtfbrsnN20140404S0244,xtfbrsnN20140404S0223,xtfbrsnN20140404S0265,xtfbrsnN20140404S0208,xtfbrsnN20140404S0247,xtfbrsnN20140404S0226,xtfbrsnN20140404S0300,xtfbrsnN20140404S0207,xtfbrsnN20140404S0202,xtfbrsnN20140404S0271,xtfbrsnN20140404S0268,xtfbrsnN20140404S0231,xtfbrsnN20140404S0253,xtfbrsnN20140404S0291,xtfbrsnN20140404S0214,xtfbrsnN20140404S0186,xtfbrsnN20140404S0306,xtfbrsnN20140404S0238,xtfbrsnN20140404S0303,xtfbrsnN20140404S0240,xtfbrsnN20140404S0217,xtfbrsnN20140404S0204,xtfbrsnN20140404S0177,xtfbrsnN20140404S0292,xtfbrsnN20140404S0250....
```
Less than 90 percent complete programs. Several programs are showing a failure to find sufficient calibrations, including arc darks and flats. There was a bug in the plus/minus one day downloader code which seemed to render it useless. In summary, I'm not sure yet what's causing these programs to be less than 90 percent complete.

Nat1405 commented 3 years ago

My batch workflow seems to be fully operational again; results coming soon.

Nat1405 commented 3 years ago

Here are the latest NIFS processing results:

|Number of science programs:    | 345  |
|Number of failures:            | 144  |
|Failures by type:              |      |
|Output not found:              | 0    |
|Unsupported wavelength:        | 52   |
|Less than 90 percent:          | 62   |
|    - CalibrationsNotFound     | 25   |
|    - SkyFrameError            | 47   |
|    - Unknown                  | 5    |
|Crashed:                       | 30   |
|   - No space left:            | 8    |
|   - IRAF:                     | 3    |
|   - Frame Failed to Download: | 3    |
|   - Connection Error:         | 1    |
|   - CADC Registry Error:      | 0    |
|   - UnboundLocal:             | 0    |
|   - IndexError:               | 4    |
|   - IOError:                  | 4    |
|   - QueryError:               | 0    |
|   - OSError:                  | 1    |
|   - ValueError:               | 2    |
|   - Interactive Wave Error:   | 4    |
|   - Unknown:                  | 0    |

Note that some of these errors may not be mutually exclusive.

Based on these results, my new, subject to change, priorities are:

Fix out of memory errors by moving some jobs to larger VMs
Figure out how many programs don't have enough calibrations, to triage the CalibrationsNotFound errors.
Triage the sky frame issue because 46 programs are being affected by it.

Nat1405 commented 3 years ago

Here are the latest batch results:

|Number of science programs:    | 345  |
|Number passed:                 | 219  |
|Number of failures:            | 126  |
|Failures by type:              |      |
|Output not found:              | 1    |
|Unsupported wavelength:        | 52   |
|Less than 90 percent:          | 41   |
|    - CalibrationsNotFound     | 41   |
|    - SkyFrameError            | 0    |
|    - Unknown                  | 0    |
|Crashed:                       | 32   |
|   - No space left:            | 14   |
|   - IRAF:                     | 1    |
|   - Frame Failed to Download: | 1    |
|   - Connection Error:         | 0    |
|   - CADC Registry Error:      | 0    |
|   - UnboundLocal:             | 0    |
|   - IndexError:               | 8    |
|   - IOError:                  | 0    |
|   - QueryError:               | 0    |
|   - OSError:                  | 1    |
|   - ValueError:               | 4    |
|   - Interactive Wave Error:   | 3    |
|   - Unknown:                  | 0    |

Based on these results, I will triage:

out of memory errors
CalibrationErrors: 28 programs are removing a directory because of insufficient arc dark frames. I can try easing the requirement for darks for the arc frames for those programs and that might help.
IndexErrors, ValueErrors, and triaging the interactive wave errors if time allows. It would be nice to try to fix #37 at some point.

Nat1405 commented 3 years ago

Latest NIFS batch results, giving 67% completion:

|Number of science programs:    | 345  |
|Number passed:                 | 230  |
|Number of failures:            | 115  |
|Failures by type:              |      |
|Output not found:              | 0    |
|Unsupported wavelength:        | 52   |
|Less than 90 percent:          | 40   |
|    - CalibrationsNotFound     | 40   |
|    - SkyFrameError            | 0    |
|    - Unknown                  | 0    |
|Crashed:                       | 23   |
GN-2009B-Q-40
GN-2010A-Q-79
GN-2011A-Q-56
|   - No space left:            | 3    |
|   - IRAF:                     | 4    |
|   - Frame Failed to Download: | 1    |
|   - Connection Error:         | 0    |
|   - CADC Registry Error:      | 0    |
|   - UnboundLocal:             | 0    |
|   - IndexError:               | 0    |
|   - IOError:                  | 2    |
|   - QueryError:               | 0    |
|   - OSError:                  | 1    |
|   - ValueError:               | 9    |
|   - Interactive Wave Error:   | 2    |
|   - Unknown:                  | 1    |

One of the out of memory errors seems fixable, so that program has been resubmitted. This gives a final completion percentage of 67%.

Nat1405 / Nifty4Gemini

NIFS Processing Progress #34