Update validation code for upcoming GSEG test

[bsunnquist]

Here are the updates that should have us ready for the upcoming GSEG test. I made the following changes:

• Generalized to check both uncal and rate images
• Added additional header checks not included in APT file (e.g. BUNIT)
• Ensured the number of existing uncal and rate files are equal, and prints any missing files

I only updated the notebook version (I find it much easier to debug/further explore the data using the notebook version), but let me know if you want me to update the python file as well (not sure if people are planning on using that).

You can do a test run if you want using these cell commands within the mirage environment (just choose your own output directory): xml_file = '/ifs/jwst/wit/witserv/data7/nrc/bsunnquist/gseg_validation_tests/gseg3otb_dryrun2/xml_files/617_DR1_v2_short.xml'

gseg_uncal_files = glob('/ifs/jwst/wit/witserv/data7/nrc/gseg3/MAST_2019-07-19T1724/JWST/*/*uncal.fits')

output_dir = '/ifs/jwst/wit/witserv/data7/nrc/bsunnquist/gseg_validation_tests/gseg3otb_dryrun2/outputs/'

validate(xml_file, output_dir, gseg_uncal_files)

A couple of other useful things to add (maybe before this upcoming test):

• correct the number of expected files (i.e. don't assume all 5 detectors will be used for each observation). I think we can check this manually for the upcoming test, but it would be nice to have it automated somehow.
• output a summary table with an entry for each file (filename, detector, aperture, etc.) for spot checks

[bsunnquist]

@arminrest just tagging you so you're aware of these changes (not sure why I can't add you as a reviewer as well)

[bsunnquist]

Thanks for the review! Let's hold off on merging for now as I'll probably make a few more additions tomorrow.

I just added some more changes to have the code verify that all expected extensions exist in each file, and that the header and data shapes match and are as expected in all of these extensions.

[aliciacanipe]

https://outerspace.stsci.edu/display/JARI/JWST+Archive%3A+Retrieve+Observations

@bhilbert4 @bsunnquist from the "API download" section, it looks like it's possible to search for data using astroquery but downloading is not yet possible.

[aliciacanipe]

@bsunnquist FYI I made a local copy and played around some of the files, messed up the headers, the shapes, etc. The notebook seems to have caught all the mismatches.

[bsunnquist]

Awesome, thanks for testing that out

[bsunnquist]

I just made another change where the image/header dimensions are compared to the expected values based on the APT subarray keyword (before we were just checking that the image shape/header keywords were internally consistent).

The function that finds the expected shape based on the APT subarray is get_expected_shape - it works for all subarrays from the previous GSEG tests (and for the upcoming test) but there may be a better way we can implement in the future.

[bsunnquist]

I'm having the code output a summary table that contains all of the info we are expected to check for each file in the upcoming test (mainly useful for confirming the correct number of files per detector, and spot checking):

As of now, this is the code I plan on using Wednesday - let me know if you want to test it out again before I merge; the recent changes all worked fine for me on the previous gseg3 data. I also confirmed it reads in John's new APT file fine for the upcoming test.