New jwst sim data for unit tests

SaOgaz commented 5 years ago

Additionally, this data need to be pre-tested for compatibility with Cubeviz (shouldn't be any problems)

Got some new level 2 s3d files for testing, this is the level 2 products for the det_iamge_ch1-short... files, which are level 3 products: det_image_seq1_MIRIFUSHORT_12SHORTexp1_s3d.fits det_image_seq2_MIRIFUSHORT_12SHORTexp1_s3d.fits

Files can be found in box folder per usual: https://stsci.app.box.com/folder/69214387860

ibusko commented 5 years ago

I tried this in two ways:

build cubeviz from master (as per instructions in here. In this case, the file reader window doesn't even allow access to the test files (they are grayed out).
install cubeviz as per user-facing instructions (here). In this case, the file reader allows access to the files, but then an exception ensues (after a few warning messages):

Traceback (most recent call last):
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz020/lib/python3.7/site-packages/glue/app/qt/application.py", line 233, in _choose_load_data_wizard
    self._choose_load_data(data_importer=data_wizard)
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz020/lib/python3.7/site-packages/glue/app/qt/application.py", line 252, in _choose_load_data
    app.add_datasets(app.data_collection, data)
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz020/lib/python3.7/site-packages/glue/core/application_base.py", line 293, in add_datasets
    data_collection.extend(datasets)
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz020/lib/python3.7/site-packages/glue/core/data_collection.py", line 93, in extend
    self.append(d)
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz020/lib/python3.7/site-packages/glue/core/data_collection.py", line 79, in append
    self.hub.broadcast(msg)
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz020/lib/python3.7/site-packages/glue/core/hub.py", line 217, in broadcast
    handler(message)
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz020/lib/python3.7/site-packages/cubeviz/listener.py", line 51, in handle_new_dataset
    self.configure_layout(data)
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz020/lib/python3.7/site-packages/cubeviz/listener.py", line 61, in configure_layout
    self.setup_data(cubeviz_layout, data)
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz020/lib/python3.7/site-packages/cubeviz/listener.py", line 119, in setup_data
    cubeviz_layout.add_data(data)
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz020/lib/python3.7/site-packages/cubeviz/layout.py", line 638, in add_data
    self.specviz._widget.add_data(data)
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz020/lib/python3.7/site-packages/specviz/third_party/glue/data_viewer.py", line 283, in add_data
    self._spectrum_from_component(layer, component, layer.coords.wcs, cid=cid)
AttributeError: 'Coordinates' object has no attribute 'wcs'

The Header Value Issues window pops up with missing CTYPEx and CUNITx keywords in all 4 extensions. Clicking on the Accept button doesn't cause further errors, but the data seems incomplete. The displayed arrays have no axis names nor units, can't be zoomed or panned, and most of the GUI controls are grayed out.

The keywords reported as problematic by the Header Value Issues all have the value NONE.

ibusko commented 5 years ago

Running Dan's tool on these files, we get this:

(cubeviz-master) anakin:/Users/busko $ jwstinfo ~/Downloads/det_image_seq1_MIRIFUSHORT_12SHORTexp1_s3d.fits
/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz-master/lib/python3.6/site-packages/asdf-2.3.2-py3.6.egg/asdf/yamlutil.py:251: AsdfConversionWarning: tag:stsci.edu:gwcs/step-1.0.0 is not recognized, converting to raw Python data structure
  "data structure".format(tag_name), AsdfConversionWarning)
/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz-master/lib/python3.6/site-packages/asdf-2.3.2-py3.6.egg/asdf/yamlutil.py:251: AsdfConversionWarning: tag:stsci.edu:gwcs/celestial_frame-1.0.0 is not recognized, converting to raw Python data structure
  "data structure".format(tag_name), AsdfConversionWarning)
/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz-master/lib/python3.6/site-packages/asdf-2.3.2-py3.6.egg/asdf/yamlutil.py:251: AsdfConversionWarning: tag:stsci.edu:gwcs/spectral_frame-1.0.0 is not recognized, converting to raw Python data structure
  "data structure".format(tag_name), AsdfConversionWarning)
/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz-master/lib/python3.6/site-packages/asdf-2.3.2-py3.6.egg/asdf/yamlutil.py:251: AsdfConversionWarning: tag:stsci.edu:gwcs/composite_frame-1.0.0 is not recognized, converting to raw Python data structure
  "data structure".format(tag_name), AsdfConversionWarning)
/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz-master/lib/python3.6/site-packages/asdf-2.3.2-py3.6.egg/asdf/yamlutil.py:251: AsdfConversionWarning: tag:stsci.edu:gwcs/wcs-1.0.0 is not recognized, converting to raw Python data structure
  "data structure".format(tag_name), AsdfConversionWarning)
Traceback (most recent call last):
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz-master/bin/jwstinfo", line 11, in <module>
    load_entry_point('jwstinfo==0.1.dev14+gbe50284', 'console_scripts', 'jwstinfo')()
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz-master/lib/python3.6/site-packages/jwstinfo-0.1.dev14+gbe50284-py3.6.egg/jwstinfo/jwstinfo.py", line 76, in main
    metadata = read_metadata(sys.argv[1])
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz-master/lib/python3.6/site-packages/jwstinfo-0.1.dev14+gbe50284-py3.6.egg/jwstinfo/jwstinfo.py", line 61, in read_metadata
    return parse_metadata(af)
  File "/Users/busko/Projects/miniconda3_mosviz/envs/cubeviz-master/lib/python3.6/site-packages/jwstinfo-0.1.dev14+gbe50284-py3.6.egg/jwstinfo/jwstinfo.py", line 38, in parse_metadata
    'target name':          metadata['target']['catalog_name'],
KeyError: 'catalog_name'
(cubeviz-master) anakin:/Users/busko $

drdavella commented 5 years ago

Looks like I need to make the tool a little more robust. Can you share that file with me?

ibusko commented 5 years ago

@drdavella the files are in Box as described in the topmost comment.

SaOgaz commented 5 years ago

An initial FITS Info does show a WCS extensions in the det_image file that's not in the other working ditherunity file we have:

In [7]: fits.info('det_image_seq1_MIRIFUSHORT_12SHORTexp1_s3d.fits')
Filename: det_image_seq1_MIRIFUSHORT_12SHORTexp1_s3d.fits No. Name Ver Type Cards Dimensions Format 0 PRIMARY 1 PrimaryHDU 244 ()
1 SCI 1 ImageHDU 64 (42, 40, 4149) float32
2 ERR 1 ImageHDU 12 (42, 40, 4149) float32
3 DQ 1 ImageHDU 12 (42, 40, 4149) int32 (rescales to uint32)
4 WMAP 1 ImageHDU 10 (42, 40, 4149) float32
5 WCS-TABLE 1 BinTableHDU 15 1R x 2C [I, 10420E]
6 ASDF 1 BinTableHDU 11 1R x 1C [133765B]

In [8]: fits.info('ditherunity_CLEAR_PRISM_M1_m1_noiseless_NRS1_modified_updatedHDR_fixintarget_s3d.fits')
Filename: ditherunity_CLEAR_PRISM_M1_m1_noiseless_NRS1_modified_updatedHDR_fixintarget_s3d.fits No. Name Ver Type Cards Dimensions Format 0 PRIMARY 1 PrimaryHDU 363 ()
1 SCI 1 ImageHDU 60 (46, 48, 941) float32
2 ERR 1 ImageHDU 12 (46, 48, 941) float32
3 DQ 1 ImageHDU 12 (46, 48, 941) int32 (rescales to uint32)
4 WMAP 1 ImageHDU 10 (46, 48, 941) float32
5 ASDF 1 BinTableHDU 11 1R x 1C [9509B]

SaOgaz commented 5 years ago

Looking at the WCS keywords in the 1st extensions for the working and not working files, I see two differences, which match what @jdavies-st updated me on about astropy WCS.

Our not working files have: CTYPE3 = 'WAVE-TAB' / third axis coordinate type PS3_0 = 'WCS-TABLE' / Coordinate table extension name
PS3_1 = 'wavelength' / Coordinate table column name

While the working files has: CTYPE3 = 'WAVE ' / third axis coordinate type

Essentially, these files have a FITS WCS object, which is not currently readable by astropy wcs coordinates, and they have the gwcs object stored in the asdf extension, which is how the pipeline uses it, but not what we are using for these tools.

James pointed out several issues to me that are open to get astropy.wcs to be able to injest the FITS WCS stuff, but said that no one is currently working on these issues.

https://github.com/spacetelescope/jwst/issues/2235 https://github.com/spacetelescope/jwst/issues/2685

drlaw1558 commented 5 years ago

In case this is helpful: the cube-building algorithm is designed so that it can produce data cubes both on linear wavelength solutions and also on arbitrarily-defined solutions. In the former case the usual FITS header keywords suffice to describe it, and in the latter case the WAVE-TAB infrastructure is employed. This is because some MRS data cubes may cover a sufficiently large wavelength range that a linear wavelength solution is undesirable (giving perhaps 10 samples per spectral resolution element at longer wavelengths, or < 1 at shorter wavelengths). Most of the default cubes produced by the SPEC3 pipeline are for short wavelength ranges, and thus use linear wavelength solutions. In contrast, the cubes produced by the SPEC2 pipeline combine data from multiple different wavelengths (e.g., 1A +2A) since both are on the same detector at the same time. Since the SPEC2 cubes are regarded as largely-internal intermediate products it was determined that these could use WAVETAB solutions without causing too much end-user confusion.

eteq commented 5 years ago

@SaOgaz or @jdavies-st : can you point to the specific issues on this in the astropy repo? I would have thought the above would work fine if it follows the WCS spec...

Or if it's easier to just answer the key question for understanding timelines: is the problem on the "astropy side" (i.e., the Python interface), the wrapped c-library, or not clear? I ask because the first is probably pretty fixible (and maybe even work-around-able in cubeviz temporarily), while the latter probably requires a fix and we might need to assign someone to do a deep dive on it.

jdavies-st commented 5 years ago

The WAVE-TAB part of the FITS WCS standard (Calabretta paper 3) is impemented in the WCSLIB C library, but it is not wrapped by astropy.wcs yet. That's where the work needs to be done I believe. Though perhaps @nden has some more insight.

This is also a useful discussion:

https://github.com/astropy/astropy/issues/2362

https://github.com/astropy/astropy/issues/5462

ds9 (employing the AST WCS library) can read these files with WCS-TABLE extensions and WAVE-TAB keywords without problem.

spacetelescope / cubeviz

New jwst sim data for unit tests #550