niriss x1d data doesn't load in Specviz, gives non-verbose error

[Jdaviz-Triage-Bot]

Reporter: Brian Cherinka

Looks like Jdaviz is failing to loading NIRISS x1d data in Specviz, which is expected but this one is returning a generic, non-verbose, "ERROR".  So not sure what the underlying reason is.  

Steps to reproduce:

1. Go to https://masttest.stsci.edu/viz/jwst/spectra?filename=jw00787-o014_s00002_niriss_f150w-gr150c-gr150r_x1d.fits&debug=True

2. See the big red ERROR

   ---

   DISCLAIMER: This issue was autocreated by the Jdaviz Issue Creation Bot on behalf of the reporter. If any information is incorrect, please contact Duy Nguyen

[havok2063]

Here's the full traceback of the error in a local notebook

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-9-f061f72b5e4e> in <module>
      1 # load the data file into the helper jdaviz class and display the application
      2 h = SpecViz()
----> 3 h.load_spectrum(local)
      4 h.app

~/Work/git/havok2063/jdaviz/jdaviz/configs/specviz/helper.py in load_spectrum(self, data, data_label, format, show_in_viewer)
     26 
     27     def load_spectrum(self, data, data_label=None, format=None, show_in_viewer=True):
---> 28         super().load_data(data,
     29                           'specviz-spectrum1d-parser',
     30                           data_label=data_label,

~/Work/git/havok2063/jdaviz/jdaviz/core/helpers.py in load_data(self, data, parser_reference, **kwargs)
     39 
     40     def load_data(self, data, parser_reference=None, **kwargs):
---> 41         self.app.load_data(data, parser_reference=parser_reference, **kwargs)
     42 
     43     @property

~/Work/git/havok2063/jdaviz/jdaviz/app.py in load_data(self, file_obj, parser_reference, **kwargs)
    306                 # If the parser returns something other than known, assume it's
    307                 #  a message we want to make the user aware of.
--> 308                 msg = parser(self, file_obj, **kwargs)
    309 
    310                 if msg is not None:

~/Work/git/havok2063/jdaviz/jdaviz/configs/specviz/plugins/parsers.py in specviz_spectrum1d_parser(app, data, data_label, format, show_in_viewer)
     36 
     37         if path.is_file():
---> 38             data = Spectrum1D.read(str(path), format=format)
     39         else:
     40             raise FileNotFoundError("No such file: " + str(path))

~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/nddata/mixins/ndio.py in __call__(self, *args, **kwargs)
     56 
     57     def __call__(self, *args, **kwargs):
---> 58         return registry.read(self._cls, *args, **kwargs)
     59 
     60 

~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/io/registry.py in read(cls, format, cache, *args, **kwargs)
    518 
    519         reader = get_reader(format, cls)
--> 520         data = reader(*args, **kwargs)
    521 
    522         if not isinstance(data, cls):

~/Work/git/havok2063/specutils/specutils/io/default_loaders/jwst_reader.py in jwst_x1d_single_loader(file_obj, **kwargs)
    165         The spectrum contained in the file.
    166     """
--> 167     spectrum_list = _jwst_spec1d_loader(file_obj, extname='EXTRACT1D', **kwargs)
    168     if len(spectrum_list) == 1:
    169         return spectrum_list[0]

~/Work/git/havok2063/specutils/specutils/io/default_loaders/jwst_reader.py in _jwst_spec1d_loader(file_obj, extname, **kwargs)
    249             if srctype == "POINT":
    250                 flux = Quantity(data["FLUX"])
--> 251                 uncertainty = StdDevUncertainty(data["ERROR"])
    252             elif srctype == "EXTENDED":
    253                 flux = Quantity(data["SURF_BRIGHT"])

~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/table/table.py in __getitem__(self, item)
   1638     def __getitem__(self, item):
   1639         if isinstance(item, str):
-> 1640             return self.columns[item]
   1641         elif isinstance(item, (int, np.integer)):
   1642             return self.Row(self, item)

~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/table/table.py in __getitem__(self, item)
    237         """
    238         if isinstance(item, str):
--> 239             return OrderedDict.__getitem__(self, item)
    240         elif isinstance(item, (int, np.integer)):
    241             return list(self.values())[item]

KeyError: 'ERROR'

[pllim]

Hah hah... It was literally ERROR. It can't find a column named ERROR. 😆

data["ERROR"]

[pllim]

The code in your traceback does not match what is now in https://github.com/astropy/specutils/blob/main/specutils/io/default_loaders/jwst_reader.py . What version of specutils are you using?

[havok2063]

Haha, that's pretty funny! Yeah I could be out of date. I'm using 1.3.dev43+g44f20a6f.d20210503 locally. And then on the mast machines it's whatever version gets installed via that branch of jdaviz, which looks like >1.3

[pllim]

Looks like specutils 1.3 was released a few days ago.

[havok2063]

Also looks like the NIRISS x1d files for pipeline version 1.2.3 has a new FITS structure for the EXTRACT1D extension. Maybe they always had this structure. I don't think the JWST loaders ever handled these cases.

ColDefs(
    name = 'WAVELENGTH'; format = 'D'; unit = 'um'
    name = 'FLUX'; format = 'D'; unit = 'Jy'
    name = 'FLUX_ERROR'; format = 'D'; unit = 'Jy'
    name = 'FLUX_VAR_POISSON'; format = 'D'; unit = 'Jy^2'
    name = 'FLUX_VAR_RNOISE'; format = 'D'; unit = 'Jy^2'
    name = 'FLUX_VAR_FLAT'; format = 'D'; unit = 'Jy^2'
    name = 'SURF_BRIGHT'; format = 'D'; unit = 'MJy/sr'
    name = 'SB_ERROR'; format = 'D'; unit = 'MJy/sr'
    name = 'SB_VAR_POISSON'; format = 'D'; unit = '(MJy/sr)^2'
    name = 'SB_VAR_RNOISE'; format = 'D'; unit = '(MJy/sr)^2'
    name = 'SB_VAR_FLAT'; format = 'D'; unit = '(MJy/sr)^2'
    name = 'DQ'; format = 'J'; bzero = 2147483648
    name = 'BACKGROUND'; format = 'D'; unit = 'MJy/sr'
    name = 'BKGD_ERROR'; format = 'D'; unit = 'MJy/sr'
    name = 'BKGD_VAR_POISSON'; format = 'D'; unit = '(MJy/sr)^2'
    name = 'BKGD_VAR_RNOISE'; format = 'D'; unit = '(MJy/sr)^2'
    name = 'BKGD_VAR_FLAT'; format = 'D'; unit = '(MJy/sr)^2'
    name = 'NPIXELS'; format = 'D'
)

[havok2063]

Oh yeah it's definitely new, https://jwst-pipeline.readthedocs.io/en/latest/jwst/data_products/science_products.html#x1d. This change might even be the case for all x1d data products, not just those for NIRISS.

[pllim]

The fix looks simple though. cc @rosteen

https://github.com/astropy/specutils/blob/3a86739d53663b528ee751eb55d01de3c252022d/specutils/io/default_loaders/jwst_reader.py#L164

Change to

uncertainty = StdDevUncertainty(data["FLUX_ERROR"])

[jdavies-st]

Yes, x1d files have a new format for jwst>=1.2.0. And this is true for all of them. Here's the schema

https://github.com/spacetelescope/jwst/blob/e79b9f16a22cd36f1921a8810c8ad02b114b2920/jwst/datamodels/schemas/spectable.schema.yaml#L8-L43

[rosteen]

I tried making the simple change @pllim recommended above and immediately ran into a problem with jw00787-o014_s00015_niriss_f150w-gr150c-gr150r_c1d.fits from @havok2063's repository of 1.3.1 files. The file has SRCTYPE = 'POINT' and thus the reader looks for the 'FLUX_ERROR' key, but it only has 'ERROR' and 'SB_ERROR'. Any idea @jdavies-st @havok2063 if there's a problem with this file (it should have FLUX_ERROR instead of ERROR, or should be SRCTYPE='EXTENDED' and use SB_ERROR) or is this actually a different case we have to handle in the specutils readers?

Edit: @ibusko is already aware of this and working on it.

[havok2063]

The table columns in the c1d science extensions are not being changed. This change only affects the EXTRACT1D extensions in all x1d files. The c1d COMBINE1D extensions are remaining unchanged. As far as I know. See https://jwst-pipeline.readthedocs.io/en/latest/jwst/data_products/science_products.html#combined-1-d-spectroscopic-data-c1d