Error when using Quickstart tutorial

[catrionamurray]

I got this error with chromatic (v0.4.2 and v0.4.3) when trying to run the Quickstart tutorial.

rainbow = read_rainbow("jwst_data/jw02734002001_04101_00001-seg00*_nis_x1dints.fits")

🌈🤖 This file contains data for 3 spectrosopic orders. Because no order= keyword was supplied, we're defaulting to first order. You can hide this warning by expliciting stating which order you want to load. For this file, the options include [1 2 3].

🌈🤖 No wavelength unit was found; assuming 'micron'.

🌈🤖 No int_times extension was found in the first file jw02734002001_04101_00001-seg001_nis_x1dints.fits

🌈🤖 Times were set by linearly interpolating between the exposure start and end points, as estimated from the 'SCI' extension using the 'TDB-BEG', 'TDB-END', and 'EFFINTTM' keywords. Times may be off by a few seconds and possibly up to the duration of one integration (= 76.916s).

---

AssertionError Traceback (most recent call last) Input In [7], in <cell line: 1>() ----> 1 rainbow = read_rainbow("jwst_data/jw02734002001_04101_00001-seg00*_nis_x1dints.fits")

File ~/opt/anaconda3/lib/python3.9/site-packages/chromatic/rainbows/init.py:29, in read_rainbow(filepath, kw) 8 def read_rainbow(filepath, kw): 9 """ 10 A friendly wrapper to load time-series spectra and/or 11 multiwavelength light curves into a chromatic Rainbow (...) 27 The loaded data! 28 """ ---> 29 r = Rainbow(filepath, kw) 30 if "model" in r.fluxlike: 31 return RainbowWithModel(r._get_core_dictionaries())

File ~/opt/anaconda3/lib/python3.9/site-packages/chromatic/rainbows/rainbow.py:227, in Rainbow.init(self, filepath, format, wavelength, time, flux, uncertainty, wavelike, timelike, fluxlike, metadata, name, kw) 225 # then try to initialize from a file 226 elif isinstance(filepath, str) or isinstance(filepath, list): --> 227 self._initialize_from_file(filepath=filepath, format=format, kw) 229 # finally, tidy up by guessing the scales 230 self._guess_wscale()

File ~/opt/anaconda3/lib/python3.9/site-packages/chromatic/rainbows/rainbow.py:463, in Rainbow._initialize_from_file(self, filepath, format, kw) 461 # pick the appropriate reader 462 reader = guess_reader(filepath=filepath, format=format) --> 463 reader(self, filepath, kw) 465 # validate that something reasonable got populated 466 self._validate_core_dictionaries()

File ~/opt/anaconda3/lib/python3.9/site-packages/chromatic/rainbows/readers/x1dints.py:254, in from_x1dints(rainbow, filepath, order, *kw) 248 N_expected_integration_extensions = ( 249 n_integrations_in_this_segment n_orders 250 ) 252 if pipeline_stage == 2: 253 # so far, all real Stage 2 files seem to behave normally --> 254 assert N_expected_integration_extensions == N_integration_extensions 255 elif pipeline_stage == 3: 256 # some Stage 3 files seem to have errors in the INTSTART + INTEND keywords 257 if N_expected_integration_extensions != N_integration_extensions:

AssertionError:

[catrionamurray]

In case it helps it looks like it's an issue with jwst_data/jw02734002001_04101_00001-seg003_nis_x1dints.fits, where N_integration_extensions = 239 and N_expected_integration_extensions = 240

[catrionamurray]

It might also be an issue with this file, it's weird there's lots of NaNs in the flux and the pcolormesh looks like this:

[zkbt]

@Pat-Wachiraphan found that WASP-96b x1dints with calibration version 1.5.3.

[zkbt]

@catrionamurray finds it works on 1.5.3 but not on 1.7.2 (both for WASP-96 NIRISS).

[zkbt]

@Pat-Wachiraphan also sees it working with version 1.6.2 on MIRI.

[zkbt]

Let's:

• [ ] Get x1dints reader working with the version of WASP-96 and HAT-P-18 that you download from MAST right now (and/or ERS datasets).
• [ ] Keep track of what version works where. Possibly introduce different formats, and choose between them gracefully.
• [ ] Add a download script to get data we know will work for the Quickstart. Download from MAST, but have link to static versions.

[catrionamurray]

I've tracked down the first assertion error: the 3rd x1dints file doesn't have an INT_TIMES in the hdu, whereas the first 2 do. In the previous version I have (1.5.3) none of the files had INT_TIMES (improvement yay!). This means that when you just do: non_spectrum_extensions = [ x for x in ["PRIMARY", "SCI", "INT_TIMES", "ASDF"] if x in hdu ] for the first file (if i_file == 0) then it's wrong (non_spectrum_extensions=4) for the 3rd file. If I just load in the 3rd file then it gets populated correctly (non_spectrum_extensions=3).

This also means that get_times_from_x1dints_files fails even though we do have INT_TIMES in two of the three files...

[catrionamurray]

I think there wouldn't be harm in taking those couple of lines in x1dints.py out of the if statement

[catrionamurray]

Ah another discovery, the third (weird) file is v1.5.3 whereas the first two are 1.7.2! This might be why we now get a disjointed LC with much more noise in the last third:

[catrionamurray]

Fix to get to this point without crashes - and throw an error/warning message when the files have different calibration versions here https://github.com/zkbt/chromatic/pull/209