Filetype / Value error during Stage 2 Background Subtraction

[mm-murphy]

Note that this issue is connected to the JOSS review at https://github.com/openjournals/joss-reviews/issues/6898.

Hi, I'm working through Stage 2 reduction in the NIRISS SOSS tutorial notebook. I run into the following error during the background subtraction step:

2024-07-11 10:27:35.490 - exoTEDRF - INFO - Starting background subtraction step.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[9], line 10
      3 step = stage2.BackgroundStep(results, background_model=background_model, baseline_ints=[150, -100], output_dir=outdir_s2)
      5 # Using all the same parameters from the group-level background subtraction again here,
      6 # including the differential scaling.
      7 # We don't need to re-add the background to the data later, so we only need to return the
      8 # corrected data.
---> 10 results = step.run(save_results=True, differential=True, force_redo=True, do_plot=False)[0]

File ~/miniconda3/envs/exotedrf/lib/python3.10/site-packages/exotedrf/stage2.py:161, in BackgroundStep.run(self, save_results, force_redo, do_plot, show_plot, **kwargs)
    159 with warnings.catch_warnings():
    160     warnings.filterwarnings('ignore')
--> 161     step_results = backgroundstep(self.datafiles,
    162                                   self.background_model,
    163                                   baseline_ints=self.baseline_ints,
    164                                   output_dir=self.output_dir,
    165                                   save_results=save_results,
    166                                   fileroots=self.fileroots,
    167                                   fileroot_noseg=self.fileroot_noseg,
    168                                   **kwargs)
    169     results, background_models = step_results
    171 # Do step plot if requested.

File ~/miniconda3/envs/exotedrf/lib/python3.10/site-packages/exotedrf/stage2.py:405, in backgroundstep(datafiles, background_model, baseline_ints, output_dir, save_results, fileroots, fileroot_noseg, scale1, background_coords1, scale2, background_coords2, differential)
    403 # Load in each of the datafiles.
    404 for i, file in enumerate(datafiles):
--> 405     with utils.open_filetype(file) as currentfile:
    406         # To create the deepstack, join all segments together.
    407         if i == 0:
    408             cube = currentfile.data

File ~/miniconda3/envs/exotedrf/lib/python3.10/site-packages/exotedrf/utils.py:698, in open_filetype(datafile)
    696     data = datafile
    697 else:
--> 698     raise ValueError('Invalid filetype: {}'.format(type(datafile)))
    700 return data

ValueError: Invalid filetype: <class 'stdatamodels.jwst.datamodels.image.ImageModel'>

The results that I pass to the background step run method is

[<ImageModel(256, 2048) from jw01366001001_04101_00001-seg001_nis_flatfieldstep.fits>,
 <ImageModel(256, 2048) from jw01366001001_04101_00001-seg002_nis_flatfieldstep.fits>,
 <ImageModel(256, 2048) from jw01366001001_04101_00001-seg003_nis_flatfieldstep.fits>,
 <ImageModel(256, 2048) from jw01366001001_04101_00001-seg004_nis_flatfieldstep.fits>]

which was output from the flat-fielding step. The background model I pass it is the background_model = np.load('model_background256.npy') downloaded from STScI. The preceding Stage 2 steps (Assign WCS, Source Type Determination, Flat Fielding) completed without issue, and I just copy+pasted+ran them as-is in the documentation. The only potential difference I can think of is that, because I'm currently on a laptop with "low" RAM, I'm beginning from stage 2 using rate.fits files rather than redoing Stage 1 in the same run.

Do you have an idea of what's happening here? Am I missing something, perhaps cached from Stage 1, that needs to be included in the results list being passed to this step? Or anything else?

[radicamc]

Ah yes, I believe I know what's going on here. Can you confirm that when you say you're using "rate.fits" files that you're using the Stage 1 products downloaded from MAST, or something equivalent? If so, then yes, this is what's causing the problem. The rate.fits files are averages over all integrations (so its a 2D -- x pixel, y pixel -- file). The step is expecting a TSO (so 3D -- integrations, x pixel, y pixel -- file). The "rateints.fits" files are the ones you would want to use in this case then (again assuming you're getting the Stage 1 products from MAST).

[mm-murphy]

Ah, my mistake! Yes, you're right, and thank you for explaining. When I instead start from the rateints files everything runs smoothly.