Flux differences in stage 3 products when combining multiple grating/filters

[stscijgbot-jp]

Issue JP-3652 was created on JIRA by Kayli Glidic:

A user submitted a Helpdesk question (INC0198692) regarding observed flux discrepancies in their stage 3 combined spectral products, particularly when processing all grating/filter combinations in one association file compared to processing each filter/grating separately. The data is from PID 1292 and has observations of a source in S200A1 and S200A2. The grating/filters used in the observations include G140H/F100LP, G140H/F070LP, and G235H/F170LP.

 

I replicated the issue and conducted additional testing on how data is combined in stage 3 processing. Attached are slides containing plots illustrating the observed flux differences resulting from various combinations of the data.

 

It appears that the flux discrepancy does not arise from combining S200A1 and S200A2 data. When processing S200A1 or S200A2 data independently for each grating/filter, the results are consistent, similar to when processing each filter/grating for S200A+S200A2 combined. The flux difference appears only when combing all grating/filters into one product. In particular, including the G235H/F170LP data with other configurations in the combination seems to be the primary cause of the larger flux differences.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Discussing with Christian Hayes - it sounds like this is happening specifically when multiple gratings are resampled together.  Extracting spectra separately and stitching together in the end works fine.

It may be related to the spectral flux conservation issues with resampling noted in JP-3547.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Kayli Glidic - do you have the input data for this issue somewhere I can grab it?

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Kayli Glidic - 

I found the data referenced in your slides in MAST and reduced it with standard spec2 and spec3 pipelines with the current dev pipeline, then also with a custom ASN file for spec3 combining all the gratings.  I found I needed to specify an override for the aperture correction in the combined case:

    --steps.extract_1d.override_apcorr="N/A"

but otherwise used default processing.

With this test, I can't reproduce the problem.  The output spectra from the individual processing and the combined processing overlap well. I do note that the peak flux in all my spectra, at about 1.3 um, is around 0.0008 Jy, similar to the lower value you get in the combined reductions.  In your plots, it looks like the peak value reducing separately is higher, around .001 Jy.

Were you using non-standard extraction parameters for your test?

[stscijgbot-jp]

Comment by Kayli Glidic on JIRA:

Hi Melanie Clarke , super sorry for the late reply, I was out for a good portion of the month. Here are the cells from my notebook with the parameters I set when running stage 2 and stage 3 on the files. 

# Stage 2 Processing
       
    spec2_params = {'bkg_subtract': {'skip': False}, 'resample_spec': {'fillval':'nan'}}
                    
    Spec2Pipeline.call(asn_file, save_results=True, output_dir=pixel_bkgsub_dir, steps=spec2_params) ```
```java
# Stage 3 Processing 
    
                    
    Spec3Pipeline.call(asn_file, save_results=True, output_dir=pixel_bkgsub_dir, steps={'extract_1d': {'apply_apcorr':False},
                                                                                       'resample_spec': {'fillval':'nan'}}) ```
I did set `apply_apcorr` to False in Stage 3 when I was working with the combined ones. This was all processed with pipeline version 1.14.0. I can try to reprocess the data with the newest version of the pipeline to see if the issue is still present.  

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Thanks Kayli Glidic - it would be helpful if you can re-run with the current dev version.  If you still see the effect, please send me your notebook and I can investigate from there.