SRCTYPE change for NIRCam and NIRISS data

[stscijgbot-jp]

Issue JP-3274 was created on JIRA by Rosa Diaz:

Found a change in SRCTYPE between b9.2 and b9.3_cal.1.10.1x_earlyResultsforINS data

The JP ticket 3098 mentions that this applies only to MIRI and NIRSpec, and possible NIC_TSOGRISM. However, I see changes in NIRISS WFSS data and NRC_WFSS, which is not indicated in the ticket or in the file https://github.com/spacetelescope/jwst/blob/master/CHANGES.rst

In ther CHANGES file it states "The SRCTYAPT takes precedence over PATTTYPE when setting the source type for MIR_LRS-FIXEDSLIT, MIR_LRS-SLITLESS, 'MIR_MRS', NRC_TSGRISM, NRS_FIXEDSLIT, NRS_BRIGHTOBJ, NRS_IFU. [#⁠7583]"

I didn't find any changes in the data mentioned above but might be that there should not be.

 

e.g. data

jw02736003001_0210a_00008_x1d.fits NIRISS

jw01076101001_02101_00002_nrcblong_x1d.fits NIRCAM

 

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

I did not find file jw02736003001_0210a_00008 in the archive for NIRISS.  I searched for any program file with '_0210a_00008' and did not find any matches in the area on disk where I look at NIRISS data.  I also am a bit puzzled by a "SRCTYAPT" flag for WFSS, since any field we look at will have a mixture of stars and extended objects.  I am therefore not sure what happened here.  If you could re-check the file name I would appreciate it.

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

HI Kevin,

 Those are the names I have and in the regression test run and are valid. However, after what you mentioned I went ahead and took a look and I think that the reason for which you didn't find them and maybe also the reason for my findings is because these are WFSS datasets. Depending on changes to the pipeline I think it might find more or less sources, and it might be that now it finds more than before.

Also, I believe that the difference between the POINT and EXTENDED is that when re-creating the WFSS products it does not keep the same order for the sources or can swap order of the sources, making it look like there is a difference when there is not. I am finding that these WFSS datasets are quite difficult to test because of that.

But it would not hurt to confirm if you can:

You can find these datasets here:

a: ██████████████████████████████████████████████████████████████████████████████  b: █████████████████████████████████████████████████████████████████████████████████████████

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

Thank you for verifying the file names, I will try to take a look at the data.  I also need to find out why I have not found the program with my MAST queries, but that is a separate issue.

 

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

I see the differences also in

jw02736003001_02102_00001_nis_x1d    jw02736003001_02102_00002_nis_cal

jw02736003001_02102_00003_nis_x1d

those are in MAST, though you might want to check also the files we have in central store. The ones in MAST are calibrated with CAL  version 1.9.6

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

I got the files: the issue was I was searching MAST for the last year and the program was executed 1 year and 10 days ago or so.  I am surprised that the one CAL file has changed, which implies a change from the jump detection, but I will see if I can find any reason for it.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

The source type value set in the APT has nothing to do with the SRCTYPE values in WFSS data sets. There's no way the user can specify a single source type value that would apply to the hundreds or thousands of sources present in WFSS observations (same as NIRSpec MOS). The SRCTYPE values in WFSS are derived from entries in the source catalog created from the direct images, which are based on empirical measurements of the data itself. Hence it's quite possible that slight changes in calibration would cause particular sources to cross the boundary (one way or the other) between what's considered point vs extended, resulting in a change of SRCTYPE for those particular sources.

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

I started looking at the files and I do not see the source catalogue .ecsv files for the NIRISS observations in either of these directories (b9.3_cal.1.10.1x_earlyResultsforINS/ and b9.2_cal.1.10.1_full/).  I am not sure why, since these are required for the pipeline to run.  It would be helpful to determine what the photometry values are since that is the basis of the source type.  But I can verify that in some cases the same source changed SRCTYPE from EXTENDED to POINT in the _x1d file.  It is the same 100 sources in the jw02736003001_02102_00001_nis_x1d.fits case in both reductions, but the order of sources is a bit different as are the reported source positions.  Hence as Howard suggested I think these are all due to small changes in the measurements due to small changes in the calibration files, the jump detection results, or other such factors.

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

As for the CAL file jw02736003001_02102_00002_nis_cal.fits, the dimensions of all the extensions are changed between the two pipeline versions, partially because of the POINT/EXTENDED source type which affects the size of the 2-D cut-out.  The very first source has a change from POINT to EXTENDED and the image size in the CAL file changes from [11, 897] to [768, 897].  Presumably these changes are triggering the comparison code to flag changes.  A quick look suggested that a fair number of the image sizes were changed even when the SRCTYPE flag was not changed.  This must have to do with the segmentation image boundaries between sources changing in the two pipeline versions.

 

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

I will asks for the ECSV files. I didn't realize we were missing them

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

the ECSV files and JSON files are now in b9.3_cal.1.11.0_rc1_ResultsforINS

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

I do agree that in this case it is because the order  changed triggering differences. Closing this ticket. Kevin Volk  thanks for checking