Updates to PHOTOM step and files for MIRI imager time-dependent throughput

[stscijgbot-jp]

Issue JP-3455 was created on JIRA by Misty Cracraft:

A time-dependent throughput issue was found for the MIRI Imager, as had previously been found for the MIRI MRS. As with the fix for the MIRI MRS, we now have updated code that can go into the pipeline to use updated photom files with a correction based on MJD of the observation taken from the header and will not rely on a set of USEAFTER date specified photom files like we currently have in place for the workaround.

Mattia Libralato  has developed the code and created the appropriate photom reference file for this.

 

[stscijgbot-jp]

Comment by Alicia Canipe on JIRA:

Reassigned to Anton Koekemoer / Greg Sloan for CalWG tracking 

[stscijgbot-jp]

Comment by Mattia Libralato on JIRA:

The updated code computes the conversion factor from DN/S to MJy/sr in the PHOTOM step using an exponential function that requires in input the MID-MJD of the image. The parameters of the exponential function are loaded from the new PHOTOM reference file. The proposed change in the code will allow users to still use the old version of the PHOTOM reference files if necessary.

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

1) Will this code be part of Build 10.1?

  1.1) New reference files have to be delivered to CRDS-TEST when the new code is merged so it can be tested.

2) Do the format of the Photom files changed such that the new files will not work with the current pipeline?

  2.1) If there is a dependency with the code then we need to make sure these are not delivered to OPS before the code

  2.2.) If the selection criteria for these files change we need to file a CRDS ticket to make that change in CRDS-TEST for the purpose of testing.

[stscijgbot-jp]

Comment by Mattia Libralato on JIRA:

1) That is the plan

   1.1) Will do

2) Yes, the new PHOTOM reference file will not work with the current version of the pipeline. The new code will allow to use the new and old PHOTOM reference files

   2.1) OK

   2.2) OK

[stscijgbot-jp]

Comment by Meaghan McDonald on JIRA:

Hi Misty Cracraft. Did you want to make a CRDS ticket for this, so we can prepare to get the new files into TEST for the build 10.1 deadline that is approaching?

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

I'm not fully aware of what changes are being made in the format of the files, since Mattia has made the changes able to handle both old and new formats, or if there are changes to the selection criteria, but I do think there should be a CRDS ticket. I'm just not quite sure yet what to put in it. I think setting up the ticket and then working with Mattia Libralato to fill out the details would be good.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

After some discussion with Mattia Libralato  amd Misty Cracraft , this has been scheduled for a brief presentation in JWST CalWG Meeting 2023-11-07

[stscijgbot-jp]

Comment by Meaghan McDonald on JIRA:

Hello checking in again Mattia Libralato Misty Cracraft. Have there been any updates following the meeting? Are we able to create a CRDS ticket for these files? We would like to get files for build 10.1 in by this Friday for testing, but the official deadline is next Friday, Nov 17th. 

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

Yes, I think we can set up the CRDS ticket, and Mattia will update the ticket to describe the differences between the format of the old and new photom files. He'll attach a sample of the file to the CRDS ticket. From my understanding, the table of values in the photom file will have extra columns that the old files don't have.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

Unless the testing described is only regarding ingest/mapping of the files, we still need a PR to review and merge the code updates as well.

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

Mattia is working on the code now, and expects a PR this week to submit the code.

[stscijgbot-jp]

Comment by Meaghan McDonald on JIRA:

I have created CRDS-750 to document these updates on the ReDCaT side of things. Mattia Libralato please update that ticket with details at your convenience. 

[stscijgbot-jp]

Comment by Mattia Libralato on JIRA:

Two PRs have been submitted: #8064 and spacetelescope/stdatamodels#235 .

[stscijgbot-jp]

Comment by Mattia Libralato on JIRA:

Does anyone has any update about the status of the ticket?

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

thanks Mattia, tagging the SCSB side (Tyler Pauly  Howard Bushouse) to get their status on the PRs you submitted or whether they need anything further from you for this

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

PR is queued for review. The last couple weeks have been impacted by ADASS and JWST Data Products workshop.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Updates to MIRI imager photom data model included in spacetelescope/stdatamodels#235

[stscijgbot-jp]

Comment by Mattia Libralato on JIRA:

PR #8064 has been closed for issues with git. The new PR is https://github.com/spacetelescope/jwst/pull/8096](https://github.com/spacetelescope/jwst/pull/8096)

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed by #8096

[stscijgbot-jp]

Comment by Mattia Libralato on JIRA:

Howard Bushouse the pipeline is not ready for testing because the stdatamodel PR has not been included, at least this is what two of us got when we installed the pipeline. The PHOTOM step has changed.

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

To clarify what we saw with testing, no matter whether we were using the older reference file in OPS or the new, updated reference file currently in CRDS-TEST, we got the same message:

2023-12-01 09:52:20,321 - stpipe.Image2Pipeline.photom - INFO -  Skipping MIRI imager time correction. Extension not found in the reference file.

Looking into the photom datamodel delivered with this version of the code, we see this:

class MirImgPhotomModel(ReferenceFileModel):     “”"     A data model for MIRI imaging photom reference files.    Parameters     ____     phot_table : numpy table         Photometric flux conversion factors table         A table-like object containing row selection criteria made up         of instrument mode parameters and photometric conversion         factors associated with those modes.       - filter: str[12]        - subarray: str[15]        - photmjsr: float32        - uncertainty: float32    “”"     schema_url = “[http://stsci.edu/schemas/jwst_datamodel/mirimg_photom.schema]”

There is no reference to the timecoeff extension, so we can't test this further until the datamodel is updated.

 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Misty Cracraft Can you try installing the latest stdatamodels/master branch from github into your environment? That would include the updated data model. You can do so using:

pip install git+https://github.com/spacetelescope/stdatamodels ```
Or you can wait till we make a new release of stdatamodels.

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

Thank you! That did it. I now see it being applied. Testing ongoing to see if the results are as they should be.

2023-12-01 13:22:52,490 - stpipe.Image2Pipeline.photom - INFO - Applying the time-dependent correction to the PHOTOM value.

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

I ran several sets of data through the new pipeline, both against CRDS OPS, which uses the previous set of photom files broken out by USEAFTER date and against CRDS-TEST, which used the new format of the photom reference file with the fitting coefficients in a new extension. The code ran fine with no errors or warnings, whichever version of CRDS was used, so it will work fine with both old and new reference files.

When testing against the old files, I see this message: INFO -  Skipping MIRI imager time correction. Extension not found in the reference file.

When I test against the new file, I see this message: INFO - Applying the time-dependent correction to the PHOTOM value.

This is as expected. Both tests pulled the proper reference file from CRDS: with the old file, it pulled the proper file based on date, and in TEST, it properly used the new reference file.

For multiple different datasets, we compared the PHOTOM* keyword values in the old and new methods to ensure that the output values are close.

For a set of F2100W data with an OBSDATE of 2023-09-16 (recent data), we see the following comparison:

For the file using OPS (multiple photom files)        

Photometry information

PHOTMJSR=   0.4428722858428955 / Flux density (MJy/steradian) producing 1 cps PHOTUJA2=   10.409460880285843 / Flux density (uJy/arcsec2) producing 1 cps

 

For the new file:        

Photometry information

PHOTMJSR=  0.44085609912872314 / Flux density (MJy/steradian) producing 1 cps PHOTUJA2=   10.362071559708726 / Flux density (uJy/arcsec2) producing 1 cps

 

For data from May of 2022 (early data where the correction is larger) for two filters we see this:

F770W, PID 1040 results:

 

Using OPS files (multiple photom files)

        Photometry information                                                                                                                         

PHOTMJSR=  0.25879502296447754 / Flux density (MJy/steradian) producing 1 cps  

PHOTUJA2=    6.082829641132806 / Flux density (uJy/arcsec2) producing 1 cps    

Using new file in CRDS-TEST

        Photometry information                                                 

PHOTMJSR=  0.25890105962753296 / Flux density (MJy/steradian) producing 1 cps  

PHOTUJA2=     6.08532197251419 / Flux density (uJy/arcsec2) producing 1 cps    

 

F2100W, PID 1040 results:

 

Using OPS files (multiple photom files)

       Photometry information                                                                                                                                

PHOTMJSR=   0.3923032581806183 / Flux density (MJy/steradian) producing 1 cps  

PHOTUJA2=    9.220864682168129 / Flux density (uJy/arcsec2) producing 1 cps    

 

Using the new file in CRDS-TEST

        Photometry information                                                                                                                                

PHOTMJSR=    0.390969455242157 / Flux density (MJy/steradian) producing 1 cps  

PHOTUJA2=     9.18951440364822 / Flux density (uJy/arcsec2) producing 1 cps    

 

When blinking between the images created with the two sets of reference files, only small differences (on order of ~1 MJy/st at the longer wavelength) are seen.

 

Mattia and I are satisfied with the results of this testing, and ready to close the ticket. This step passes the MIRI team's testing.

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

Passed testing

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Misty Cracraft Woot!