Closed stscijgbot-rstdms closed 3 years ago
Comment by David Davis on JIRA:
We should also take another look at level1.schema.yaml. It contains "Zeroframe array" which I don't think is being down linked and "Reference Output" which is for the IRS2 readout pattern for JWST NIRSpec. I don't think we need these in our schema.
Comment by Tyler Desjardins on JIRA:
Following up on David Davis's comment, I also think the dimensionality of the data array should be 3D (it's 4D in the schema file).
Zeropoint is slated for removal in RCAL-99: https://jira.stsci.edu/browse/RCAL-99
I would be happy to add these other items, as well. Please advise.
Comment by David Grumm on JIRA:
To establish the basic core metadata that should be included in a Level 1 science exposure file, I've reviewed the following documents and files :
"Roman Metadata - Level 1" :
https://innerspace.stsci.edu/pages/viewpage.action?spaceKey=SCSB&title=Roman+Metadata+-+Level1
"Core schema: <https"://github.com/spacetelescope/romancal/blob/main/romancal/datamodels/schemas/core.schema.yaml>
"Metadata for Processing" : https://outerspace.stsci.edu/pages/viewpage.action?spaceKey=RSEC&title=Metadata+for+Processing
JWST readthedocs documentation for steps that are in the Roman calibration pipeline
Various Roman github PRs : 39, 75, 106
Tyler Desjardins' Level 1 files in ████████████████████
https://innerspace.stsci.edu/display/ROMAN/Science+Data+Formatting+Discovery
With RCAL-75, all of the Level 1 Metadata is in the basic core schema.
I've compared the metadata specified in "Roman Metadata - Level 1" with the metadata already appearing in the core schema and noted the following inconsistencies or deficiencies.
"Roman Metadata-Level 1" : ephemeris.reference_frame
core schema file: undefined
"Roman Metadata-Level 1" : wcsinfo.v3yang core schema file: wcsinfo.v3yangle (typo ?)
"Roman Metadata-Level 1" : exposure.frame_divisor core schema file: exposure.frame_divissor (which I assume is a typo)
"Roman Metadata-Level 1" : 3 cases of'WFI/packet ; "This needs metadata location/name" core schema file: Undefined
"Roman Metadata-Level 1" : guidestar.acq_exec_stat core schema file: guidestar.gs_acq_execstat (additional 'acq' )
"Roman Metadata-Level 1" : guidestar.ctd_x core schema file: guidestar.gs_ctd_x
"Roman Metadata-Level 1" :guidestar.ctd_y core schema file: guidestar.gs_ctd_y
"Roman Metadata-Level 1" :guidestar.mudec core schema file: guidestar.gs_mudec
For each step in the Roman pipeline, I also reviewed the JWST readthedocs documentation to see what Level 1 metadata CRDS needs to retrieve for the step's execution. As the CRDS selection rules and documentation for JWST may change, the CRDS selection rules described below for Roman may also change.
Here are all of the steps that will be in the Roman pipeline, and metadata required by CRDS:
STEP: Initialize data quality
CRDS looks for these to select the appropriate "MASK" ref file :
INSTRUME, DETECTOR, DATE-OBS, TIME-OBS
STEP: Perform saturation check CRDS looks for these to select the appropriate "SATURATION" ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS
STEP: Perform IPC correction CRDS looks for these to select the appropriate "IPC" ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS
STEP: Perform superbias correction CRDS looks for these to select the appropriate "SUPERBIAS" ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS
STEP: Perform reference pixel correction No reference files, no CRDS selectors required.
STEP: Perform non-linearity correction CRDS looks for these to select the appropriate "LINEARITY" ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS
STEP: Perform dark correction CRDS looks for these to select the appropriate DARK ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS
STEP: Perform persistence correction CRDS looks for these to select the appropriate "TRAPDENSITY" ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS
STEP: Perform jump detection CRDS looks for these to select the appropriate GAIN and READNOISE ref files : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS
STEP: Perform ramp fit CRDS looks for these to select the appropriate GAIN AND READNOISE ref files : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS
STEP: Perform gain scaling CRDS looks for these to select the appropriate GAIN ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS
STEP: Assign WCS information In the Level 1 metadata, the observing mode should be given by EXP_TYPE, and for Roman should be NRC_IMAGE. The type of ref file that CRDS will try to retrieve will be given by REFTYPE, which can either distortion, or filteroffset.
STEP: Perform flat-fielding ROMAN
CRDS looks for these to select the appropriate FLAT ref file
INSTRUME, DETECTOR, DATE-OBS, TIME-OBS
STEP: Perform photometric calibration This step requires PHOTOM and AREA reference files. CRDS looks for these to select the appropriate PHOTOM ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS
CRDS looks for these to select the appropriate AREA ref file : INSTRUME, DETECTOR, DATE-OBS, and TIME-OBS.
Comment by Tyler Desjardins on JIRA:
Just a word of caution that the JWST pipeline steps and CRDS matching rules are a good starting point, but they are not necessarily correct for Roman.
Comment by Erica Kolatch on JIRA:
Megan Sosey should this information be captured in the design document somewhere or is it already there? (with awareness for Tyler's caveat)
Comment by Megan Sosey on JIRA:
Erica Kolatch no. what we already have in the design document is sufficient. This information is captured in romancal.
Comment by Megan Sosey on JIRA:
David Grumm willl you also have a look at the following page? https://innerspace.stsci.edu/display/ROMAN/Science+Data+Formatting+Discovery](https://innerspace.stsci.edu/display/ROMAN/Science+Data+Formatting+Discovery,)
Yes, I've looked at that. I will include it in my list.
More generally:
did we decide about whether we need array gain or single value gain?
I don't know of any specific discussion regarding areal vs scalar gain values for Roman; JWST ramp fitting uses the gain values in the pixel-specific gain reference file, but JWST gain scaling uses the single value from the 'GAINFACT' keyword.
we've discussed area reference file being something we can add to crds for retrieval as needed, I'm keeping the extra array for bookkeeping cause I think we may need another error array-> we'll discuss at the meeting I scheduled with karl
CRDS looks for these to select the appropriate AREA ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS, FILTER, and PUPIL.
let's make sure we aren't putting in JWST specific information like PUPIL and FILTER
RIght: 'PUPIL' & 'FILTER' should not included for Roman - I will remove them. Should 'OPTICAL_ELEMENT' replace 'FILTER' ?
we should revisit the guidestar centroid items, these wont go into the level 1 science files
STEP: Perform background subtraction CRDS looks for these to select the appropriate WFSS ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS . This assumes that WFI has, as JWST's NIRCam, a Wide-Field Slitless Spectroscopy (WFSS) mode.
I dont believe we will have a background subtraction step, Roman doesn't have background exposures. There is no WFSS mode per say, there are a grism and prism element, we are processing them as we would imaging mode.
Okay, since there will be no WFSS mode, I will remove the background subtraction section here.
Tyler Desjardins what's the status/need of having trapdensity information for persistence correction from current testing?
STEP: Perform persistence correction CRDS looks for these to select the appropriate "TRAPDENSITY" ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS
Comment by David Grumm on JIRA:
Reopened (then will reclose) this ticket as I had just made some edits (in response to Megan) that may not have been noticed otherwise.
Comment by Megan Sosey on JIRA:
David Grumm I dont think imaging mode will have background subtraction either, Tyler Desjardins can correct me if this is wrong (i.e. Roman isn't scheduling background image specific observations with programs)
Yes, optical_element should replace filter
Issue RCAL-55 was created on JIRA by Megan Sosey:
Establish the basic core metadata that should be included in a Level 1 science exposure file. This is data that will eventually be created by SDF and ready for input into the science calibration pipeline. For now, INS is making an example file for testing purposes.
It will likely include things such as the aperture, pointing information sufficient to construct a basic wcs, and should not include metadata that are fits specific.
A core schema file has been started, work with INS to make sure it has included the information needed for reading in a roman, level 1, science file. Focus on the basic information needed to read level 1 files and not everything that may eventually be included in the datamodel.