search

spacetelescope / romancal

Python library to process science observations from the Nancy Grace Roman Space Telescope
https://roman-pipeline.readthedocs.io/en/latest/
Other
32 stars 28 forks source link

Establish core metadata for level 1 science file #39

Closed stscijgbot-rstdms closed 3 years ago

stscijgbot-rstdms commented 3 years ago

Issue RCAL-55 was created on JIRA by Megan Sosey:

Establish the basic core metadata that should be included in a Level 1 science exposure file. This is data that will eventually be created by SDF and ready for input into the science calibration pipeline. For now, INS is making an example file for testing purposes.

It will likely include things such as the aperture, pointing information sufficient to construct a basic wcs, and should not include metadata that are fits specific.

A core schema file has been started, work with INS to make sure it has included the information needed for reading in a roman, level 1, science file. Focus on the basic information needed to read level 1 files and not everything that may eventually be included in the datamodel. 

stscijgbot-rstdms commented 3 years ago

Comment by David Davis on JIRA:

We should also take another look at level1.schema.yaml. It contains "Zeroframe array" which I don't think is being down linked and "Reference Output" which is for the IRS2 readout pattern for JWST NIRSpec. I don't think we need these in our schema.

stscijgbot-rstdms commented 3 years ago

Comment by Tyler Desjardins on JIRA:

Following up on David Davis's comment, I also think the dimensionality of the data array should be 3D (it's 4D in the schema file).

stscijgbot-rstdms commented 3 years ago

Comment by Paul Huwe on JIRA:

Zeropoint is slated for removal in RCAL-99: https://jira.stsci.edu/browse/RCAL-99

 

I would be happy to add these other items, as well. Please advise.

stscijgbot-rstdms commented 3 years ago

Comment by David Grumm on JIRA:

To establish the basic core metadata that should be included in a Level 1 science exposure file, I've reviewed the following documents and files :

With RCAL-75, all of the Level 1 Metadata is in the basic core schema.

I've compared the metadata specified in "Roman Metadata - Level 1" with the metadata already appearing in the core schema and noted the following inconsistencies or deficiencies.

"Roman Metadata-Level 1" : ephemeris.reference_frame
core schema file: undefined

"Roman Metadata-Level 1" : wcsinfo.v3yang core schema file: wcsinfo.v3yangle (typo ?)

"Roman Metadata-Level 1" : exposure.frame_divisor core schema file: exposure.frame_divissor (which I assume is a typo)

"Roman Metadata-Level 1" : 3 cases of'WFI/packet ; "This needs metadata location/name" core schema file: Undefined

"Roman Metadata-Level 1" : guidestar.acq_exec_stat core schema file: guidestar.gs_acq_execstat (additional 'acq' )

"Roman Metadata-Level 1" : guidestar.ctd_x core schema file: guidestar.gs_ctd_x

"Roman Metadata-Level 1" :guidestar.ctd_y core schema file: guidestar.gs_ctd_y

"Roman Metadata-Level 1" :guidestar.mudec core schema file: guidestar.gs_mudec

For each step in the Roman pipeline, I also reviewed the JWST readthedocs documentation to see what Level 1 metadata CRDS needs to retrieve for the step's execution. As the CRDS selection rules and documentation for JWST may change, the CRDS selection rules described below for Roman may also change.

Here are all of the steps that will be in the Roman pipeline, and metadata required by CRDS:

STEP: Initialize data quality
CRDS looks for these to select the appropriate "MASK" ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS

STEP: Perform saturation check CRDS looks for these to select the appropriate "SATURATION" ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS

STEP: Perform IPC correction CRDS looks for these to select the appropriate "IPC" ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS

STEP: Perform superbias correction CRDS looks for these to select the appropriate "SUPERBIAS" ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS

STEP: Perform reference pixel correction No reference files, no CRDS selectors required.

STEP: Perform non-linearity correction CRDS looks for these to select the appropriate "LINEARITY" ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS

STEP: Perform dark correction CRDS looks for these to select the appropriate DARK ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS

STEP: Perform persistence correction CRDS looks for these to select the appropriate "TRAPDENSITY" ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS

STEP: Perform jump detection CRDS looks for these to select the appropriate GAIN and READNOISE ref files : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS

STEP: Perform ramp fit CRDS looks for these to select the appropriate GAIN AND READNOISE ref files : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS

STEP: Perform gain scaling CRDS looks for these to select the appropriate GAIN ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS

STEP: Assign WCS information In the Level 1 metadata, the observing mode should be given by EXP_TYPE, and for Roman should be NRC_IMAGE. The type of ref file that CRDS will try to retrieve will be given by REFTYPE, which can either distortion, or filteroffset.

STEP: Perform flat-fielding ROMAN CRDS looks for these to select the appropriate FLAT ref file
INSTRUME, DETECTOR, DATE-OBS, TIME-OBS

STEP: Perform photometric calibration This step requires PHOTOM and AREA reference files. CRDS looks for these to select the appropriate PHOTOM ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS

CRDS looks for these to select the appropriate AREA ref file : INSTRUME, DETECTOR, DATE-OBS, and TIME-OBS.

stscijgbot-rstdms commented 3 years ago

Comment by Tyler Desjardins on JIRA:

Just a word of caution that the JWST pipeline steps and CRDS matching rules are a good starting point, but they are not necessarily correct for Roman.

stscijgbot-rstdms commented 3 years ago

Comment by Erica Kolatch on JIRA:

Megan Sosey should this information be captured in the design document somewhere or is it already there? (with awareness for Tyler's caveat)

stscijgbot-rstdms commented 3 years ago

Comment by Megan Sosey on JIRA:

Erica Kolatch no. what we already have in the design document is sufficient. This information is captured in romancal.

stscijgbot-rstdms commented 3 years ago

Comment by Megan Sosey on JIRA:

David Grumm willl you also have a look at the following page? https://innerspace.stsci.edu/display/ROMAN/Science+Data+Formatting+Discovery](https://innerspace.stsci.edu/display/ROMAN/Science+Data+Formatting+Discovery,)

Yes, I've looked at that. I will include it in my list.

 

More generally:

did we decide about whether we need array gain or single value gain?

I don't know of any specific discussion regarding areal vs scalar gain values for Roman; JWST ramp fitting uses the gain values in the pixel-specific gain reference file, but JWST gain scaling uses the single value from the 'GAINFACT' keyword.

we've discussed area reference file being something we can add to crds for retrieval as needed, I'm keeping the extra array for bookkeeping cause I think we may need another error array-> we'll discuss at the meeting I scheduled with karl

CRDS looks for these to select the appropriate AREA ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS, FILTER, and PUPIL.

let's make sure we aren't putting in JWST specific information like PUPIL and FILTER

RIght: 'PUPIL' & 'FILTER' should not included for Roman - I will remove them. Should 'OPTICAL_ELEMENT' replace 'FILTER' ?  

we should revisit the guidestar centroid items, these wont go into the level 1 science files

STEP: Perform background subtraction CRDS looks for these to select the appropriate WFSS ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS . This assumes that WFI has, as JWST's NIRCam, a Wide-Field Slitless Spectroscopy (WFSS) mode.

I dont believe we will have a background subtraction step, Roman doesn't have background exposures. There is no WFSS mode per say, there are a grism and prism element, we are processing them as we would imaging mode.

Okay, since there will be no WFSS mode, I will remove the background subtraction section here.

 

Tyler Desjardins what's the status/need of having trapdensity information for persistence correction from current testing?

STEP: Perform persistence correction CRDS looks for these to select the appropriate "TRAPDENSITY" ref file : INSTRUME, DETECTOR, DATE-OBS, TIME-OBS

 

 

stscijgbot-rstdms commented 3 years ago

Comment by David Grumm on JIRA:

Reopened (then will reclose) this ticket as I had just made some edits (in response to Megan) that may not have been noticed otherwise.

stscijgbot-rstdms commented 3 years ago

Comment by Megan Sosey on JIRA:

David Grumm I dont think imaging mode will have background subtraction either, Tyler Desjardins can correct me if this is wrong (i.e. Roman isn't scheduling background image specific observations with programs)

 

Yes, optical_element should replace filter