Implement engineering metadata mode

[zonca]

FITS keyword: "INSTMODE" Value: "ENG", "SCI", "CAL"

[zonca]

@ikashell @arunsurya77

JWST has a keyword named EXP_TYPE, see its possible values:

https://jwst-pipeline.readthedocs.io/en/latest/jwst/assign_wcs/exp_types.html#wcs-reference-file-information-per-exp-type

for example for nircam: NRC_IMAGE, NRC_TSIMAGE, NRC_FOCUS, NRC_TACONFIRM, NRC_TACQ

Currently I am using IRIS_IMAGE for all files.

In the file naming document https://docs.google.com/document/d/1ppkyuNsjVyTtxLoCQM04nfjET_ZmqywTiw9LiR-ny78/edit I found this about exposure types:

DRK     MasterDark Image 
IDP       Instrumental Dispersion
NFF      Normalized Flat field Image
TEL      Telluric Standard 
FLX      Flux Standard
ARC     Calibration lamp
SKY     Sky Frame
SCI      Science Frame

Consider also that we use the DATAMODL keyword to identify the file structure, see https://github.com/oirlab/iris_pipeline/pull/23

So how do we reconcile all of this?

[arunsurya77]

So EXP_TYPE makes sense, but I also remember seeing JWST using ref_type to mark dark,flat etc. Is this redundant?

In the DRS metadata sheet I used FILETYPE to denote the type of file. But maybe better to stick with JWST keywords as much as possible. https://docs.google.com/spreadsheets/d/1Ja0kASKm5SySP79FgYpg1Fjcz5OOTrEx1waK398kQHw/edit#gid=0

On Thu, 7 May 2020 at 13:47, Andrea Zonca notifications@github.com wrote:

@ikashell https://github.com/ikashell @arunsurya77 https://github.com/arunsurya77

JWST has a keyword named EXP_TYPE, see its possible values:

https://jwst-pipeline.readthedocs.io/en/latest/jwst/assign_wcs/exp_types.html#wcs-reference-file-information-per-exp-type

for example for nircam: NRC_IMAGE, NRC_TSIMAGE, NRC_FOCUS, NRC_TACONFIRM, NRC_TACQ

Currently I am using IRIS_IMAGE for all files.

In the file naming document https://docs.google.com/document/d/1ppkyuNsjVyTtxLoCQM04nfjET_ZmqywTiw9LiR-ny78/edit I found this about exposure types:

DRK MasterDark Image IDP Instrumental Dispersion NFF Normalized Flat field Image TEL Telluric Standard FLX Flux Standard ARC Calibration lamp SKY Sky Frame SCI Science Frame

Consider also that we use the DATAMODL keyword to identify the file structure, see #23 https://github.com/oirlab/iris_pipeline/pull/23

So how do we reconcile all of this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oirlab/iris_pipeline/issues/22#issuecomment-625488852, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKB37LTFXYERBGXXDIMCITTRQMM57ANCNFSM4M22NYUQ .

[zonca]

also when we specify the association, we can input a exptype, see:

https://github.com/oirlab/iris_pipeline/blob/master/iris_pipeline/tests/data/asn_subtract_bg_flat.json

hopefully, this is optional and if missing, picks up the exp_type in the file. I will test and report here.

[zonca]

unfortunately it is not the case 'exptype' is a required property of the association file, so it is quite annoying that we cannot just pass all files to the pipeline and have it figure out the role of each file internally.

@arunsurya77 is the type of file (background, science) encoded also in the filename somehow?

[zonca]

sorry @arunsurya77 for the unnecessary question, checking in your document, file name is rYYMMDD_i/s_obsid_exp#_sub#.fits, so there is not exp type at all.

Anyway, the DRS assembly knows from the sequencer what is the observation plan so knows which observation ID is background.

[zonca]

My proposal is to keep:

EXP_TYPE as one of:

DRK     MasterDark Image 
IDP       Instrumental Dispersion
NFF      Normalized Flat field Image
TEL      Telluric Standard 
FLX      Flux Standard
ARC     Calibration lamp
SKY     Sky Frame
SCI      Science Frame

Checking the JWST definition https://github.com/spacetelescope/jwst/blob/master/jwst/datamodels/schemas/keyword_exptype.schema.yaml I think we should also prepend with IRIS_ (or should we have 2 separate prefixes for Image and spectra?)

And then have another FITS keyword: INST_MODE

ENG    Engineering
SCI      Science
CAL     Calibration

@arunsurya77 @ikashell is this redundant? or is it fine?

[arunsurya77]

The exp_type is mentioned for processed frames pYYMMDD_i/s_obsid_exp#_sub#_type.fits where type is SCI,DRK,SKY etc. But I agree with the proposal to have EXP_TYPE and INST_MODE inside the files. I will change the file format convention accordingly.