Enable cube_build in the spec2pipeline for NRS lamp exposures

[stscijgbot-jp]

Issue JP-1828 was created on JIRA by Robert Jedrzejewski:

The pars-spec2pipeline reference file needs to be updated to enable the cube-build step and set the coord_system parameter to internal_cal

Also the cubepar rmap needs to be updated to select the reference file based on EXP_TYPE and OPMODE/meta.instrument.lamp_mode and a new cubepar reference file delivered.

[stscijgbot-jp]

Comment by Robert Jedrzejewski on JIRA:

James Muzerolle is there any reason to use a different cubepar reference file or different cube_build parameters for NIRSpec lamp data as compared to science data, apart from setting coord_system to 'internal_cal'?  Also, should the cubepar reference file select on the LAMP keyword (i.e. do you think flat lamp exposures should use a different reference file from line lamp exposures?  I know flat lamp exposures aren't run through cube_build in the default spec2pipeline, but in principle you might want to run it offline)

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Robert Jedrzejewski The spatial parameters may be incompatible; in the reference files, they (SPAXELSIZE in CUBEPAR, ROISPATIAL and SCALERAD in CUBEPAR_EMSM) are given in units of arcsec, while the instrument transforms create x,y positions in the MSA plane in units of meters.  However, I'm not sure how Jane Morrison originally set this up; presumably, it is already using the existing cubepar reference files, since we have tested it successfully on lamp data? If that is the case, then we shouldn't need separate cubepar files, since the sampling does not need to be any different.  Likewise, there should not need to be a selection rule for LAMP, the sampling does not depend on the light source, only the disperser.

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

When internal_cal is set then the code really does not use the cubepars data. I had to look it up again because I forgot. But internal_cal sets the interpolation = area. That means that overlap between the detector and cube plane is determined in along slice and wavelength dimension. There is a one to one mapping in the across slice dimension. So all the parameters in the cube pars related to weighting is not used (SCALERAD, ROISPATIAL, CUBEPAR_EMSM.... ect are NOT USED). 

The WCS value of cube are determined from the data.  The pixel data is converted to the slicer plane coordinates. The min  and max coordinate in both spacial dimension are determined. The cdelt values of the cube are the (max across slice coord - min across slice coord)/nslices    nslices = 30 and the coordinates are in arc seconds. 

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Ok, thanks Jane Morrison!  We can just stick with the one cubepar file, then.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

What's the status of this work? Has a new pars-spec2pipeline ref file been implemented? Is there a corresponding CRDS ticket for that? Has this been included in B7.7?

[stscijgbot-jp]

Comment by Matt McMaster on JIRA:

Two new pars-spec2pipeline reference files for REF_TYPEs NRS_LAMP and NRS_AUTOWAVE covering LAMP_MODEs BRIGHTOBJ, FIXEDSLIT, and MSASPEC were delivered on Monday, January 11.

[hbushouse]

Given that this issue relates to turning on the cube_build step, it only pertains to OPMODE='IFU', for which the pars-spec2pipeline ref files are still from last year and both have the cube_build step set to be skipped. So I'm guessing that answers my question, in that updated pars-spec2pipeline ref files for IFU are obviously not yet available (or at least haven't been installed). @stscirij @jemorrison can you comment?

[stscirij]

The changes @matt-mcmaster is referring to enable resample_spec for NIRSpec lamp mode exposures that are MSA or FIXEDSLIT. NRS IFU lamp data need to run through cube_build. Currently this needs a CRDS spec file update (currently just assigned to Jonathan) for the cubepar reference file, once this is in place a new cubepar rmap can be delivered and then a new pars-spec2pipeline file delivered to enable cube_build for NRS lamp IFU exposures in the spec2pipeline.

[hbushouse]

OK, thanks @stscirij . Just wanted to know if I could declare this as already done and hence includable in our last B7.7 release, or still in progress. Obviously the latter is the case.

[stscijgbot-jp]

Comment by Robert Jedrzejewski on JIRA:

CRDS work covered in ticket CRDS-401

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

Robert Jedrzejewski James Muzerolle Beth Sargent

When we talked about needing a cubepars reference table description for lamp mode (last month ) I forgot that we need add one row to the cubepars reference file to define the default spaxel size and spectral size to make cube build for lamp exposures.  All that needs to be updated it in the first extension, CUBEPAR,  add a single row for lamp type data. If filter = 'opaque' I have set the grating in cube_build to be 'any'. I am assuming that that the spectral and spatial size is the same for filter = 'opaque'. If this is not true then I we will need to add a row that defines the spectral and spatial size for each opaque + grating option.  I have a branch now that 

1. sets grating = 'any' when filter = 'opaque'
   2. hard codes the spectral size for this data to 0.005 and spatial size to 0.1. I harded  coded the values to see if I that is all that was needed. If it those values are ok to use we could use these version if updating the cubepars reference file would take any time. 

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

James Muzerolle Robert Jedrzejewski

After talking to Robert some on this he wondered if maybe resample_spec might be a better place to resample this type of data since we don't want to combine wavelength planes or slices. In cube_build  I can force the spatial scale to be what is needed so the output files have an IFU cube with 30 as the dimension in the across slice dimension. James Muzerolle what do you think would resample_spec be better ? If we decide to resample lamp data using cube_build how do you want me to handle the wavelength sampling?  Should we put in an entry for each opaque/grating type or do something different (reading something in the header to determine the spectral scale?) 

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Jane Morrison Robert Jedrzejewski This might actually be a better solution, since the team had earlier discussed having 1D products for each slice.  Just to be clear - the proposed processing flow would be more similar to MOS/FS, with resample_spec producing an s2d product, from which extract_1d would produce x1d data?  Since we don't run extract_2d on IFU data, would having all the 2D spectra in a single image/extension be a problem?

[jemorrison]

I actually do not know much about the details of resample_spec. Let me research this more.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Switching to using resample_spec to resample IFU images and then having extract_1d perform extractions on a per-slice basis could entail significant refactoring for both steps and hence such a change should not be proposed lightly and without very strong justification.

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

James Muzerolle For now lets just get this working using cube_build - later if you don't think it is doing what you want we can look at other options.  How do you want the wavelengths mapped in the ifucube i.e. what do you want the spectral scale in the IFU cube to be ? 

 

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Jane Morrison I think the spectral sampling should be the same as for on-sky cubes, there should be no difference between the two cases (unlike in the spatial dimensions).

[jemorrison]

James what are the grating options if filter = lamp ? Below is a table in the cube pars reference files that sets the spectral scale for all the filter, grating options. If the grating that can be used with LAMP are covered in this table then I can use this table to set the spectral size of the IFU CUBE

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Jane Morrison there are specific combinations of gratings and lamps.  In all cases when a lamp is used, filter=opaque as that directs the lamp light onto the detector; each lamp has an internal filter that is analogous to the corresponding filter on the filter wheel.  I thought there was a ticket long ago that gave the mapping, but I can't seem to find it, so I'll list it below.  Essentially, the cube pars table could still be used, except that there is no LINE lamp equivalent to the filter=F070LP, so that particular case would never be used.  I don't list any of the FLAT lamps because we don't need cubes for those.

LINE1 - G140M and H + F100LP

LINE2 - G235M and H + F170LP

LINE3 - G395M and H + F290LP

LINE4 - PRISM + CLEAR

REF - G140M and H + F100LP

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

James Muzerolle I just looked at the only file I have for testing this (it is failing in regression testing) it has Grating = G235, Filter = opaque and Lamp = Ref

In your table you have ref = G140M or H.  So that is confusing. Is the file I have wrong ?

So should cube_build be reading in the LAMP key word ? Or if filter = opaque then look at grating and find the grating in the cube pars file ? As long as the spectral size is always the same for the same grating in the table - which I think it always should be then that would work. 

And we would not need to update the cubepars reference file (Beth Sargent)

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Oops, my apologies, I forgot about that case - we use a non-standard combination like that for calibration (it effectively samples the REF lamp at higher spectral orders) .  But that should still be ok because the sampling/wavelength coverage only depends on the grating, not the lamp.  The only difference with on-sky observations is that there is no equivalent to G140M/H+F070LP, so the minimum wavelength for those gratings should always be 0.97 microns (based on the current spectralconfigurations reference file).

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

James Muzerolle  Beth Sargent I think the cleanest way to have opaque be an option is to  update the cube_pars reference file. The first extension table needs to be expanded to contain all the possible combinations of 'opaque' + EACH grating. 

cube build reads the grating and filter and figures out what row it should use in this table. 

Beth if you can get me an preliminary updated cube_pars table I can finalize the changes. We are trying to get this done buy Friday to make it into Build 7.7  I have started adding what I think is needed

we will have a row for the following:

prism, opaque  G140M, opaque G140H, opaque G235M, opaque G395 M, opaque G395H, opaque

 

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Jane Morrison this looks fine, thank you.  Just to be clear, we need the extra rows because the spaxel size has to be different (units of meters instead of arcsec)?  All the spectral parameters should be the same.  What values have you been using for testing?  We can just start with that, and optimize later if necessary.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Looks like the list above is missing an entry for G235H (it's only got G235M). Each grating should have 2 entries, and M and an H, right?

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

So the way cube build is set up is first thing it does is it reads the header to know what the filter and grating the data covers. Next thing is it reads the reference file and matches the filter and grating in the reference file with what the data covers. If it can not find opaque in the reference file it stops. To make it clean and consistent with how cube build is set up if we add these rows to the reference file then it is easy. It really just needs the spectral size but to make it easy just put all the other information in. Don't change the units of the spaxel size cube build does this automatically for internal_cal data. I was trying to use the existing reference file but the code was getting messy with its selection when multiple rows contained the same grating. For example which row should it read if grating = G140M , the one with F070LP or FL100LP. The only difference is the wavelength coverage but I was not sure which one to use. If you add a row to the reference table for this grating+opaque then it is clear what is being used

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

Yes I missed G235M. The new reference file should have 7 new rows. 

[stscijgbot-jp]

Comment by Beth Sargent on JIRA:

I added a draft NIRSpec Cubepar file just now.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Partially fixed by #5757 but still needs CRDS-401 to complete.

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

resolved by #⁠5757 and CRDS-401