Dither positions in calspec2

[philhodge]

There was an email discussion recently between Sarah Kendrew, @hbushouse, and @philhodge regarding dither positions (nod offsets) for MIRI LRS fixed-slit data, in particular how to change the offsets. Here is most of the email thread, older messages first.

Hi Howard

We recently found that the APT dither positions for MIRI LRS-slit (2 along-slit nod positions) are different from what we had originally specified (years ago). It’s only a small difference and the offsets implemented are fine, but I’d like to verify that there is no inconsistency in any other subsystems as a result. How does the pipeline (CALSPEC2) get information on the location of the dithers in the slit exactly? Are relevant FITS header keywords populated by APT or OPGS...?

Thanks! Sarah

Hi Sarah,

There are a couple different ways in which we get or infer source positions for the nods. First, there are the regular WCS pointing keywords that tell us the sky position of the aperture reference point that was used for the observations. Those come directly from the telescope pointing information for each exposure, so small differences will automatically be reflected in that information. We then also have available the offsets from the aperture reference point that were used to place the source in each exposure. Those are computed within either APT or PPS and we then get the values from the PPS database and use them populate keywords XOFFSET and YOFFSET. So again, if anything changes to use slightly different offsets, we should get the correct info passed along to us.

Finally, the only place that we currently have anything close to "hardwired" in the cal pipeline, is in the CRDS reference file that's used by the extract_1d pipeline step to extract 1D spectra from the 2D slit images. It currently uses a parameter that specifies the size of the offset between nods, so that the extract_1d step knows where to center the extraction region for nodded exposures. So if the offsets change, that reference file could be updated to use a modified value. Theoretically, we could at some future time upgrade that approach so that extract_1d uses actual RA/Dec info for the source - in conjunction with the WCS of the exposure - to figure out where to center the extraction region. If we did that, then we wouldn't need to know anything about the size of the nod offsets.

Hope this helps, Howard

Great, thanks for the information.

So if you receive pointing information from APT/PPS then this should be consistent.

I had a look at the CRDS file (jwst_miri_extract1d_0004.json), and I can see a “nod2_offset” that is set to 15 for slit operation. I presume this is the parameter you mention? In what units is this value specified? I think this is the one we may need to verify to make sure it agrees with the APT offset.

Cheers Sarah

Yes, that's the parameter I was talking about. It's in units of pixels and I /believe /it's treated as the offset from the nominal center of the slit, i.e. it assumes one nod is +15 pixels and the other is -15 pixels, which implies a total separation of 30 pixels. Does that make sense relative to the current LRS nod setting?

It's done in units of whole pixels because right now we're using just a simple "boxcar" extraction that sums up whole pixel values in the cross-dispersion direction. Once optimal extraction is implemented it will of course automatically center the extraction profile to fractional pixel levels, as well as follow any non-linearities in the trace.

Hi Howard,

OK thanks. I’ve included in attachment a schematic drawing of the slit layout, with the dither locations as they are implemented in APT. These positions are ~8 pixels either side of the slit centre position. If the current extraction apertures are centred on ± 15 pixels from the slit centre, this would place a point source quite far off centre in the aperture. The effect of this would probably be minimal given the wide aperture we’re using for this baseline option, but it’s probably best to have these locations consistent between subsystems. What is the process for changing this value in the CRDS?

Looking at how this is implemented, I wonder if it would also make sense to make this aperture a bit smaller. Having a 40-px wide box actually includes quite a few pixels that fall outside of the slit region, which doesn’t make much sense. I will discuss that with Jeroen first.

Cheers Sarah

Sarah,

Howard forwarded to me a thread regarding the nod offsets for MIRI LRS fixed-slit observations. You mentioned that the offsets are about 8.25 pixels on either side of the nominal position, while the current value in the reference file is 15 pixels. Since the reference file is a plain text file, you can easily change the value and submit the modified file for inclusion in CRDS. The current extraction width is 11 pixels, and if that is too large (or too small) it is also easy to change that. Currently, both the nod offset and extraction width are integer values.

The reference file jwst_miri_extract1d_0004.json includes this section for MIRI LRS fixed-slit:

    {
    "id": "MIR_LRS-FIXEDSLIT",
    "region_type": "target",
    "bkg_order": 0,
    "dispaxis": 2,
    "extract_width": 11,
    "nod2_offset" : 15
    },

nod2_offset is the offset in pixels for two-position nodding, and extract_width is the number of pixels in the extraction region in the cross-dispersion direction. So if you want to change the nod offsets to be +8 and -8 pixels, and change the extraction width to 8 (just for example), you could make a copy of jwst_miri_extract1d_0004.json and change the above section to:

    {
    "id": "MIR_LRS-FIXEDSLIT",
    "region_type": "target",
    "bkg_order": 0,
    "dispaxis": 2,
    "extract_width": 8,
    "nod2_offset" : 8
    },

Once you have the modified reference file (suppose for example you called it jwst_miri_extract1d_SK.json), you can run extract_1d using that file as a test. If jw_MiriSpec2Pipeline_assign_wcs.fits is MIRI LRS fixed-slit data (a rate image, after running assign_wcs), you can run extract_1d from the command line and specify that your reference file should be used instead of the one in CRDS:

strun extract_1d.cfg jw_MiriSpec2Pipeline_assign_wcs.fits --override_extract1d=jwst_miri_extract1d_SK.json -v

By including "-v" (as I did above), the step will print info about the extraction location.

When you're confident that the modified file works as you expect, you can send email to the "redcat" team to ask them to install it in CRDS.

Phil

Thanks - this is very helpful. Also thanks for pointing out the extraction width parameter, I had made a wrong assumption for this - I think 11 pixels is fine for this simple boxcar extraction.

I’ll investigate how important it is to centre the nod position locations with some tests (thanks for the pipeline tips!), if the CRDS file needs updating I will follow your instructions below.

Cheers Sarah

I'm a little concerned about the CRDS reference file for this mode. Note that it doesn't specify the extraction location, and the actual region where the spectrum is found is small compared with the full image. What happens is that extract_1d obtains the so-called bounding_box from the wcs information, and the spectrum is assumed to be centered within that region. For the test file that I've been using for this mode, the bounding box is:

m = datamodels.open("jw_MiriSpec2Pipeline_assign_wcs.fits") wcs = m.meta.wcs wcs.bounding_box ((299.77272403247969, 342.40556828997506), (4.7003852824524301, 392.99741287034306))

That is, roughly x = 300 to 342 (cross-dispersion direction) and y = 5 to 393 (along the dispersion direction), zero indexed. extract_1d takes x = (300 + 342) / 2 as the center of the extraction width, and then the specified extract_width = 11 is used to get the limits, which are 315.5 to 326.5 (where the .5 indicates the edge of a pixel, so that range means pixels 316 to 326, inclusive, and also including the entire pixel value at each endpoint).

My concern is that the bounding box might shift with the nod offset, because the bounding box is a part of the wcs. If that's the case, then we would not need to specify a nod offset. It might be better to explicitly specify the nominal extraction location in the reference file (where the spectrum would be found if there were no nod offset), and also explicitly specify the nod offset (as is currently done, but change the value from 15 to 8). Then the nod offset would be used (by extract_1d) to shift the nominal extraction location to the actual target location. In the example above, one could specify integer pixel limits (in x, i.e. cross-dispersion) by including these keys:

    "xstart": 316,
    "xstop": 326,

or specify precise lower and upper edges (float) using this key:

    "src_coeff" = [[315.5], [326.5]],

"src" means the source (i.e. target, as opposed to a background region), and "coeff" means that what is specified is a list of one or more polynomial coefficients for possibly curved extraction limits. [315.5] is the list of coefficients (just the constant term) for the lower limit, and [326.5] is the list of coefficients for the upper limit (also just the constant term).

Phil

Yes, this is where I was getting confused between extraction widths. I am aware that in our CDP we specify this 40-px width (which is basically the slit length), but this does not specify anything for the different nod positions specifically. As you say, I think there is scope for rationalizing the approach for this step. Give me a few days to look into it, I will need to run this by my colleague Jeroen Bouwman in Heidelberg, who has worked on this calibration step in the past and who created the CDP.

Is it worth moving this discussion into a Github issue by the way? This might make it more traceable.

Cheers Sarah

Good question. This would be a question for Nadia, as to how the bounding box is constructed from the WCS-related reference data for the LRS slit mode. I have a hunch it might be based on some limits over which the WCS transforms are defined in the reference data for the LRS slit, and if that's the case, then I would expect those limits to remain fixed in detector space and we would have to account for nod offsets somewhere downstream, either via the offset param in the extract1d reference file, or by starting to make use of the X/YOFFSET keyword values in the extract_1d step in order to offset the nominal center of the extraction region. That would be more robust for future changes in the offset size than having it come from the reference file param.

[stscicrawford]

cc: @nden @skendrew

This does seem related to #1604

[hbushouse]

@stscicrawford Possibly, but even though the MIRI LRS spectra fall on the imager detector, I believe the true FOV for the Imager portion has it's own separate bounding box that excludes the regions occupied by the LRS spectra, the coronagraphic masks, etc.

[JuergenSchreiber]

After this item was addressed in our last LRS telecon I want to write a comment on the extract_1d step for LRS. I think the addressed concept of a bounding box with nod offsets etc. might only serve as a kind of first approach. We have more detailed information in a LRS CDP file called MIRI_FM_MIRIMAGE_P750L_DISTORTION_7B.02.00.fits where the exact positions of the slit on the imager array for each wavelength is listed, especially the center, the left and right edge of the slit. You will recognize that these values are slightly changing in y (with wavelength) in a way that the x bounding box does not stay exactly at the same columns (about 2 pixels difference from top to bottom). Therefore, concerning the spectral extraction in the pipeline, these values should be considered. My understanding was anyhow that there will be a one - to - one translation of the CDP files to CRDS files which we would definitely need for the spectral extraction step.

[skendrew]

Thanks for weighing in @JuergenSchreiber, I agree that the current approach seems inconsistent between the CDP and CRDS. Would it make sense to tie the open question on the PHOTOM step for LRS into this as well? The PHOTOM CDP for LRS was currently measured for a particular aperture, which is different again from the extraction aperture used in the extract_1d step.

[nden]

To answer some of the questions raised above. The reference file, MIRI_FM_MIRIMAGE_P750L_DISTORTION_xxx.fits, is delivered to CRDS exactly as it is in the CDP. I believe it's the only WCS reference file in FITS format. Its reference type is specwcs, not distortion .

The bounding_box defines the valid region on the detector code here. I don't think it shifts with nodding. The code for slitless mode needs to be updated. It still has the zero-point hard-coded because of a problem with the first delivered reference file.

[JuergenSchreiber]

Hi Nadia, thanks a lot for clarification. So, this information from assign_wcs for LRS can also be used in extract_1d. Indeed, the slit area does not change with nodding, but the source position inside the slit changes, therefore, the extraction box should always be centered on the source position which you get from the telescope pointing information.

[philhodge]

The extract_1d step has two different options which can be used to handle the case that the extraction region changes along the dispersion direction. The lower and upper limits (or left and right, in this case) can be specified as polynomial functions; the polynomials may be functions of either pixel number in the dispersion direction or wavelength in microns. Both the extraction region and zero or more background regions may be defined by polynomials. The second option is to use a reference image, with values -1, 0, or 1 (float type, but integer values). Set the value in the reference image to +1 to indicate that the corresponding pixel in the science image should be included in the extraction; use -1 to flag background pixels (if any); and 0 means the pixel should be excluded. The polynomial option is probably the better choice for MIRI LRS, because fractions of edge pixels will be included; with the reference image, every pixel is either included completely or not at all.

There is code in extract_1d to compute the cross-dispersion pixel offset due to a nod or dither. This is implemented for NIRSpec, based on keywords XOFFSET and YOFFSET, and using WCS parameters, but if I understand correctly the relevant WCS reference information is not yet available for MIRI. This function is called, and the value of the offset is written to the log (if verbose was specified), but the offset is not currently applied. The reason it's not applied is simply that I don't know how to test it. When realistic test data are available that include nod and/or dither, the offset that we would see in the image can be compared with the offset written to the log.

[skendrew]

thanks Phil. the jwst_miri_extract1d_0004.json file does have a parameter called nod2_offset, which I thought was used for the extraction of nodded exposures. Is that not the case?

[philhodge]

It was supposed to. I think it would have worked for a two-position MIRI nod pattern, but I'm pretty sure it would not have worked for any NIRSpec nod pattern (even the two-position pattern has different offsets for each position). Also, it looks to me as if a dither offset would not be taken into account at all. So I removed that option (nod2_offset, etc) because it wasn't reliable. I could put it back for MIRI, but that would be a short-term solution. It should be possible to compute the offset from the XOFFSET and YOFFSET keywords as long as the WCS reference files are available. @nden are the relevant reference files for MIRI available now?

[nden]

@philhodge The X/YOFFSET method requires a transform from V2V3 to detector. AFAIK there is no spatial information for MIRI LRS (other than the location of the target), i.e. there is no transform from detector to V2V3.

[JuergenSchreiber]

Thanks @philhodge for clarification. My remark for MIRI-LRS extract_1d would then be, why do you not draw your extraction region (slit edges) directly from the specwcs CRDS file or from the result of assign_wcs using interpolation if necessary?

[philhodge]

Are you referring to wcs.bounding_box? That's only an outer limit, the region over which the WCS is defined.

The X/YOFFSET method does use the result of assign_wcs.

[skendrew]

OK, so perhaps the way to take this forward is to provide some test data that we run through the code? Once we have that, I'd be keen to meet up in person to go through the code in detail to understand how this step works if that's ok? @JuergenSchreiber is currently updating MIRISim with the locations of the nod positions (which we updated for APT 26.1), once we have this implemented we can provide some nodded test data.

(I will copy this info to the JIRA ticket)

[philhodge]

@skendrew Yes, I'd like to run some test data through extract_1d. However, please see Nadia's comment (dated 31 Aug 2018, 16:25) regarding the lack of a transform from the MIRI detectors to V2V3. extract_1d can't use the X/YOFFSET method without the WCS. I would be glad to meet with you and go through the code.

[nden]

I'm also interested in meeting and going over the calspec2 code for the LRS. I am available after Oct 6th.

[skendrew]

[By the way, I noted this issue: https://github.com/spacetelescope/jwst/issues/268 from 2 years, which contains info from @drlaw1558 on the question of conversion from detector -> v2v3 -> sky for LRS. ]

[hbushouse]

JP-1808 and #5535 contain a continuing discussion of the proper centering of the extraction aperture for MIRI LRS fixedslit nodded exposures and has been declared to now be working correctly. So I'm closing this.