search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
570 stars 167 forks source link

Record extract_1d aperture location #7773

Closed stscijgbot-jp closed 8 months ago

stscijgbot-jp commented 1 year ago

Issue JP-3169 was created on JIRA by Howard Bushouse:

Given the unreliability of the automatic positioning of the spectral extraction aperture that's based on target coords, it would be very useful to have the actual positions (start/stop) that were used recorded somewhere in the x1d products, so that users can verify whether or not a good position was used.

stscijgbot-jp commented 1 year ago

Comment by Sarah Kendrew on JIRA:

Not critical for MIRI LRS but definitely adds value to have this information easily accessible. 

stscijgbot-jp commented 1 year ago

Comment by Howard Bushouse on JIRA:

Background info:

Currently the extract_1d step allows the user to specify xstart/xstop/ystart/ystop (or calculate them via extract_width) for 2D slit-like extractions and center_xy for 3D IFU extractions.

Info reported for various modes:

stscijgbot-jp commented 1 year ago

Comment by Howard Bushouse on JIRA:

For IFU modes with an EXTENDED source, the entire region covering the spatial axes of the IFU cube is extracted, which in effect is equivalent to using xstart/xstop/ystart/ystop values, where those values correspond to the edges of the cube.

For IFU modes with a POINT source, a circular aperture is used, with radius being a function of wavelength (as specified in the extract1d ref file for IFU mode). So this is essentially the only mode that does not fit the simple xstart/xstop/ystart/ystop paradigm. 

stscijgbot-jp commented 1 year ago

Comment by Greg Sloan on JIRA:

Could we store the parameters defining the aperture for spectral extraction as specific header keywords, as opposed to in the history comments? It would be convenient for users because that would make the parameters easily searchable.

stscijgbot-jp commented 1 year ago

Comment by Melanie Clarke on JIRA:

We have had some helpdesk inquiries related to this issue, from users looking to find out where their extraction apertures were placed or where in the image the source location is expected to be. 

The workaround for finding the extraction aperture is to re-run the extraction algorithm and note the location from the log, but this may not be entirely repeatable, since the default extraction parameters or algorithm may change over time.  

For finding the source location, for point sources in NIRSpec MOS or FS mode, the information is recorded in the SRCXPOS and SRCYPOS header keywords in the SCI extensions, but these values are in slit units and do not translate directly to pixel values in the s2d image. There is a workaround using information in the data model, e.g. in Python:

import numpy as np from jwst import datamodels

s2d = datamodels.open('test_s2d.fits')  for slit in s2d.slits:     slit2det = slit.meta.wcs.get_transform('slit_frame', 'detector')     middle_wl = np.nanmean(slit.wavelength)     loc_x, loc_y = slit2det(slit.source_xpos, slit.source_ypos, middle_wl)     print(f'Slit {slit.name}: source is at y = {loc_y}')

The loc_y value is the assumed location of the source in the corresponding SCI extension in the s2d file.

stscijgbot-jp commented 9 months ago

Comment by Jane Morrison on JIRA:

Melanie Clarke Howard Bushouse Greg Sloan David Law  I have gotten a little lost on what is needed in this ticket. Except for the IFU pt source mode the start/xend ystart/yend  is reported in the log. Do we just want to add that information into the header ?  Or/AND we do we want to record the extraction center - which would be at least give some information on the IFU pt source case.

stscijgbot-jp commented 9 months ago

Comment by Melanie Clarke on JIRA:

All of those would be useful to have as header keywords: x/y start/end values, aperture center, and aperture width. 

stscijgbot-jp commented 9 months ago

Comment by Jane Morrison on JIRA:

Sarah Kendrew Greg Sloan I am working on this ticket. I would like some flight LRS slitless data. I first looked in our regression testing and I was surprised we did not have a set (at least not one I found). So it would probably be best if I started with some uncal data and added this data to the regression testing.

Could you point me to some data that would be useful. Otherwise I can just grab something off MAST

stscijgbot-jp commented 9 months ago

Comment by Jane Morrison on JIRA:

Sarah Kendrew Greg Sloan My bad I found LRS slitless on the regression site. 

 

stscijgbot-jp commented 8 months ago

Comment by Tyler Pauly on JIRA:

TODO: Create JWSTKD ticket once new keyword names are finalized/merged.

stscijgbot-jp commented 8 months ago

Comment by Rachel Plesha on JIRA:

Am I mistaken, or would this affect all instruments? I notice that only MIRI and NIRSpec are listed in the instrument field, but all instruments are listed in Howard's comment above, so that's why I'm clarifying.

 

For NIRISS WFSS, I'm able to get the "slit" position from the cal file headers:

SLTNAME = '1       '           / Name of the slit                                SLTSTRT1=                  449 / Starting pixel in x direction                   SLTSIZE1=                   85 / Number of pixels in x direction                 SLTSTRT2=                  506 / Starting pixel in y direction                   SLTSIZE2=                   11 / Number of pixels in y direction     

Although I agree that it is strange this isn't populated to the x1d files. Is this the information that is being discussed in this ticket to populate?

stscijgbot-jp commented 8 months ago

Comment by Howard Bushouse on JIRA:

Rachel Plesha No, those SLTxxxx keywords record the location of the 2-D cutout made during the extract_2d step. This ticket is referring to the location of the aperture used to create the 1-D spectrum in the extract_1d step.

stscijgbot-jp commented 8 months ago

Comment by Rachel Plesha on JIRA:

Chatting a little bit offline, I think I understand now that these numbers will be slightly different than those produced by the extract_2d step in that the extract_1d step has the potential to truncate the extract_2d box a bit further (most likely in the cross-dispersion direction), and the values will likely be in the cutout/extraction box reference frame rather than the full frame.

I still believe this is relevant to NIRISS data, so I plan to add NIRISS to the instrument list unless there are strong objections.

jemorrison commented 8 months ago

@Rplesha When I put together the PR for this work #8278 the location of the aperture for the 1-D spectrum for NIRISS is not available in the code. I do not know how to get that information for the NIRISS data.

hbushouse commented 8 months ago

Are we talking about NIRISS SOSS or WFSS? I assume the latter. SOSS extractions are highly specialized and don't use fixed limits, at least not in the cross-dispersion direction, because the spectral trace is highly curved.

This could be applied to NIRISS WFSS (as well as NIRCam WFSS), as those do use fixed limits in the cross-dispersion direction for each spectrum being extracted.

stscijgbot-jp commented 8 months ago

Comment by Greg Sloan on JIRA:

For SOSS, we could identify the extraction aperture by specifying the position in a given column

stscijgbot-jp commented 8 months ago

Comment by Howard Bushouse on JIRA:

Keywords added to schema in spacetelescope/stdatamodels#264

stscijgbot-jp commented 8 months ago

Comment by Howard Bushouse on JIRA:

Greg Sloan For SOSS, I'm not sure it's worth the effort trying to record any kind of extraction aperture location, due to the fact that any user who really knows anything about SOSS data is aware of the fact that the trace is so highly curved and changes width as a function of wavelength, that any single representation of an aperture region would be semi-meaningless, or would take more time to explain (e.g. at what wavelength do those limits apply) than the amount of useful info they provide (IMHO). I think it really only makes sense to record these values for modes in which there's one single set of xstart/xstop (or ystart/ystop) values used for the entire spectrum. This is why, for example, we don't bother to try to record the aperture size for IFU data, because it's wavelength dependent and hence a single value is meaningless.

stscijgbot-jp commented 8 months ago

Comment by Greg Sloan on JIRA:

In code of mine for a previous mission not to be named, the spectral trace was definitely a polynomial, and I specified the position by just taking the y_intercept at a default position (row 0 in my case - dispersion direction was up/down). So for the extraction boundaries, something similar could be done, if users found the information helpful. For SOSS, I'm not sure either how useful this value would be. Not saying we should do this, just that we could.

stscijgbot-jp commented 8 months ago

Comment by Rachel Plesha on JIRA:

Howard Bushouse Initially I was thinking both WFSS and SOSS, but you raise some good points about SOSS that I will need to follow up with those who are more familiar to see if this is desired.

stscijgbot-jp commented 8 months ago

Comment by Howard Bushouse on JIRA:

Fixed by #8278

stscijgbot-jp commented 7 months ago

Comment by Christian Hayes on JIRA:

Looking at this fix for NIRSpec (specifically for the Fixed Slits and MOS modes) there is a 1-pixel difference between the xstart/xstop/ystart/ystop reported in the headers of reduced x1d products and those listed in the log/reference files/user specifications.  It looks like values in the new header keywords are adjusted to a 1-index, whereas the log etc. use a 0-index.

In an example the log reports:  xstart=0, xstop=3269, ystart=26.5, ystop=31.5 (matching the user input)

And the associated x1d header lists:  extraction_xstart=1, extraction_xstop=3270, extraction_ystart=27.5, extraction_ystop=32.5

This might be confusing for users who are trying to reproduce the same results using the header keywords or using these values to inspect their data.  Would it be possible to change the header keyword assignment to match the other instances of these values?

jemorrison commented 7 months ago

All of our other existing keywords that record either subarray or 2-D cutout start/stop limits use values that are 1-indexed (in keeping with FITS conventions).We made them 1-indexed for consistency with all the other similar keywords.

stscijgbot-jp commented 7 months ago

Comment by Melanie Clarke on JIRA:

Thanks Jane Morrison. Is that choice documented somewhere, so users can translate the header keywords appropriately?

stscijgbot-jp commented 6 months ago

Comment by Rachel Plesha on JIRA:

In testing this for NIRISS WFSS, I think this could be really helpful to have in the x1d files, but there is still a link missing that actually makes this useful for WFSS. As it's written currently for WFSS, it appears from my tests that the new keyword "EXTRYSTP" is always equivalent to the y size of the cutout which is equivalent to the cal file keyword "SLTSIZE2" (and EXTRXSTP = SLTSIZE1 = x size of cutout). I think there's potential for this to be a tremendous value add to not have to go back to the cal file for this information, especially in having to identify the right extension, but what is missing from this new x1d file is the starting location on the grism image (SLTSTRT2 / SLTSTRT1 in the cal files).

For a random source in a random file I'm testing, in the cal file:

SLTSTRT1=                  457 / Starting pixel in x direction                   SLTSIZE1=                 1335 / Number of pixels in x direction                 SLTSTRT2=                   48 / Starting pixel in y direction                   SLTSIZE2=                  755 / Number of pixels in y direction                 SOURCEID=                   25 / Source ID                          SRCXPOS =   1249.6156515288842 / Source position in slit (x-axis)                SRCYPOS =    370.8518033884632 / Source position in slit (y-axis)    

From this I'm able to go to the grism image and see what source is at (457, 755) and look to see if maybe the slit is miscentered or needs to be adjusted on the entire image

in the x1d file:

SRCXPOS =   1249.6156515288842 / Source position in slit (x-axis)                SRCYPOS =    370.8518033884632 / Source position in slit (y-axis)                            EXTRXSTR=                    1 / X axis start of extraction region               EXTRXSTP=                 1335 / X axis end of extraction region                 EXTRYSTR=                    1 / Y axis start of extraction region               EXTRYSTP=                  755 / Y axis end of extraction region    

From this I can only tell how big the cutout is and where the center of the source is. I still need to go back to the cal image to find where the start of the cutout actually is.

It would be great to be able to go from the x1d directly to the grism image, and I think only the starting x and y pixel positions are needed for that additional information, but I don't have a good grasp on 1) if that information is stored going into extract_1d and 2) how that would translate for the other modes.

One last note about this that is probably way too late in the game for this, but I thought that "STP" was "step" when I first read this, and it might be better as "EXTRXEND" / "EXTRYEND"

stscijgbot-jp commented 5 months ago

Comment by David Law on JIRA:

Howard Bushouse Any thoughts on Rachel Plesha 's comments above with regard to propagating the SLITSTRT1 and SLITSTRT2 keywords to X1d files?

stscijgbot-jp commented 2 months ago

Comment by David Law on JIRA:

Rachel Plesha I think your suggestions above have been getting lost at the bottom of this old RTT ticket.  I'm going to close this ticket since the original work has been done, but can you please open a new ticket requesting better functionality for WFSS?  (I concur that it would be useful to be able to map back from 1d info to the original locations, though not sure what the easiest way to convey this information is).

stscijgbot-jp commented 2 months ago

Comment by Rachel Plesha on JIRA:

Thanks David Law! I've created the ticket at: JP-3751