search

spacetelescope / mirage

This code can be used to generate simulated NIRCam, NIRISS, or FGS data
https://mirage-data-simulator.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
39 stars 41 forks source link

Implement WFSS mode for NIRISS #220

Closed bhilbert4 closed 5 years ago

bhilbert4 commented 5 years ago

I think all we should need to do is collect the sensitivity files, add a custom seed image size (see the value used for POM observations that is already in mirage), and then allow the mode 'WFSS' for NIRISS.

bhilbert4 commented 5 years ago

I've put a copy of the configuration and sensitivity files for NIRISS into the local copy of the Mirage reference files. I just checked with Nor to see if the background files exist anywhere. https://github.com/npirzkal/GRISM_NIRISS

bhilbert4 commented 5 years ago

@KevinVolkSTScI, Nor says he does not have WFSS background files for NIRISS. Do you know if those files exist anywhere?

KevinVolkSTScI commented 5 years ago

If you mean the background shape template files per filter/grism combination then yes I have something for these from CV3. If you want background estimates I have those as well. I can dig these items up for Nor next week.

bhilbert4 commented 5 years ago

@KevinVolkSTScI I think the former is what I'm looking for. The NIRISS equivalent to the files in our local mirage_data directory, in: mirage_data/nircam/GRISM_NIRCAM/*back.fits

KevinVolkSTScI commented 5 years ago

I should have what you want available back home. I will look at the above NIRCam background files and then see if there is any normalization issue or anything that needs to be adjusted with the files, and then post them somewhere where Nor can get to them.

bhilbert4 commented 5 years ago

Thanks! Once you have them, easiest would be to place them into our local mirage_data directory, in: mirage_data/niriss/GRISM_NIRISS at which point mirage will be able to access them. @NorPirzkal may be interested in adding them to his GRISM_NIRISS github repo as well.

KevinVolkSTScI commented 5 years ago

I have put some FITS images in $MIRAGE_DATA/niriss/GRISM_NIRISS

normalized_background_F090W-GR150C.fits normalized_background_F090W-GR150R.fits normalized_background_F115W-GR150C.fits normalized_background_F115W-GR150R.fits normalized_background_F140M-GR150C.fits normalized_background_F140M-GR150R.fits normalized_background_F150W-GR150C.fits normalized_background_F150W-GR150R.fits normalized_background_F158M-GR150C.fits normalized_background_F158M-GR150R.fits normalized_background_F200W-GR150C.fits normalized_background_F200W-GR150R.fits

These are normalized external flats through the different filter/grism combinations for NIRISS, with the mean value for a region near the center of the images normalzed to 1.0. If one has a value for the total background in all orders for a filter and grism, say from the ETC background calculator, then one can multiply the image by this value to get an expected background image. In some instances this will possibly produce signal images that are high by up to 10% because of structure seen in the images. Given the expected low background signals in NIRISS this difference should not affect the simulations significantly. I was not able to deduce how the NIRCam grism/filter background images are normalized: possibly to a particular assumed background spectrum. I can do the same for the NIRISS images if someone can tell me what the normalization should be. Note that these background images include the pixel area map and flat field effects in them already as they were made from external illumination of NIRISS. Of course we do not know if the illumination was uniform. The images are noisy and show various detector features along with bad pixels. The NIRCam background images appear to be simulations. While I have equivalent simulations for NIRISS, these do not correspond perfectly to the observed background. It seems that the real background images are rotated slightly and offset compared to the simulations.

bhilbert4 commented 5 years ago

I'll look back to my conversation with @NorPirzkal to find the details about these. I'm sure he remembers better than I do, unless he's already left for vacation.

bhilbert4 commented 5 years ago

@KevinVolkSTScI I'm working on adding the NIRISS WFSS template to Mirage's APT reader. Is it true that WFSS observations will always be full frame? I don't see any option in APT to set a subarray.

KevinVolkSTScI commented 5 years ago

There are sub-arrays for WFSS that are only used in calibration. The following are all the NIRSS_WFSS sub-arrays in the list:

NIS_WFSS_OFFSET FULL 4 NIS_WFSS64 FULL 4 NIS_WFSS64R WFSS64R 1 NIS_WFSS64R3 WFSS64R 1 NIS_WFSS64C WFSS64C 4 NIS_WFSS64C3 WFSS64C 4 NIS_WFSS128 FULL 4 NIS_WFSS128R WFSS128R 1 NIS_WFSS128R3 WFSS128R 1 NIS_WFSS128C WFSS128C 4 NIS_WFSS128C3 WFSS128C 4

The ones labelled WFSS128R/C and WFSS64R/C are calibration-only sub-arrays for WFSS.

bhilbert4 commented 5 years ago

Is there an example APT file somewhere that includes this? I just downloaded the NIRISS grism photometric and trace calibration programs. Those use the NIRISS External Calibration template rather than WFSS, but I still don't see any option for choosing a subarray in those.

KevinVolkSTScI commented 5 years ago

The trace calibration is done full-frame. All the photometric calibration is done in the sub-arrays, which are selected within the individual observation in the NIRISS external calibration mode.

bhilbert4 commented 5 years ago

Ah ok. I see it now. Thanks.

bhilbert4 commented 5 years ago

Tagging @jotaylor so she can stay up to date with this issue.

bhilbert4 commented 5 years ago

Circling back around to this now that the APT -> yaml file generator seems to be working for NIRISS WFSS.

@KevinVolkSTScI, here is a description from Nor of exactly what the NIRCam background files contain.

The NIRCAM backgrounds were generated using the spectral background components from the ETC, using a medium level. The spectrum was then dispersed to produce the proper background models. The value the ETC gives corresponds to the largest (in most cases, ignoring 2nd order) values of the backgrounds I generated. One thing the WFSS disperser should do is to allow one to scale these. Right now, the .finalize() function does not do that but adding an option to set the level (in e-/s) would be trivial.

So in the interest of getting WFSS simulations functional in Mirage quickly, and giving some time to think about the best way to implement renormalization, do you think you could produce background files for NIRISS using the same method as was used for NIRCam?

I've started working on the modifications to the seed generator and wfss_simulator.py to enable NIRISS, and the changes needed are very minor. It shouldn't take me very long to get everything working.

bhilbert4 commented 5 years ago

And just to be explicit, the units of the signals in the background images are e/s (i.e. outputs from the ETC).

KevinVolkSTScI commented 5 years ago

Hi Bryan,

I have such simulated dispersed zodiacal light images and there are also CV3 lamp flats through the GR150R/C grisms (these are similar but not exactly the same). Hence creating these files is straightforward and should not be difficult. I will try to get these to you soon….certainly getting something that is sufficient to start with is easy.

There were issues with the ETC background calculations for NIRISS WFSS that we have been trying to get resolved. All these stemed from only considering the first order while in some cases for our grisms the second and zero order components are significant. I think these issues have now been fixed for NIRISS but they probably did not think of doing the same for NIRCam WFSS.

Kevin

From: Bryan Hilbert notifications@github.com Reply-To: spacetelescope/mirage reply@reply.github.com Date: Friday, January 25, 2019 at 10:08 AM To: spacetelescope/mirage mirage@noreply.github.com Cc: Kevin Volk volk@stsci.edu, Mention mention@noreply.github.com Subject: Re: [spacetelescope/mirage] Implement WFSS mode for NIRISS (#220)

Circling back around to this now that the APT -> yaml file generator seems to be working for NIRISS WFSS.

@KevinVolkSTScIhttps://github.com/KevinVolkSTScI, here is a description from Nor of exactly what the NIRCam background files contain.

The NIRCAM backgrounds were generated using the spectral background components from the ETC, using a medium level. The spectrum was then dispersed to produce the proper background models. The value the ETC gives corresponds to the largest (in most cases, ignoring 2nd order) values of the backgrounds I generated. One thing the WFSS disperser should do is to allow one to scale these. Right now, the .finalize() function does not do that but adding an option to set the level (in e-/s) would be trivial.

So in the interest of getting WFSS simulations functional in Mirage quickly, and giving some time to think about the best way to implement renormalization, do you think you could produce background files for NIRISS using the same method as was used for NIRCam?

I've started working on the modifications to the seed generator and wfss_simulator.py to enable NIRISS, and the changes needed are very minor. It shouldn't take me very long to get everything working.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/spacetelescope/mirage/issues/220#issuecomment-457602731, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AjSUEE48K-UtCaiBZU_YX6_rQuxCBWkAks5vGx4BgaJpZM4YenaF.

NorPirzkal commented 5 years ago

Hi

Actually, the NIRCAM did think about it… It is just that we do not have multiple orders. There is a second order but that only affects the bluest filters and the edges of the FOV.

I have fully simulated NIRCAM WFSS background level estimates for all positions on the sky and time of the year. This is a lot of data and I used it mainly to determine how much the dispersed background would vary in NIRCAM over the course of the year since the different components (thermal, zodi etc) make up different fraction of the dispersed background and these have different spectral characteristics. Right now, the goal with my Mirage simulations has been to get realistic S/N in 2D spectra so that we can simulate and guide our expectation with our CEERS ERS NIRCAM WFSS observations.

The backgrounds I generated for NIRCAM match the medium setting in the ETC, using an extraction width of 0.032”, which corresponds (not intuitively) to a single pixel.

N

On Jan 25, 2019, at 10:56 AM, Kevin Volk notifications@github.com<mailto:notifications@github.com> wrote:

Hi Bryan,

I have such simulated dispersed zodiacal light images and there are also CV3 lamp flats through the GR150R/C grisms (these are similar but not exactly the same). Hence creating these files is straightforward and should not be difficult. I will try to get these to you soon….certainly getting something that is sufficient to start with is easy.

There were issues with the ETC background calculations for NIRISS WFSS that we have been trying to get resolved. All these stemed from only considering the first order while in some cases for our grisms the second and zero order components are significant. I think these issues have now been fixed for NIRISS but they probably did not think of doing the same for NIRCam WFSS.

Kevin

From: Bryan Hilbert notifications@github.com<mailto:notifications@github.com> Reply-To: spacetelescope/mirage reply@reply.github.com<mailto:reply@reply.github.com> Date: Friday, January 25, 2019 at 10:08 AM To: spacetelescope/mirage mirage@noreply.github.com<mailto:mirage@noreply.github.com> Cc: Kevin Volk volk@stsci.edu<mailto:volk@stsci.edu>, Mention mention@noreply.github.com<mailto:mention@noreply.github.com> Subject: Re: [spacetelescope/mirage] Implement WFSS mode for NIRISS (#220)

Circling back around to this now that the APT -> yaml file generator seems to be working for NIRISS WFSS.

@KevinVolkSTScIhttps://github.com/KevinVolkSTScI, here is a description from Nor of exactly what the NIRCam background files contain.

The NIRCAM backgrounds were generated using the spectral background components from the ETC, using a medium level. The spectrum was then dispersed to produce the proper background models. The value the ETC gives corresponds to the largest (in most cases, ignoring 2nd order) values of the backgrounds I generated. One thing the WFSS disperser should do is to allow one to scale these. Right now, the .finalize() function does not do that but adding an option to set the level (in e-/s) would be trivial.

So in the interest of getting WFSS simulations functional in Mirage quickly, and giving some time to think about the best way to implement renormalization, do you think you could produce background files for NIRISS using the same method as was used for NIRCam?

I've started working on the modifications to the seed generator and wfss_simulator.py to enable NIRISS, and the changes needed are very minor. It shouldn't take me very long to get everything working.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/spacetelescope/mirage/issues/220#issuecomment-457602731, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AjSUEE48K-UtCaiBZU_YX6_rQuxCBWkAks5vGx4BgaJpZM4YenaF.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/spacetelescope/mirage/issues/220#issuecomment-457619701, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AlQfMQQ1G4KmsfGwBBTVvtQ_vGr14hEOks5vGylHgaJpZM4YenaF.

KevinVolkSTScI commented 5 years ago

Nor:

The background simulations for NIRISS were made using the "standard" ETC background spectra (from some years ago, not the ones from the more recent simulation code that is position dependent). The spectrum was convolved with the grism response to calculate the signals in each order. Separately a simulation was made of the relative structure of the background to compare with CV3 observations using the ISIM external flat lamp.

Above there is a comment from November where I posted the normalized background images for the 12 grism/filter combinations. These can be scaled to a given total spectrum count rate if that is needed.

It is not entirely clear to me what you mean when you say that the ETC was used with the medium option to produce the background, since the values will depend on the date entered into the ETC. If I simply scale the CV3 lamp images already posted to the appropriate signal calculated for the particular grism + filter combination will that suit your needs?

p.s. Above when I commented that "they probably did not think of doing the same for NIRCam WFSS" I was referring to the ETC programmers. We had them fix the NIRISS grism background to include all orders, but they probably did not do the same thing for the NIRCam grism case. As you say that the additional orders are usually not important this may not matter.

KevinVolkSTScI commented 5 years ago

I have put new files into the directory /ifs/jwst/wit/mirage_data/niriss/GRISM_NIRISS with the expected NIRISS WFSS background in electrons/second for the ETC medium background spectrum defined some years ago (i.e. this is not the same as the current ETC background that is position dependent...)

The file names are

f090w_gr150c_medium_background.fits f150w_gr150c_medium_background.fits f090w_gr150r_medium_background.fits f150w_gr150r_medium_background.fits f115w_gr150c_medium_background.fits f158m_gr150c_medium_background.fits f115w_gr150r_medium_background.fits f158m_gr150r_medium_background.fits f140m_gr150c_medium_background.fits f200w_gr150c_medium_background.fits f140m_gr150r_medium_background.fits f200w_gr150r_medium_background.fits

I removed the previous normalized background files. These are slightly better versions made last week.

bhilbert4 commented 5 years ago

NIRISS WFSS was enabled in #254. Note that #278 discusses countrates as well as backgrounds for this mode. Closing this issue. Further developments will be discussed in #278.