NIRCam team to review and agree with photom reference file updates

[stscijgbot]

Issue JP-712 was created by Alicia Canipe:

Please review the epic ticket for photom reference file updates and provide any comments or feedback. When you are done with your review, please close this ticket (or let Alicia know the task is completed).

[stscijgbot]

Comment by Bryan Hilbert: After conferring a bit with [~lajoie], we found that observations using the two DHS elements are categorized as NRC_IMAGE even though the DHSs disperse the incoming light. So the imaging mode photom reference files will have to contain rows for the DHSs. And since they disperse the light we will also need to keep the wavelength and relresponse columns in the imaging mode photom reference file, in order to support them.

[stscijgbot]

Comment by Alicia Canipe: According to Howard's comment on the MIRI ticket (https://jira.stsci.edu/browse/JP-708?focusedCommentId=418526&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-418526) I think we will need to keep the columns, regardless.

[stscijgbot]

Comment by Charles-Philippe Lajoie: Note that this is based on review of DMS archival products from past WF Rehearsals (e.g. PID 00747).

[stscijgbot]

Comment by Bryan Hilbert: Interesting, thanks. The last time I spoke to [~bushouse] about this (long ago), he said that I needed to pad out the wavelength and relresponse columns to contain the expected number of elements (3000). I've been doing that with all zeros for the imaging modes. Should I instead use empty columns? [~hodge] said at the meeting this morning that an empty column would be a problem.

[stscijgbot]

Comment by Howard Bushouse: Yes, the columns must be present and they must NOT be empty. Unfortunately (again due to the brain-dead schema interface to FITS tables) we have to use a hardwired length for column arrays (e.g. 3000) with any or all unused values set to zero (actually it doesn't matter what value you set them to, but zero is the most obvious thing that indicates unused values). When I said in my earlier message that you set NELEM=0 to indicate that the wavelength and relresponse arrays are not to be used, I was referring to the value put in the table column that has the name "NELEM". I didn't mean to set the length of the wavelength and relresponse arrays to zero. If the column NELEM has a value of 0, the photom step won't bother doing anything with the data in the WAVELENGTH and RELRESPONSE columns.

[stscijgbot]

Comment by Philip Hodge: It doesn't look as if it is possible to create a truly empty column in a FITS BINTABLE.  (With an ASCII table, it is possible to have a column that has a name and possibly other information, such as units, but no data, not even dummy values.)

Howard is correct that the columns defined in the schema for a jwst datamodel must be padded to the number given in the schema.  For NircamPhotomModel, the "wavelength" and "relresponse" columns are specified as being arrays of 3000 elements.  We are hoping to find a way around this limitation, but for the time being, the columns must have the specified shape.  They don't have to be padded with any specific value; the "nelem" column specifies how many elements (which can be 0) are to be used.

[stscijgbot]

Comment by Bryan Hilbert: Thanks. I'll keep doing what I'm doing now.

I have another question about the contents of the photom reference file. For cases where nelem is non-zero, what exactly is the photmjsr value expected to be? Is it exactly like the imaging case, where it is the conversion factor as calculated over the entire bandpass? Or is it the conversion factor calculated at a particular (e.g. pivot) wavelength, and the values in relresponse are all relative to that value?

Looking at some of NIRISS's recent photom reference files, it looks like it is the former, but I wanted to be sure. 

[stscijgbot]

Comment by Bryan Hilbert: And assuming the former case above, then the relresponse values should be normalized to 1.0 at the pivot wavelength?

[stscijgbot]

Comment by Howard Bushouse: You can make the photmjsr value represent whatever you want. It simply gets multiplied into the pixel values as a scalar along with the relresponse values (i.e.  flux(lambda) = pix(lambda) photmjsr relresponse(lambda)). I think the original intent was that it would represent something like the calibration at the pivot wavelength, in which case relresponse could have values greater than and less than 1. Or you can take the approach of normalizing the calibration over the whole bandpass to 1, let photmjsr be the absolute cal at the normalized peak, and then relresponse would have a peak of 1.0 and decrease at other wavelengths. Or you can set photmjsr=1.0 and put all of the actual calibration into relresponse (although that's not the way it was envisioned to be used). It all works out the same mathematically.

[stscijgbot]

Comment by Alicia Canipe: [~hilbert] can you add the keywords PIXAR_SR and PIXAR_A2 to the spectroscopic photom files? 

From Phil: {quote}        The photom step copies keywords for the pixel area from either the         photom or area reference file to the output file; however, this is only         done for imaging data.  As far as I can tell, none of the spectroscopic         area reference files include these keywords anyway.  The extract_1d step         needs the pixel area (solid angle) in order to convert the input surface         brightness data to flux density, and currently there's no good way to         get this information.  I would like to request that the new photom         reference files for spectroscopic modes include the keywords for pixel         area; the photom step can easily be modified to copy those keywords to         output.{quote}

[stscijgbot]

Comment by Bryan Hilbert: Will do.

[stscijgbot]

Comment by Bryan Hilbert: [~bushouse] [~hodge], I have one more question. In my code from the last time I was looking at photom reference files, I calculated photmjsr by taking the ratio of the signal in MJy/str to that in counts/sec, and then I have this "conversion_factor" that I'm multiplying into photmjsr before saving it. I have the conversion factor set to be 1000, with a note saying that I need to find the real value. Note that I have this only in the WFSS photom calculations, and not the imaging calculations. Does that make any sense to you? photmjsr should simply be the ratio, right? Multiplying the ratio by some factor makes no sense to me unless the pipeline is expecting photmjsr to be in some strange units.

Ignoring the conversion factor for the moment, I'm simply creating an array of signal in MJy/str at all wavelengths. I convert this to an array of countrates at all wavelengths. I multiply by the dispersion since each pixel will see signal from a small range of wavelengths. (I was multiplying only the countrate array by the dispersion, but now that I think about it, the MJy/str array should also be multiplied by dispersion, right? In which case it should cancel out.). Then I take the ratio of these two arrays, and finally I normalize by the value of the ratio at the pivot wavelength. The ratio at the pivot wavelength is photmjsr, and the array of normalized ratios is relresponse.

 

[stscijgbot]

Comment by Howard Bushouse: If I remember correctly, the original flux calibration plan specified final units of MJy/str for imaging modes, but Jy or mJy for spectroscopic. So perhaps your conversion factor for spectroscopic modes is somehow related to that? Of course 1000 would only get you between Jy and mJy, not MJy.

[stscijgbot]

Comment by Philip Hodge: [~hilbert], by "MJy / sr at all wavelengths", do you mean the flux density integrated over all wavelengths?  Note that a jansky is per unit of frequency, not wavelength:  1.e-26 W m^-2 Hz^-1.  Are you taking that into account when you calculate the value for each pixel?

[stscijgbot]

Comment by Bryan Hilbert: I'm letting synphot do the heavy lifting in my code. I provide a filter bandpass and a source spectrum. Using the source spectrum only, Synphot calculates the flux density of the source at each of an array of wavelengths, and then converts those values to MJy. I divide that by the pixel area in strrad. Then I have synphot calculate the count rate of the source convolved with the filter bandpass at each wavelength in my array. This is multiplied by the dispersion. (I think it does make sense then that dispersion is multiplied only into the count rate and not the flux density values.) I then create the ratio of flux density to count rate. I take the ratio value at the pivot wavelength to be photmjsr, and normalize the ratio array by this value in order to create the relresponse array.

Phil, do you know if photmjsr in the spectroscopic case is supposed to be mJy or Jy? Also, for the NIRCam DHS observations, which currently have an exptype of NRC_IMAGE, but do contain dispersed data and therefore do use the relresponse column, would those still be considered spectroscopic such that the pipeline will assume photmjsr is in units of mJy or Jy? Or since the data are NRC_IMAGE will the pipeline assume MJy?

[stscijgbot]

Comment by Philip Hodge: My understanding is that after running the photom step, all data (but maybe not IFU?) should be in megajanskys / steradian, so photmjsr should be (MJy / sr) / (counts / s).

Should DHS data be treated the same as any other slitless spectroscopy data?  We might need some info other than EXP_TYPE to tell us to run extract_2d and subsequent steps (such as photom).

[stscijgbot]

Comment by Alicia Canipe: From the CalWG specifications, (MJy/sr)(DN/s) is correct ([https://outerspace.stsci.edu/display/JWSTCC/Vanilla+Spectral+Flux+Calibration):] "The photom {color:#172b4d}multiplies the images by calibration factors in units of (MJy/sr)/(DN/s) that are a function of wavelength". I have been hearing rumors that there are on-going discussions about the units for spec flux cal, but (for now) this is what the CalWG decided. {color}

[stscijgbot]

Comment by Bryan Hilbert: Thanks! I'll keep everything in MJy.

My (rather naive) impression is that DHS data probably should be treated the same as WFSS data. I say this never having seen actual DHS data though. [~lajoie] has promised to give me a tutorial at some point. At the moment, I think DHS observations are used only for WFSC, and I'm not sure what level of data reduction they are taken to. It may be that by default the processing stops after caldetector1. Charles would know better than I.

I do know that there are ongoing discussions to modify the detector readout patterns for DHS data such that they would be more useful for science observations, so maybe in the future the data reduction steps will be more critical.

[stscijgbot]

Comment by Bryan Hilbert: There is no mechanism at the moment to specify multiple exposure types for a single photom reference file, right? (e.g. p_exptyp)

I'm thinking that there won't be any difference between NRC_IMAGE and NRC_TSIMAGE in terms of flux cal, and similarly for NRC_WFSS and NRC_TSGRISM. It's easy enough to make copies of the files and tweak the EXP_TYPE value, but I figured I should check.

[stscijgbot]

Comment by Steven Crawford: If you want to use the same file for multiple exp_types, then you can set that in the CRDS selections to not use 'exp_type' to select for those modes.   

[stscijgbot]

Comment by Philip Hodge: EXP_TYPE is not a selection criterion for the NIRCam photom reference file, so you should be able to use the same reference file for NRC_IMAGE and for NRC_TSIMAGE.  You might need a different file for GRISM data (but yes, the same file for NRC_WFSS and NRC_TSGRISM), because the pixel area might be different between direct image and grism, and the keywords are in the primary header.

[stscijgbot]

Comment by Alicia Canipe: Sorry, I think I don't quite understand – in this case, how would CRDS know to pull either the NRC_IMAGE/TS_IMAGE file or the NRC_WFSS/TSGRISM file, if EXP_TYPE isn't in the selection criteria?

[stscijgbot]

Comment by Philip Hodge: You're right, [~acanipe], CRDS wouldn't be able to select different files for imaging vs spectra.  The NIRCam photom reference file is selected by INSTRUME, DETECTOR, and the date and time of observation.

[stscijgbot]

Comment by Bryan Hilbert: I was assuming that EXP_TYPE would be added to the selection criteria for the photom reference file, since we are now creating a separate file for each EXP_TYPE.

[stscijgbot]

Comment by Alicia Canipe: [~scrawford] if we need a way to determine the appropriate reference file between imaging and spec modes, it seems that we would need EXP_TYPE in the selection criteria. Would it be possible to keep EXP_TYPE in the criteria, and say for EXP_TYPE in [NRC_IMAGE, NRC_TSIMAGE] use the imaging photom file, and vice versa for the spec file? 

[stscijgbot]

Comment by Steven Crawford: [~acanipe] Yes you should be able to do that and select only on the exposures that would be relevant for that case. 

[stscijgbot]

Comment by Steven Crawford: [~hodge] I think I didn't forward on the information on to you, but the proposal is to update the CRDS selection to include exp_type so that it can select different modes

[stscijgbot]

Comment by Bryan Hilbert: So I'll (manually for the moment) add a EXPTYPE header keyword. Or if we're going to allow multiple exposure types for a single file, should it follow the P* convention?

(e.g. P_EXPTYP = NRC_IMAGE|NRC_TSIMAGE|)

[stscijgbot]

Comment by Steven Crawford: I believe the P_* convention should be used, but I've add [~rdiaz] and [~mcmaster] for any of their comments on this

[stscijgbot]

Comment by Rosa Diaz: There is a keyword for this. It is P_EXP_TY. SO for your files it should be P_EXP_TY=NRC_IMAGE|NRC_TSIMAGE|

 

 

[stscijgbot]

Comment by Bryan Hilbert: I'm running into another complication when planning out different files for different exp_types.

NRC_IMAGE and NRC_TSIMAGE use the same optics and can be combined in a single file. Similarly NRC_WFSS and NRC_TSGRISM can be combined. NRC_CORON and NRC_TACONFIRM can also be grouped together in their own file. However, target acquisition exposues can be selected within the time series, grism time series, and coronagraphy templates in APT. TA images can be collected using the imaging optics as well as the coronagraphy optics. (There are no TA images using the grisms, so no need to worry about that option.) But this means a photom reference file for NRC_TACQ would have to contain all of the imaging mode information as well as the coronagraphy information (or at least subsets of the filters allowed for TA). At that rate the options would be to create a NRC_TACQ file that repeats the information in the NRC_IMAGE and NRC_CORON reference files, or combine them all into a single reference file. Avoiding the repetition of data seems the wiser choice to me, but this means NIRCam would go back closer to the old model, where flux cal information for many modes is combined in a single file. Only the WFSS and TSGRISM modes would be split out into a different file (NRC_WFSC can also be its own file, assuming folks want to flux calibrate their WFSC data). Also, this is assuming that a TA exposure will always be labeled with NRC_TACQ regardless of what APT template it is contained within. 

[stscijgbot]

Comment by Alicia Canipe: John tried to argue for not processing TA exposures past level 1b or 2a because of reference files: https://jira.stsci.edu/browse/JSOCINT-209 but I think ultimately the decision was to just skip the dark step and process them through level 2b (same for TA_CONFIRM). I believe all TA exposures will have EXP_TYPE=NRC_TACQ, at least. I think since NRC_TACQ exposures are the odd man out and all will have the same exposure type, maybe it makes sense to group them together in one file?

[stscijgbot]

Comment by Bryan Hilbert: A little more information:

TA images in the imaging or grism time series template are always taken with the F335M filter and CLEAR pupil elements.

TA images in coronagraphy are taken with the F210M or F335M filters, and one of the two Lyot stops in the pupil wheel.

So a NRC_TACQ-only photom file would repeat the F335M/CLEAR row from the imaging photom file, and would repeat the 4 rows containing F210M/F335M and the round/bar Lyot stops from the coronagraphy photom file.

[stscijgbot]

Comment by Rosa Diaz: [~hilbert], I am trying to understand what you mean. Are you saying that there will be one Photom file for all TAs but because the FILTER/PUPIL combination is different for Imaging, or coronography are different you have to repeat rows? If that is the case

[~bushouse] or [~hodge], do we have "wild cards" for any of these; i.e. cases where you can use N/A or ANY (at the row level) and hece apply for any FILTER or PUPIL not specified?

[stscijgbot]

Comment by Bryan Hilbert: I don't think a wildcard would work, because we would always need a row for a specific filter and pupil.

For example, you can have a NRC_TSIMAGE exposure with F335M/CLEAR. You can also have a NRC_TACQ image with F335M/CLEAR. Similarly, you can have an NRC_CORON exposure with F335M/MASKBAR and you can also have a NRC_TACQ exposure with F335M/MASKBAR. This can be solved by having a F335M/MASKBAR row in a single flux cal file that is selected for both NRC_CORON and NRC_TACQ. But since you can also have a NRC_TACQ exposure that is F335M/CLEAR, you would need to include this row in the flux cal file as well. This basically means you'd end up with a flux cal file that has all the imaging and coronagraphy rows in it.

The other solution is to keep separate files for each exposure type, but repeat some of those rows in the two files. Since NRC_TACQ exposures are only allowed to use 2 filters, it seems easier to have a dedicated NRC_TACQ flux cal file that contains a copy of the F335M/CLEAR row from the NRC_TSIMAGE flux cal file, and a copy of the F335M/MASKBAR row from the NRC_CORON flux cal file.

[stscijgbot]

Comment by Howard Bushouse: IMHO, an image is an image is an image, whether it's labeled as NRC_IMAGE, NRC_TACQ, or NRC_CORON and regardless of whether you're doing science on it or the telescope FSW used it for pointing. The flux calibration depends on detector, filter, and pupil ... period. The photons don't care what EXP_TYPE we're writing into the FITS header. So personally I'd put all of these imaging-like modes in a single photom ref file, and expand the list of values in the P_EXP_TY keyword, rather than maintaining duplicate entries in multiple ref files. In that case the only separate photom ref files for NIRCam might be just imaging (including TSIMAGE and WFSC) and spectroscopy (WFSS and TSGRISM).

[stscijgbot]

Comment by Bryan Hilbert: Also, one of our filter throughput curves has a small red leak beyond the end of the main bandpass. In the wavelengths between the main bandpass and the red leak, the throughput is exactly 0. This means that the count rate at these wavelengths is also zero, which leads to relresponse values of infinity. I looked at the photom code, and it looks to me like it won't crash due to these values. It'll just convert the signal at those wavelengths to infinity. But I'm still wondering if it would be better to convert any infinite relresponse values to some large number manually before saving the file?  [~bushouse] [~hodge] 

[stscijgbot]

Comment by Philip Hodge: IMHO the relresponse values where the throughput is 0 should be 0, but these pixels should be flagged in the DQ extension of the science data with DO_NOT_USE.

[stscijgbot]

Comment by Bryan Hilbert: Since the flux cal value is going to be multiplied into the image, making it 0 in these cases makes sense I guess. Flagging the corresponding pixels with DO_NOT_USE also makes sense. You could probably argue the same thing for any relresponse value higher than some threshold (like those at the edges of the bandpass).

[stscijgbot]

Comment by Alicia Canipe: Final format decisions are in https://jira.stsci.edu/browse/JP-1004 and https://jira.stsci.edu/browse/JP-1018. Closing this subtask since the files are in the process of delivery, so the review is complete.