Convert IFU extract1d and apcorr reference files from fits to asdf

[stscijgbot-jp]

Issue JP-1749 was created on JIRA by Jane Morrison:

A new extract1d reference file for IFU data has been created. This preliminary file has been supplied by the MIRI INS team (David Law). JP-1736 (closed) created a datamodel for this new reference file. The reference file is a fits file containing two extensions. The first extension contains basic data (single values in a table). The second extension contains a table with integer values and five columns with arrays (size set by integer value in the table). This type of data may not be easily represented by an asdf file and it may be appropriate to leave it as a fits file. 

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Jane Morrison We wanted to have the apcor as a function of wavelength, aperture size, and "pixphase" (where within a pixel the PSF is centered at a given wavelength).  The latter two parameters are really only relevant for MOS data, though, because of the generally small aperture sizes that are needed.  For IFU, we don't need to worry about pixphase because the aperture radius should be big enough for that to have a negligible effect on the extracted flux.  I don't think we need multiple aperture radii, either, so both the extract_1d and apcorr reference files should just be 1D vectors per disperser.

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

James Muzerolle

Just double checking. The extract1d radius size is 0.5. So I assume I should grab the second column in the apcorr radius reference file (that is in CRDS) when making the new apcor reference file (that one is also 0.5)

So the format of the apcorr and apcorr_err  in the current one on CRDS is (3,2048,3)

I have been pulling  out (0,:,0) to use for the ID apcor for the new IFU apcor ref file - is that the correct column ?

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Jane Morrison Yes, I think that's right (although, because this is a dummy file, all APCORR values are set to 1, so any column could be used).  I just noticed that APCORR_ERR also contains all 1s; that should really be 0 since we don't have real measurements yet.

[stscijgbot-jp]

Comment by David Law on JIRA:

James Muzerolle Would it be possible to keep the aperture size dimension in the apcorr reference files for NIRSpec?  We just changed both the MIRI reference files and the pipeline code to match how NIRSpec had two-dimensional arrays, and it would be ideal not to have the complexity of needing to handle both 1d and 2d cases (now swapped to NIRSpec/MIRI respectively instead of MIRI/NIRSpec...).  We aren't using this functionality for MIRI at the moment (since the aperture correction values are the same for all choices of radius at a given wavelength), but it's now trivial to enable it in future with a simple reference file update.

[stscijgbot-jp]

Comment by David Law on JIRA:

Jane Morrison Will try to look back at this towards the end of the week

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

David Law Sure, that's fine.  Sorry for the confusion, this is part of the drawback to us initially combining MOS & IFU into the same data model.  I'll just need to modify the latest version of the extract_1d reference file to add a dimension for multiple radii.

 

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

James Muzerolle David Law

I think I have gotten confused. The extract_1d reference file  for MIRI has 1 D data for radius. David am I looking at the the wrong file ?  I think what David means is the radius in the apcorr file as well as the apcor correction has 2 dimensions. 

 

[stscijgbot-jp]

Comment by David Law on JIRA:

Yes, I meant just the apcorr file for 2 dimensions

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Jane Morrison David Law Ok, now I'm confused!  If you're going to allow for multiple radii in the apcor reference file, don't you also need it in the extract_1d file?

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

James Muzerolle David Law So you can look at the MIRI fits reference files - the extract1d and apcor on █████████████████████████████████ These files have been converted to asdf and it is the asdf files I am using to test the code.

It seems to me (David can correct me) that the extract1d Reference files is a 1D array that varies with wavelength from about 0.5 to 2.75. The radius in the apcorr file contains 3 values per wavelength that I believe fall in this range.

You two can figure out what you want in these reference files, but it would make the code easier to maintain if they have the same format. For now I am take my routine that converts the nirspec apcor fits file to asdf and just put in some dummy extra arrays. We can circle back later and finalize how to do this once we have a final format on the reference files

 

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

James Muzerolle

So here is some more information. The extraction radius size is saved and past to the apcorr correction step.

From the apcorr file we have a 2D radius (3,2048) and a 2D apcorr (3,2048) At the wavelength in question we pull out the apcorr radius values and apcorr correction  values.

using the extract1d save radius we interpolate what the correction should be.  I have taken the nirspec apcorr file and used the 2d radius values and pulled in the 2d corresponding apcorr values and  made a new reference file.

Here is a cut and paste of the code in apply_corr.py for finding the correction: (I hope this makes sense now)

 

 def find_apcorr_func(self, iwave ,radius_ifu):

at ifu wavelength plane (iwave), the extraction radius is radius_ifu                                       

pull out the radius values (self.size)  to use in the apcor ref file for this iwave                        

        radius_apcor = self.size[:,iwave]

        temparray=self.apcorr[:,iwave]

        fap=interp1d(radius_apcor,temparray,fill_value="extrapolate")

        correction=fap(radius_ifu)

        self.apcorr_correction.append(correction)

        return

 

 

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Ah, I understand now, thanks Jane Morrison!  So, you have apcor values for three different default radii per wavelength, and interpolate at the radius actually used.  I presume that this is to accommodate any feasible radius a user might want to use, and the extract_1d reference file specifies only a single radius per wavelength?  I was initially confused thinking that MIRI wanted to do multiple extractions using multiple radii.  Since that is not the case, I don't need to create a new extract_1d file with multiple columns.

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

James Muzerolle David Law

I have finalized the datamdoels. I have converted the MIRI and NIRSpec extract1d and apcorr ref file  to asdf to work with these data models. The GitHub PR that contains the new datamodels and code changes in extract_1d is found in PR#⁠5383

You can find an example of the new reference files in ████████████████████████████████

I have code to convert both the NIRSpec and MIRI extract1d fits files to asdf - see convert_ifu_extract1d_dict.py This creates: 1. miri-extract1d-59121_dict.asdf    2. jwst_nirspec_extr1d_0001_dict.asdf

Code to convert MIRI apcorr fits to asdf: convert_ifu_miri_apcorr_dict.py .   This creates: miri-apcorrpar-59136_dict.asdf

Code to convert NIRSpec apcorr fits to asdf: convert_ifu_nirspec_apcorr_dict.py (Note the table to use for data with more than one filter or more than grating needs to be defined and set by the NIRSPEC team. I just copied the clear Prism apcorr table to use in that case) This creates: jwst_nirspec_apcorr_0004_dict.asdf

I have provide the code so you can see how to make the files to be inline with the data models.  Feel free to take this code and  use the parts you need to create the reference files. We can iterate on further changes but it would be good if you guys could look at PR#⁠5383 and the new asdf reference files and provide feedback on what you want changed or added. 

 

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

datamodels defined in PR #⁠5383 code for convert reference files found in ████████████████████████████████

[stscijgbot-jp]

Comment by Alicia Canipe on JIRA:

Hi all:  David Law  Jane Morrison  James Muzerolle

Just a heads up (you may already know this) – new reference files for Build 7.7 should be delivered by November 24th, if you think your files will be ready.