MIRI Subarrays 390 Hz EMI

[stscijgbot-jp]

Issue JP-3248 was created on JIRA by Michael Regan:

The majority of the MIRI subarrays have an 390 Hz EMI noise pattern in the raw data. This 390 Hz maps exactly to 256 pixels and has a constant amplitude of +- 4 DN. The effect of this is to imprint this into the rate images. For short integrations in LRSSLITLESS the correlated noise from this is quite apparent in the rate images. For longer integrations the net effect is to increase the read noise by ~20%.

The long term plan is a change to the sizes and locations of the subarrays to get the frame times to be in phase with the 390 Hz like the full frame images. For the previous and near term observations this can be fixed by adding a new step to the pipeline. A prototype has been implemented and shown to remove the 390 Hz to a fraction of a DN.

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

ok, so just to confirm Rosa Diaz Eddie Bergeron Michael Regan, you want me to change the pipeline code to get a the reference file given a subarray name and detector, correct?

This means the work on the CRDS side needs to stop for the moment since the format of the multiple reference files will be different. I will modify the reference file schema in the datamodels repo and create a new ASDF example file.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

No, it doesn't have to be multiple reference files. There can still be just one reference file containing all the frequency/wave/subarray/detector combos as separate records. The emicorr code itself can then read the entire asdf reference file, look through it to see which record(s) correspond to the current dataset (subarray/det) being calibrated and only apply the appropriate corrections. It can even skip the emicorr step if there are no matching cases in the reference file. You could effectively turn the step off by providing a reference file with no matching cases.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Unrelated to the current discussion, here are some figures I made to demonstrate the performance of the idl tool. Some of these example figures can be reproduced using the instructions at the top of fix_miri_emi.pro.

First figure shows the effect of recursive calls for different frequencies on some CDS images and the resulting reduction in RMS readnoise. Note the 28% reduction from 390Hz correction alone. Most of the additional noise power is there, however it is worth considering that the 10Hz noise, although lower in amplitude, is correlated over many more pixels due to the longer period. It's appearance is very similar to the lowest frequency component of 1/f seen in the NIR detectors.

Second figure shows power spectra of the CDS images in the first figure. Note a couple of additional discrete frequencies that might be worth investigating. It is interesting that the power spectrum of BRIGHTSKY subarray images, which are in-phase for 390Hz and don't need correction, don't have any of these other associated frequencies. Same is true of fullframe fast and slow images. I believe this is a strong reason to consider defining new MIRI subarrays that are in-phase for 390. That may clear up all the other crud for free. We'll always have 10Hz of course.

Two other two figure just show what 10Hz looks like in a dark and an external science exposure. There's a small forest of 10Hz and its aliases. This is easiest to see in fullframe fast exposures because there are several cycles per frame to quickly bring up the S/N.

Finally, one more example showing the utility of the refwave amplitude scaling option, which I think should be the default behavior.

!miri_emi_masklyot_example.png|thumbnail! !miri_emi_power_spectrum_example.png|thumbnail! !miri_emi_10hz_power_spectrum_example_dark_ccc_closed.png|thumbnail! [^miri_emi_10hz_power_spectrum_example_10um_science.png]

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

I spoke to Taylor Bell after his talk on their "Eureka!" JWST TSO pipeline at the data products workshop last week. He mentioned that they have their own 390Hz correction. He also noticed some other frequencies and thought it was odd that they didn't seem related to 390 at all (not integer multiples). Their correction is 390-only.

He also mentioned that they see a small phase-jitter on the 390Hz signal over some number of frame times. I hadn't noticed that but haven't really looked at that level. Was mostly concerned with the phase coherence across long time intervals (throughout long exposures). When I get a chance I'll take a look. In any case, it can't be large and would be difficult to measure and correct in a general purpose pipeline. Maybe for large TSO sets. They are using periodograms rather than fft powerspectra to look for frequencies in the noise...right out of the TSO toolbox! :)

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

For example, any phase jitter would have to fall within the noise envelope of this wave phased over many, many frames and integrations:

!miri_390hz_phased_example.png|thumbnail!

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

Maria Pena-Guerrero  regarding your comment above on [#comment-673000]. Waiting to hear for you so Michael Regan can create the reference file. We should try to deliver by Wednesday or Thursday to make sure all is ready for the build.

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

Maria Pena-Guerrero My concern is with the code based dictionary. I think that this step would be more flexible if the dictionary were implemented via a combination of CRDS and reference file parameters. So the selection of frequencies for each subarray would be independent  of the code. I don't think this needs to be done for the first delivery though. So, I'd just like to get the 390 Hz correction is this build and then we can figure out how to get the others implemented. 

I've uploaded a draft reference file to this ticket. It is jwst_miri_emimodel.asdf. This only has a 390 Hz waveform in it. Let me know how if it works.

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

Michael Regan Eddie Bergeron The problem with removing the dictionary from the pipeline code is that it will no longer be able to run if there is no reference file (i.e. find the on-the-fly phase amplitudes). Should I go ahead and remove the dictionary and place that information in the reference file?

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

I understand that things won't work without the dictionary. I think we should stay with the current organization and make an update in the next build.

 

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

Can you set those values just as default if there is no reference file? There will be a reference file once Michael Regan delivers it and should be before build 10.0. Unless you don't have time to do this now and he does not have time to make it before then. I would strongly recommend to allow for those to be changed with a reference file. Can that be done?

Rossy

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

Rosa Diaz I would like a clean 390 Hz only for the first delivery. The other noise terms are quite small and I'd like us to only work on one thing at a time. Debugging will be much easier if there is only one correction being made.

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

Michael Regan Ok, but even if we leave the 390 you still need to submit the file with the current format Maria sets, correct? Let's get that defined so we can deliver this files ASAP. Thanks.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

A single frequency 390-only first delivery reference file is fine. Mike has posted such a file to this ticket.

Independent of that....

If I understand the "self-correction" issue (i.e. the on-the-fly phase amplitudes that Maria mentioned), the problem for self-correction is there is no way to pass the desired phasing frequency other than through a dictionary value. If that's correct, putting that dictionary into the reference file is still fine. Any user wishing to self-correct would have to pass a reference file with that frequency (e.g. 390. hz) and just "sets the switch" (sets a parameter) for self-correction. The reference waveform (i.e. the "phased amplitude") in the reference file would be ignored in that case, but at least the frequency can be pulled from the dictionary in the reference file. Users could also make their own reference file if they wanted to use a different frequency, even for self-correction. If there is no reference file, the emicorr step should be skipped by the pipeline. That is, no hardcoded default emi frequencies for self-correction.

So the pipeline user always provides a reference file with a freq+refwave and uses both, or requests a self-correction using the frequency in the reference file and simply ignores the refwave in the reference file. Pipeline default should be not self-correction.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Mike posted a draft 390-only reference file to this ticket earlier today:

[^jwst_miri_emimodel.asdf]

Have a look and let him know if there are any issues with it. If Maria decides to move the dictionary into the reference file, please post the new template here for Mike to modify, or pull the 390hz reference wave from the above.

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

Eddie, thanks for pointing that out. Will check the file.

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

Michael Regan I came up with this new format that contains the phase amplitudes and the dictionary (see [^jw01386007001_04101_00006_mirimage_emi_ref_waves_new.asdf]). If this file format works for you I will make changes in the pipeline code to get the info from the file and have some default values if there is no file).

the format is the following:

• Each frequency entry, named e.g. Hz164, is a dictionary containing two keyes, the exact frequency and the phase amplitudes array

• There are 10 frequency dictionaries: Hz10, Hz10_slow_MIRIMAGE, Hz10_slow_MIRIFULONG, Hz10_slow_MIRIFUSHORT, Hz164, Hz218a, Hz218b, Hz218c, Hz390, Hz390_sub128 Each frequency dictionary contains the following:

  
  'Hz10': {
  'frequency': 10.039216,
  'phase_amplitudes': array([ 0.59307419, ..., 0.40522677]) 
  }```

• There is a "subarray_cases" dictionary containing the following:

subarray_cases = {
 'SLITLESSPRISM': {
     'rowclocks': 28,
     'frameclocks': 15904,
     'freqs': {
        'FASTR1': [390.625, 164.9305, 10.039216],
        'SLOWR1': {
           'MIRIMAGE': [390.625, 164.9305, 10.039216],
           'MIRIFULONG': [390.625, 164.9305, 10.039216],
           'MIRIFUSHORT': [390.625, 164.9305, 10.039216] } } },

  'MASKLYOT': {
     'rowclocks': 90,
     'frameclocks': 32400,
     'freqs': {
        'FASTR1': [390.625, 164.9305, 10.039216],
        'SLOWR1': {
           'MIRIMAGE': [390.625, 164.9305, 10.039216],
           'MIRIFULONG': [390.625, 164.9305, 10.039216],
           'MIRIFUSHORT': [390.625, 164.9305, 10.039216]}}},

 'SUB64': {
    'rowclocks': 28,
    'frameclocks': 8512,
    'freqs': {
       'FASTR1': [390.625, 164.9305, 10.039216],
       'SLOWR1': {
          'MIRIMAGE': [390.625, 164.9305, 10.039216],
          'MIRIFULONG': [390.625, 164.9305, 10.039216],
          'MIRIFUSHORT': [390.625, 164.9305, 10.039216]}}},
 
 'SUB128': {
   'rowclocks': 44,
   'frameclocks': 11904,
   'freqs': {
      'FASTR1': [390.625, 164.9305, 10.039216],
      'SLOWR1': {
         'MIRIMAGE': [390.625, 164.9305, 10.039216],
         'MIRIFULONG': [390.625, 164.9305, 10.039216],
         'MIRIFUSHORT': [390.625, 164.9305, 10.039216]}}},

 'MASK1140': {
   'rowclocks': 82,
   'frameclocks': 23968,
   'freqs': {
      'FASTR1': [390.625, 164.9305, 10.039216],
      'SLOWR1': {
         'MIRIMAGE': [390.625, 164.9305, 10.039216],
         'MIRIFULONG': [390.625, 164.9305, 10.039216],
         'MIRIFUSHORT': [390.625, 164.9305, 10.039216]}}},
 
'MASK1150': {
  'rowclocks': 82,
  'frameclocks': 23968,
  'freqs': {
     'FASTR1': [390.625, 164.9305, 10.039216],
     'SLOWR1': {
        'MIRIMAGE': [390.625, 164.9305, 10.039216],
        'MIRIFULONG': [390.625, 164.9305, 10.039216],
        'MIRIFUSHORT': [390.625, 164.9305, 10.039216]}}},
 
'FULL_FASTR1': {
  'rowclocks': 271,
  'frameclocks': 277504,
  'freqs': {
     'FASTR1': [164.9305, 10.039216]}},

'FULL_SLOWR1': {
  'rowclocks': 2333,
  'frameclocks': 2388992,
  'freqs': {
     'SLOWR1': {
        'MIRIMAGE': [164.9305, 10.039216],
        'MIRIFULONG': [164.9305, 10.039216],
        'MIRIFUSHORT': [164.9305, 10.039216]}}},

'BRIGHTSKY': {
  'rowclocks': 82,
  'frameclocks': 23968,
  'freqs': {
     'FASTR1': [164.9305, 10.039216],
     'SLOWR1': {
        'MIRIMAGE': [164.9305, 10.039216],
        'MIRIFULONG': [164.9305, 10.039216],
        'MIRIFUSHORT': [164.9305, 10.039216]}}},

'SUB256': {
  'rowclocks': 96,
  'frameclocks': 29952,
  'freqs': {
     'FASTR1': [164.9305, 10.039216],
     'SLOWR1': {
        'MIRIMAGE': [164.9305, 10.039216],
        'MIRIFULONG': [164.9305, 10.039216],
        'MIRIFUSHORT': [164.9305, 10.039216]}}}}```
To avoid/skip the emi correction you can either skip the step or the phase amplitudes array can be either empty or all zeros or the frequencies can be removed from the 'freqs' key for the corresponding subarray.

Does this format work for you?

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

Michael Regan if you decide to recreate file, please change the meta.reftype to "emicorr". The one you had before was not correct and I had to change it to be able to deliver.

Also SCSB is fixing something today before we can deliver so you still have time to recreate the file.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Hey Maria, I think this looks good. Plenty of flexibility to cover our current understanding. I presume it would be possible (in a future reference file, if necessary) to add the three detectors separately under FASTR1 just like you have for SLOWR1? Not needed now, but if we wanted to add separation by detector in fast mode will the code handle that as written? That is, could we just create a reference file with that breakout and it would work?

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Also, how does the code decide which frequency dictionary to use for a given subarray/detector case? They can't match by frequency, because some of the frequencies will be the same for different phase_amplitude definitions (e.g. Hz390 vs Hz390_sub128).

Seems like in the subarray_cases disctionary you'd want to use the frequency dictionary names rather than the actual frequencies.

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

Hi Eddie, you are correct. In fact I am working on a version that has the names instead of the frequencies. Also, I am quite new to the datamodels repo so I might have to tweak it a bit more so that it is generic enough to not have to put in a PR in that repo each time you submit a new reference file with new frequencies. Rosa Diaz I do not think we will be able to make this delivery because of that.

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

ok, slight modification to the reference file. The content is as follows in the news file [^jw01386007001_04101_00006_mirimage_emi_ref_waves_new-2.asdf]:

root (AsdfObject)
├─asdf_library (Software)
│ ├─author (str): The ASDF Developers
│ ├─homepage (str): http://github.com/asdf-format/asdf
│ ├─name (str): asdf
│ └─version (str): 2.15.1
├─history (dict)
│ └─extensions (list)
│   └─[0] (ExtensionMetadata) ...
├─frequencies (dict)
│ ├─Hz10 (dict)
│ │ ├─frequency (float): 10.039216
│ │ └─pahse_amplitudes (NDArrayType): shape=(128,), dtype=float64
│ ├─Hz164 (dict) ...
│ └─Hz390 (dict) ...
├─meta (dict)
│ ├─date (str): 2023-11-28T18:40:46.016
│ ├─filename (str): jw01386007001_04101_00006_mirimage_emi_ref_waves.asdf
│ ├─instrument (dict)
│ ├─model_type (str): EmiModel
│ └─telescope (str): JWST
└─subarray_cases (dict)
  └─MASK1550 (dict) ...

So the frequencies are now all under the key 'frequencies', and the 'subarray_cases' key looks the same, except using the frequency names instead of the numbers. I'll submit PRs for the datamodels repo and the pipeline repo.

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

Eddie Bergeron to answer your question:  "I presume it would be possible (in a future reference file, if necessary) to add the three detectors separately under FASTR1 just like you have for SLOWR1? Not needed now, but if we wanted to add separation by detector in fast mode will the code handle that as written? That is, could we just create a reference file with that breakout and it would work?" The answer is yes and no. With the new datamodel I am pushing in the PR (pending approval) the reference file is pretty flexible but the pipeline code would have to be slightly modified to expect a dictionary instead of a list (not a big deal at all).

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

Maria Pena-Guerrero Michael ReganPlease don't forget the meta.reftype :

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

Eddie Bergeronif you want something different per detector then we can have a different reference file per detector. Changing the selection criteria in CRDS does not need to wait for a build (in most cases)

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

Michael Regan Eddie Bergeron and Maria Pena-Guerrero So are we delivering the file Mike attached to this ticket before and not wait for any new datamodel for this build? We still have a couple of days

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Mike is working on a new version based on Maria's latest "new-2.asdf" template. Should have it posted here today or tomorrow.

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

Maria Pena-Guerrero Sorry for jumping back in so late. I'm confused about the new format. Your schema looks good to me. I think I did the wrong thing by reading in the .._new.asdf. It looks like this is still the old format.

Can you create an example asdf file that I could modify?

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

Michael Regan here is the newest version of the file [^jw01386007001_04101_00006_mirimage_emi_ref_waves_new3.asdf]

please let me know if you see something missing (Rosa Diaz I added reftype=emicorr).

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

I'm struggling with the datamodels. I can't seem to figure out how to print all the nodes in the example. The docs say that I can control this via the max_rows parameter using emi.info(). Is there an example of how to set max_rows? I can't even find the .info method.

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

nevermine, I was using max_row instead of max_rows.

 

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

[^jwst_miri_emimodel.asdf]Here's an update of the model with the waveforms for 10Hz and 390 Hz.

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

 Michael Regan Thanks for working on this file. I tried to certify it and failed. How come the reftype change to EMI file? ;) it should be emicorr. I will change it tomorrow and will certify again.

[stscijgbot-jp]

Comment by Meaghan McDonald on JIRA:

I have updated the file to have 'reftype' = 'emicorr'. -The file to be delivered is [^jwst_miri_emicorr_new.asdf].-

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

Michael Regan Rosa Diaz Meaghan McDonald neither one of those files has the new format expected by the datamodels repo. 

The file is expected to have the format as example file [^jw01386007001_04101_00006_mirimage_emi_ref_waves_new3.asdf]

^The contents should look like this:^

In [4]: af.info(max_rows=2000)
root (AsdfObject)
├─asdf_library (Software)
│ ├─author (str): The ASDF Developers
│ ├─homepage (str): http://github.com/asdf-format/asdf
│ ├─name (str): asdf
│ └─version (str): 2.15.1
├─history (dict)
│ └─extensions (list)
│   └─[0] (ExtensionMetadata)
│     ├─extension_class (str): asdf.extension.BuiltinExtension
│     └─software (Software)
│       ├─name (str): asdf
│       └─version (str): 2.15.1
├─frequencies (dict)
│ ├─Hz10 (dict)
│ │ ├─frequency (float): 10.039216
│ │ └─pahse_amplitudes (NDArrayType): shape=(128,), dtype=float64
│ ├─Hz10_slow_MIRIFULONG (dict)
│ │ ├─frequency (float): 10.039216
│ │ └─phase_amplitudes (NDArrayType): shape=(128,), dtype=float64
│ ├─Hz10_slow_MIRIFUSHORT (dict)
│ │ ├─frequency (float): 10.039216
│ │ └─phase_amplitudes (NDArrayType): shape=(128,), dtype=float64
│ ├─Hz10_slow_MIRIMAGE (dict)
│ │ ├─frequency (float): 10.039216
│ │ └─phase_amplitudes (NDArrayType): shape=(128,), dtype=float64
│ ├─Hz164 (dict)
│ │ ├─frequency (float): 164.9305
│ │ └─pahse_amplitudes (NDArrayType): shape=(128,), dtype=float64
│ ├─Hz390 (dict)
│ │ ├─frequency (float): 390.625
│ │ └─pahse_amplitudes (NDArrayType): shape=(128,), dtype=float64
│ └─Hz390_sub128 (dict)
│   ├─frequency (float): 390.625
│   └─phase_amplitudes (NDArrayType): shape=(128,), dtype=float64
├─meta (dict)
│ ├─date (str): 2023-11-28T18:40:46.016
│ ├─filename (str): jw01386007001_04101_00006_mirimage_emi_ref_waves_new2.asdf
│ ├─instrument (dict)
│ ├─model_type (str): EmiModel
│ ├─reftype (str): emicorr
│ └─telescope (str): JWST
└─subarray_cases (dict)
  ├─BRIGHTSKY (dict)
  │ ├─frameclocks (int): 23968
  │ ├─freqs (dict)
  │ │ ├─FASTR1 (list)
  │ │ │ ├─[0] (str): Hz164
  │ │ │ └─[1] (str): Hz10
  │ │ └─SLOWR1 (dict)
  │ │   ├─MIRIFULONG (list)
  │ │   │ ├─[0] (str): Hz164
  │ │   │ └─[1] (str): Hz10_slow_MIRIFULONG
  │ │   ├─MIRIFUSHORT (list)
  │ │   │ ├─[0] (str): Hz164
  │ │   │ └─[1] (str): Hz10_slow_MIRIFUSHORT
  │ │   └─MIRIMAGE (list)
  │ │     ├─[0] (str): Hz164
  │ │     └─[1] (str): Hz10_slow_MIRIMAGE
  │ └─rowclocks (int): 82
  ├─FULL_FASTR1 (dict)
  │ ├─frameclocks (int): 277504
  │ ├─freqs (dict)
  │ │ └─FASTR1 (list)
  │ │   ├─[0] (str): Hz164
  │ │   └─[1] (str): Hz10
  │ └─rowclocks (int): 271
  ├─FULL_SLOWR1 (dict)
  │ ├─frameclocks (int): 2388992
  │ ├─freqs (dict)
  │ │ └─SLOWR1 (dict)
  │ │   ├─MIRIFULONG (list)
  │ │   │ ├─[0] (str): Hz164
  │ │   │ └─[1] (str): Hz10_slow_MIRIFULONG
  │ │   ├─MIRIFUSHORT (list)
  │ │   │ ├─[0] (str): Hz164
  │ │   │ └─[1] (str): Hz10_slow_MIRIFUSHORT
  │ │   └─MIRIMAGE (list)
  │ │     ├─[0] (str): Hz164
  │ │     └─[1] (str): Hz10_slow_MIRIMAGE
  │ └─rowclocks (int): 2333
  ├─MASK1140 (dict)
  │ ├─frameclocks (int): 23968
  │ ├─freqs (dict)
  │ │ ├─FASTR1 (list)
  │ │ │ ├─[0] (str): Hz390
  │ │ │ ├─[1] (str): Hz164
  │ │ │ └─[2] (str): Hz10
  │ │ └─SLOWR1 (dict)
  │ │   ├─MIRIFULONG (list)
  │ │   │ ├─[0] (str): Hz390
  │ │   │ ├─[1] (str): Hz164
  │ │   │ └─[2] (str): Hz10_slow_MIRIFULONG
  │ │   ├─MIRIFUSHORT (list)
  │ │   │ ├─[0] (str): Hz390
  │ │   │ ├─[1] (str): Hz164
  │ │   │ └─[2] (str): Hz10_slow_MIRIFUSHORT
  │ │   └─MIRIMAGE (list)
  │ │     ├─[0] (str): Hz390
  │ │     ├─[1] (str): Hz164
  │ │     └─[2] (str): Hz10_slow_MIRIMAGE
  │ └─rowclocks (int): 82
  ├─MASK1150 (dict)
  │ ├─frameclocks (int): 23968
  │ ├─freqs (dict)
  │ │ ├─FASTR1 (list)
  │ │ │ ├─[0] (str): Hz390
  │ │ │ ├─[1] (str): Hz164
  │ │ │ └─[2] (str): Hz10
  │ │ └─SLOWR1 (dict)
  │ │   ├─MIRIFULONG (list)
  │ │   │ ├─[0] (str): Hz390
  │ │   │ ├─[1] (str): Hz164
  │ │   │ └─[2] (str): Hz10_slow_MIRIFULONG
  │ │   ├─MIRIFUSHORT (list)
  │ │   │ ├─[0] (str): Hz390
  │ │   │ ├─[1] (str): Hz164
  │ │   │ └─[2] (str): Hz10_slow_MIRIFUSHORT
  │ │   └─MIRIMAGE (list)
  │ │     ├─[0] (str): Hz390
  │ │     ├─[1] (str): Hz164
  │ │     └─[2] (str): Hz10_slow_MIRIMAGE
  │ └─rowclocks (int): 82
  ├─MASKLYOT (dict)
  │ ├─frameclocks (int): 32400
  │ ├─freqs (dict)
  │ │ ├─FASTR1 (list)
  │ │ │ ├─[0] (str): Hz390
  │ │ │ ├─[1] (str): Hz164
  │ │ │ └─[2] (str): Hz10
  │ │ └─SLOWR1 (dict)
  │ │   ├─MIRIFULONG (list)
  │ │   │ ├─[0] (str): Hz390
  │ │   │ ├─[1] (str): Hz164
  │ │   │ └─[2] (str): Hz10_slow_MIRIFULONG
  │ │   ├─MIRIFUSHORT (list)
  │ │   │ ├─[0] (str): Hz390
  │ │   │ ├─[1] (str): Hz164
  │ │   │ └─[2] (str): Hz10_slow_MIRIFUSHORT
  │ │   └─MIRIMAGE (list)
  │ │     ├─[0] (str): Hz390
  │ │     ├─[1] (str): Hz164
  │ │     └─[2] (str): Hz10_slow_MIRIMAGE
  │ └─rowclocks (int): 90
  ├─SLITLESSPRISM (dict)
  │ ├─frameclocks (int): 15904
  │ ├─freqs (dict)
  │ │ ├─FASTR1 (list)
  │ │ │ ├─[0] (str): Hz390
  │ │ │ ├─[1] (str): Hz164
  │ │ │ └─[2] (str): Hz10
  │ │ └─SLOWR1 (dict)
  │ │   ├─MIRIFULONG (list)
  │ │   │ ├─[0] (str): Hz390
  │ │   │ ├─[1] (str): Hz164
  │ │   │ └─[2] (str): Hz10_slow_MIRIFULONG
  │ │   ├─MIRIFUSHORT (list)
  │ │   │ ├─[0] (str): Hz390
  │ │   │ ├─[1] (str): Hz164
  │ │   │ └─[2] (str): Hz10_slow_MIRIFUSHORT
  │ │   └─MIRIMAGE (list)
  │ │     ├─[0] (str): Hz390
  │ │     ├─[1] (str): Hz164
  │ │     └─[2] (str): Hz10_slow_MIRIMAGE
  │ └─rowclocks (int): 28
  ├─SUB128 (dict)
  │ ├─frameclocks (int): 11904
  │ ├─freqs (dict)
  │ │ ├─FASTR1 (list)
  │ │ │ ├─[0] (str): Hz390_sub128
  │ │ │ ├─[1] (str): Hz164
  │ │ │ └─[2] (str): Hz10
  │ │ └─SLOWR1 (dict)
  │ │   ├─MIRIFULONG (list)
  │ │   │ ├─[0] (str): Hz390_sub128
  │ │   │ ├─[1] (str): Hz164
  │ │   │ └─[2] (str): Hz10_slow_MIRIFULONG
  │ │   ├─MIRIFUSHORT (list)
  │ │   │ ├─[0] (str): Hz390_sub128
  │ │   │ ├─[1] (str): Hz164
  │ │   │ └─[2] (str): Hz10_slow_MIRIFUSHORT
  │ │   └─MIRIMAGE (list)
  │ │     ├─[0] (str): Hz390_sub128
  │ │     ├─[1] (str): Hz164
  │ │     └─[2] (str): Hz10_slow_MIRIMAGE
  │ └─rowclocks (int): 44
  ├─SUB256 (dict)
  │ ├─frameclocks (int): 29952
  │ ├─freqs (dict)
  │ │ ├─FASTR1 (list)
  │ │ │ ├─[0] (str): Hz164
  │ │ │ └─[1] (str): Hz10
  │ │ └─SLOWR1 (dict)
  │ │   ├─MIRIFULONG (list)
  │ │   │ ├─[0] (str): Hz164
  │ │   │ └─[1] (str): Hz10_slow_MIRIFULONG
  │ │   ├─MIRIFUSHORT (list)
  │ │   │ ├─[0] (str): Hz164
  │ │   │ └─[1] (str): Hz10_slow_MIRIFUSHORT
  │ │   └─MIRIMAGE (list)
  │ │     ├─[0] (str): Hz164
  │ │     └─[1] (str): Hz10_slow_MIRIMAGE
  │ └─rowclocks (int): 96
  └─SUB64 (dict)
    ├─frameclocks (int): 8512
    ├─freqs (dict)
    │ ├─FASTR1 (list)
    │ │ ├─[0] (str): Hz390
    │ │ ├─[1] (str): Hz164
    │ │ └─[2] (str): Hz10
    │ └─SLOWR1 (dict)
    │   ├─MIRIFULONG (list)
    │   │ ├─[0] (str): Hz390
    │   │ ├─[1] (str): Hz164
    │   │ └─[2] (str): Hz10_slow_MIRIFULONG
    │   ├─MIRIFUSHORT (list)
    │   │ ├─[0] (str): Hz390
    │   │ ├─[1] (str): Hz164
    │   │ └─[2] (str): Hz10_slow_MIRIFUSHORT
    │   └─MIRIMAGE (list)
    │     ├─[0] (str): Hz390
    │     ├─[1] (str): Hz164
    │     └─[2] (str): Hz10_slow_MIRIMAGE
    └─rowclocks (int): 28

[stscijgbot-jp]

Comment by Meaghan McDonald on JIRA:

Oof okay. Thanks for catching that. Michael Regan could you please recreate the file with the correct datamodels format and with reftype = emicorr? Thank you.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Hey Maria, what specifically is the issue with the new file Mike/Meaghan posted? I know Mike only included Hz390, maybe a Hz390_sub128 and one or more Hz10 entries. We do not want to include any other definitions like Hz164 or those old Hz218 definitions at this time.

The definitions in the file should not be tied to any other datamodel repo, so that the MIRI team can (in the future) create and deliver new reference files with any wave definitions they want without having to deliver or modify anything else. The code should be able to get everything it needs from the dictionaries in the reference file itself.

If the problem is some other structural issue with the file let us know what the specific issue is and we can fix it. I think Mike used the format from _new3.asdf, but maybe he pulled in one of the older versions (_new2.asdf) by mistake.

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

Eddie Bergeron when I downloaded and opened the file I see this: 

root (AsdfObject)
├─asdf_library (Software)
│ ├─author (str): The ASDF Developers
│ ├─homepage (str): http://github.com/asdf-format/asdf
│ ├─name (str): asdf
│ └─version (str): 2.15.1
├─history (dict)
│ └─extensions (list)
│   └─[0] (ExtensionMetadata)
│     ├─extension_class (str): asdf.extension.BuiltinExtension
│     └─software (Software)
│       ├─name (str): asdf
│       └─version (str): 2.15.1
├─Hz10 (NDArrayType): shape=(128,), dtype=float64
├─Hz164 (NDArrayType): shape=(128,), dtype=float64
├─Hz218a (NDArrayType): shape=(128,), dtype=float64
├─Hz218b (NDArrayType): shape=(128,), dtype=float64
├─Hz218c (NDArrayType): shape=(128,), dtype=float64
├─Hz390 (NDArrayType): shape=(256,), dtype=float64
└─meta (dict)
  ├─author (str): Michael Regan
  ├─date (str): 2023-11-27T16:28:32.485
  ├─description (str): EMI parameters
  ├─filename (str): jwst_miri_emimodel.asdf
  ├─instrument (dict)
  │ └─name (str): MIRI
  ├─model_type (str): EmiModel
  ├─pedigree (str): INFLIGHT 2023-11-27 2023-11-11
  ├─reftype (str): emicorr
  ├─telescope (str): JWST
  └─useafter (str): 2000-01-01T00:00:00

There are 3 problems:

All the frequencies should be under a key called 'frequencies'

Each subkey of frequencies, should have its own dictionary containing 2 keys: 'frequency', which is the number, and 'phase_amplidudes' with is the 1-D array, e.g.

├─frequencies (dict)
│ ├─Hz10 (dict)
│ │ ├─frequency (float): 10.039216
│ │ └─pahse_amplitudes (NDArrayType): shape=(128,), dtype=float64

3. there should be another key at the same level as 'frequencies' called 'subarray_cases', which should contain what frequencies to correct for and the other info (frame clocks, freqs, rowclocks), as above, e.g.

   └─subarray_cases (dict)
     ├─BRIGHTSKY (dict)
     │ ├─frameclocks (int): 23968
     │ ├─freqs (dict)
     │ │ ├─FASTR1 (list)
     │ │ │ ├─[0] (str): Hz164
     │ │ │ └─[1] (str): Hz10
     │ │ └─SLOWR1 (dict)
     │ │   ├─MIRIFULONG (list)
     │ │   │ ├─[0] (str): Hz164
     │ │   │ └─[1] (str): Hz10_slow_MIRIFULONG
     │ │   ├─MIRIFUSHORT (list)
     │ │   │ ├─[0] (str): Hz164
     │ │   │ └─[1] (str): Hz10_slow_MIRIFUSHORT
     │ │   └─MIRIMAGE (list)
     │ │     ├─[0] (str): Hz164
     │ │     └─[1] (str): Hz10_slow_MIRIMAGE
     │ └─rowclocks (int): 82

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

Eddie Bergeron if you have the files of the frequencies you want to include in this first delivery, I can create the file 

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Ok, thanks, I'll post some simple fits files with the PA vectors and the freq/sub/det values for each this afternoon (around 2 PM). Mike and I are on another call at the moment.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Ok, here are 6 reference waves (PA vectors) to put into this reference file. Frequencies are also in the fits headers:

Hz390_reference_wave.fits - for fast, all detectors, subarrays: [SLITLESSPRISM, MASKLYOT, SUB64, MASK1140, MASK1065, MASK1550], freq=390.62500

Hz390_SUB128_reference_wave.fits - for fast, all detectors, subarrays: [SUB128], freq=390.62500

Hz10_reference_wave.fits - for fast, all detectors, all subarrays: [SLITLESSPRISM, MASKLYOT, SUB64, MASK1140, MASK1065, MASK1550, SUB128, BRIGHTSKY, SUB256, FULL_FASTR1], freq=10.039216

Hz10_slow_MIRIMAGE_reference_wave.fits - for slow, MIRIMAGE detector, all subarrays: [FULL_SLOWR1], freq=10.039216

Hz10_slow_MIRIFUSHORT_reference_wave.fits - for slow, MIRIFUSHORT detector, all subarrays: [FULL_SLOWR1], freq=10.039216

Hz10_slow_MIRIFULONG_reference_wave.fits - for slow, MIRIFULONG detector, all subarrays: [FULL_SLOWR1], freq=10.039216

 

All the fits files are attached at the top of this ticket.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

For completeness, here are the rowclocks (10us clocks per row) and frameclocks (10us clocks per frame) values for all of the currently defined MIRI subarrays. You can find these near the top of my main idl routine, fix_miri_emi.pro. They can also be calculated for any given subarray definition using my miri_frametime.pro routine or Mike Ressler's python code. Here they are for all the current full and subarrays:

'SLITLESSPRISM' ;frametime=miri_frametime(1,18,529,944,rowclocks=rowclocks,frameclocks=frameclocks)   rowclocks=28   frameclocks=15904L

'MASKLYOT' ;frametime=miri_frametime(1,80,719,1022,rowclocks=rowclocks,frameclocks=frameclocks)   rowclocks=90   frameclocks=32400L

'SUB64' ;frametime=miri_frametime(1,18,779,842,rowclocks=rowclocks,frameclocks=frameclocks)   rowclocks=28   frameclocks=8512L

'SUB128' ;frametime=miri_frametime(1,34,891,1018,rowclocks=rowclocks,frameclocks=frameclocks)   rowclocks=44   frameclocks=11904L

'MASK1140'       ;frametime=miri_frametime(1,72,243,466,rowclocks=rowclocks,frameclocks=frameclocks)   rowclocks=82   frameclocks=23968L          'MASK1065'       ;frametime=miri_frametime(1,72,21,244,rowclocks=rowclocks,frameclocks=frameclocks)   rowclocks=82   frameclocks=23968L          'MASK1550'       ;frametime=miri_frametime(1,72,463,686,rowclocks=rowclocks,frameclocks=frameclocks)   rowclocks=82   frameclocks=23968L

 

; ##################################################         ; 390Hz already in-phase for these, but may need corr for other frequencies (e.g. 10Hz heater noise) ; ##################################################     

 

'FULL_FASTR1' ;frametime=miri_frametime(1,258,1,1024,rowclocks=rowclocks,frameclocks=frameclocks) rowclocks=271 frameclocks=277504L

'FULL_SLOWR1' ;frametime=miri_frametime(1,258,1,1024,rowclocks=rowclocks,frameclocks=frameclocks,/slow) rowclocks=2333 frameclocks=2388992L

'BRIGHTSKY' ;frametime=miri_frametime(115,242,51,562,rowclocks=rowclocks,frameclocks=frameclocks) rowclocks=162 frameclocks=86528L

'SUB256' ;frametime=miri_frametime(104,167,51,306,rowclocks=rowclocks,frameclocks=frameclocks) rowclocks=96 frameclocks=29952L

 

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

Ok, we will check about this file on Monday

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

Let me know if I need to do something. I'm not sure where we are.

 

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

ok, I created this file and tested it against the code. I think this is good to go unless the MIRI team wants to review it? Michael Regan Eddie Bergeron 

[^jwst_miri_emimodel_first_delivery.asdf]

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

idl 8.9 has built-in ASDF and YAML handling functions. Last week I asked ITSD to push that version out via self-service (which had been v 8.8) and they got it out just this morning. Just go to the self service app and reinstall. On a mac, be sure to restart XQuartz afterwards so the default paths are updated.

I managed to use that to read the PA vectors in this first delivery file. Looks good to me. Here's a little script and the resulting plots:

.r

_a=asdf_parse('jwst_miri_emimodel_firstdelivery.asdf') b=(a['frequencies'].keys()).toarray() print,b _n=nelements(b)

window,0,xs=1024,ys=900 !p.multi=[0,2,fix(n/2.+0.5)]

for i=0,n-1 do begin         _nx=a['frequencies',b[i],'phaseamplitudes','shape']*1.         x=findgen(nx[0])/nx[0]         y=a['frequencies',b[i],'phase_amplitudes'].data         plot,x,y,title=b[i],charsize=2,xtitle='Phase',ytitle='Amplitude (DN)',xst=1 endfor

!p.multi=0

end

_!emimodel_firstdelivery.png!

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Reference file schema and step keywords added in spacetelescope/stdatamodels#200

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

Eddie Bergeron and Michael Regan, currently the default value for scale_reference is set to True, is this correct?

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Yes, that's correct.

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

Eddie Bergeron in the IDL routine do the following lines remove permanently bad reference columns for all 3 MIRI detectors?

      dd(1,*)=(dd(0,*)+dd(3,*))/2. ; fix a bad ref col?                 
      dd(2,*)=(dd(0,*)+dd(3,*))/2. ; fix a bad ref col?

 

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

I don't know the details of this feature. It's just a bright col that I noticed at the beginning of each miri IMAGER subarray row, so I masked it to reduce the impact on the wave phasing. Maybe Misty or someone else with MIRI knows what it is?

I think it had a bigger impact on my fft power spectrum results than on the phasing (it should be effectively clipped out of the phase curves even if it was ignored - except at exactly a row time).