WIP: Adding DHS to PandExo, including front end

[natashabatalha]

This PR adds the short wave DHS grism functionality to NIRCam. It includes additional options at the front end to run the short wave simulations. It also allows users to run "NIRCam DHS" via the run_pandexo front end.

Remaining to-dos:

• [ ] Add function to let people know what the options for filters are for NIRCam DHS
• [ ] There is a new key "pandexofilterpair` created through the online interface that we can use to hack the correct wavelength ranges since Pandeia/JWST ETC does not plan on providing the real cutoffs before cy4
• [ ] change version number to 4

[natashabatalha]

@hover2pi added as a last to-do after testing is complete! thanks for the heads up

[nespinoza]

Hi @natashabatalha --- just a note that we are actively working on this. @hover2pi is doing some final tweaks to put the page as it will be seen by users for internal testing, and I have a suite of tests I'll check against the ETC once that's up and running. Will come back with any feedback here. This might happen tomorrow, otherwise early next week at the latest (hopping on a plane back to the US tomorrow on my end, so a bit chaotic over here).

One piece of feedback I can already give you is this: I know I suggested to you that SHALLOW2 was likely going to be the standard some (several?) weeks ago. However, after speaking/discussing with several experts at STScI, we arrived (mainly thanks to the huge brains on top of Jeff Valenti, Mario Gennaro, John Stansberry and others) that a better way forward involves actually implementing new readout modes for DHS after the Cycle 4 proposal deadline. Of course that begs the question: how should proposers this Cycle then should deal with this --- and the answer is that we are just asking users to use RAPID calculations following the strategy described here: https://jwst-docs.stsci.edu/jwst-near-infrared-camera/nircam-observing-strategies/nircam-time-series-observation-recommended-strategies/nircam-short-wavelength-grism-time-series-observing-strategies#gsc.tab=0.

So, bottom line: instead of SHALLOW I think RAPID should be the PandExo default given this strategy we suggest in JDox (and on today's JWebbinar as well). Perhaps adding a link to that docs on PandExo would be a nice to have.

Will come back with more (likely minor) feedback soon!

N.

[nespinoza]

All right @natashabatalha. First of all, very sorry for the delay on testing this. It took 1 software engineer and 2 astronomers to figure out how to set the service in the test server on our side with the new upgrades :).

I've tested this branch with a few targets against the JWST ETC, with the modification that I wrote above, i.e., changing shallow2 -> rapid in the nircam_dhs_input.json file. General comments:

1. The PandExo SNR estimates are slightly off with respect to the JWST ETC. I wonder if this might be because I simply replaced the rapid readout in the nircam_dhs_input.json file from shallow2. If I do a RAPID calculation for WASP-39b in the JWST ETC (J = 10.663, Teff 5500, logg 4.5), SUB256STRIPE4_DHS subarray, 30 groups per integration, I get this SNR plot from the ETC:

If I try the very same calculation in PandExo, however, I get this:

So the change in precision is a factor of ~2 (with PandExo suggesting a better SNR). I tried doing the same calculation with a known/old instrument (thinking perhaps I did something wrong). Ran the same calculations on NIRSpec/PRISM at 4 groups per integration. The JWST ETC gives me this for the SNR:

(Note below 2 microns its saturated, hence not showing that). PandExo gives me:

Which are consistent with each other; PandExo only slightly better. So for NIRspec/PRISM, all is good --- but for DHS short-wavelength, it is not.

I also tried an experiment with the same DHS setup but looking at the long-wavelength SNR and got an error (see below, number 2).

Not sure where the problem might be with the DHS SNR in PandExo. I tried deactivating, e.g., the sky noise in the JWST ETC. This slightly bumps the SNR to ~100 at the peak . Wonder, again, if this is because I just went and changed the readout mode?

2. When trying to check the NIRcam/DHS Long Wavelength results with PandExo, I get an error. I tried the exact same setup as above but selected "Display Simulations For?" -> "Long Wavelength" and got this error:

concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandexo/engine/pandexo.py", line 36, in wrapper
    return compute_full_sim(dictinput, verbose=verbose)
  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandexo/engine/jwst.py", line 170, in compute_full_sim
    out = perform_out(pandeia_input, pandexo_input,timing, both_spec)
  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandexo/engine/jwst.py", line 579, in perform_out
    report_out = perform_calculation(pandeia_input, dict_report=False)
  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandeia/engine/perform_calculation.py", line 35, in perform_calculation
    report = calculate_sn(calc_input, webapp=webapp, validate=validate)
  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandeia/engine/etc3D.py", line 252, in calculate_sn
    calc_config, instrument, strategy, scene_configuration, background, background_level, warnings = setup(calc_input, webapp=webapp, validate=validate)
  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandeia/engine/etc3D.py", line 75, in setup
    strategy = StrategyFactory(instrument, config=strategy_configuration, webapp=webapp, validate=validate)
  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandeia/engine/strategy.py", line 4960, in StrategyFactory
    cls = type_map[method](instrument=instrument, webapp=webapp, validate=validate, config=config, **kwargs)
  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandeia/engine/strategy.py", line 2014, in __init__
    ImagingApPhot.__init__(self, instrument=instrument, config=config, webapp=webapp, **kwargs)
  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandeia/engine/strategy.py", line 1121, in __init__
    Strategy.__init__(self, instrument=instrument, config=config, webapp=webapp, **kwargs)
  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandeia/engine/strategy.py", line 111, in __init__
    DefaultConfig.__init__(self, webapp=webapp, config=config, **kwargs)
  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandeia/engine/config.py", line 47, in __init__
    all_config = merge_data(self._get_config(), config, dict(**kwargs))
  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandeia/engine/strategy.py", line 146, in _get_config
    inst_config['aperture_size'] = sizes[self.instrument.instrument[key]]
KeyError: 'f444w'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/tornado/web.py", line 1788, in _execute
    result = method(*self.path_args, **self.path_kwargs)

  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandexo/engine/run_online.py", line 928, in get
    result = self._get_task_result(id)

  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/site-packages/pandexo/engine/run_online.py", line 197, in _get_task_result
    return task.result()

  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()

  File "/internal/data1/anaconda3/envs/pandexo-3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception

KeyError: 'f444w'

3. When PandExo is set to optimize, it can go up to a very large amount of groups (but in NIRCam groups are capped). For example, setting a simulation for WASP-39b with DHS (F150W2+F444W, sub256stripe4) and checking the short wavelength, PandExo optimizes this observation at 362 groups and 42 integrations. The reality is that at least the NIRCam RAPID readout pattern only allows up to 30 groups per integration as of today (JDox says right now it is up to 10 groups, but I checked, and APT 2024.5.2 lets you go to 30 groups for time series --- so 30 groups it is). This is important as this changes the observational efficiency significantly, and at least according to PandExo it significantly changes the attained precision (by a factor of ~3). I wonder if a cap could be set for PandExo on the number of groups for NIRCam "by hand"?

4. In the NIRCam subarray selection button, add the number of spectra. In the selection of the subarrays, just as the frametime is written by hand, it would be perhaps nice to write the number of spectra each subarray imply. This confused me a couple of times.

And that's it. Thanks so much for working on this! Happy to hop on a call to figure 1 and 2 above out.

N.

[nespinoza]

One last comment I forgot to add: it might be beneficial for the future to allow the user to select the readout mode (leave rapid for now as the default). There's going to be new readout patterns implemented soon-ish that are uniquely tailored to DHS, so adding a button so users can choose the readout pattern in PandExo will pay dividends in a ~few months.

[natashabatalha]

@nespinoza great. I resolved your points 3 and 4. In doing so I realized it was pulling the 8 spectra aperture for everything so I resolved that as well. Maybe that was part of issue 1??

Issue 2 might be caused by me not understanding how they want to specify DHS setup with the long wave filter.. maybe I need to switch the mode to lw_tsgrism and then keep the aperture and disperser the same. Will test it..

[natashabatalha]

Ah I found this hiding https://github.com/natashabatalha/PandExo/blob/ba94c83a42d3935af98aad8335ceb37a520b62a2/pandexo/engine/utils/plotters.py#L218 This might resolve the SNR problem, which would hopefully just make this a visualization error.

[nespinoza]

Thanks for all this work @natashabatalha, this is awesome.

Issue 2 might be caused by me not understanding how they want to specify DHS setup with the long wave filter.. maybe I need to switch the mode to lw_tsgrism and then keep the aperture and disperser the same. Will test it..

Let me know when this is in so I can test. I can get to testing very late today or early tomorrow.

N.

[natashabatalha]

When I run this configuration:

'configuration': {'instrument': {'instrument': 'nircam',
     'mode': 'lw_tsgrism',
     'filter': 'f444w',
     'aperture': 'dhs0spec8',
     'disperser': 'dhs0'},
    'detector': {'readout_pattern': 'rapid',
     'subarray': 'sub256stripe4_dhs',
     'ngroup': 30,
     'nint': 173,
     'nexp': 1}},
   'strategy': {'method': 'specapphot',
    'background_subtraction': True,
    'aperture_size': 0.7,
    'sky_annulus': [0.8, 1.6],
    'target_xy': [0.0, 0.0],
    'reference_wavelength': None,
    'units': 'arcsec'}}

I get this wavelength solution from Pandeia

  '1d': {'wave_pix': array([0.77015998, 0.77044998, 0.77073998, ..., 2.34920988, 2.34949988,
          2.34978988]),
   'wave_calc': array([0.77015998, 0.77044998, 0.77073998, ..., 2.34920988, 2.34949988,
          2.34978988]),

I dont understand why it is returning a spectrum from 0.7-2.3 with F444. I must be setting up the configuration incorrectly. You can recreate this by doing this with the dhs branch:

inst_dict = jdi.load_mode_dict('NIRCam DHS')
inst_dict['configuration']['instrument']['mode'] = 'lw_tsgrism'
inst_dict['configuration']['instrument']['filter']='f444w' 
result = jdi.run_pandexo(exo_dict,inst_dict)