Missing v3idlyangle for NRSFSS during wavecorr

[stscijgbot-jp]

Issue JP-2125 was created on JIRA by Tyler Pauly:

During a spec2 run of an NRSFSS association, an error is reported in the wavecorr step:

2021-05-29 17:01:03,735 - CRDS - DEBUG -  Reference type 'wavecorr' defined as 'jwst_nirspec_wavecorr_0001.asdf'
2021-05-29 17:01:03,770 - stpipe.Spec2Pipeline.wavecorr - INFO - Using WAVECORR reference file /ifs/archive/test/jwst/ref/tmp_crds/cache/references/jwst/nirspec/jwst_nirspec_wavecorr_0001.asdf
2021-05-29 17:01:04,052 - stpipe.Spec2Pipeline.wavecorr - INFO - Detected a POINT source type in slit S200A2
2021-05-29 17:01:04,054 - stpipe.AssignWcsStep - INFO - AssignWcsStep instance created.
Traceback (most recent call last):
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.0.20210524-py38/lib/python3.8/site-packages/jwst/pipeline/calwebb_spec2.py", line 113, in process
    result = self.process_exposure_product(
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.0.20210524-py38/lib/python3.8/site-packages/jwst/pipeline/calwebb_spec2.py", line 246, in process_exposure_product
    calibrated = self._process_nirspec_slits(calibrated)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.0.20210524-py38/lib/python3.8/site-packages/jwst/pipeline/calwebb_spec2.py", line 395, in _process_nirspec_slits
    calibrated = self.wavecorr(calibrated)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.0.20210524-py38/lib/python3.8/site-packages/stpipe/step.py", line 407, in run
    step_result = self.process(*args)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.0.20210524-py38/lib/python3.8/site-packages/jwst/wavecorr/wavecorr_step.py", line 62, in process
    output_model = wavecorr.do_correction(input_model, reffile)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.0.20210524-py38/lib/python3.8/site-packages/jwst/wavecorr/wavecorr.py", line 59, in do_correction
    apply_zero_point_correction(slit, wavecorr_file)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.0.20210524-py38/lib/python3.8/site-packages/jwst/wavecorr/wavecorr.py", line 91, in apply_zero_point_correction
    source_xpos = get_source_xpos(slit, slit_wcs, lam=2,
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.0.20210524-py38/lib/python3.8/site-packages/jwst/wavecorr/wavecorr.py", line 252, in get_source_xpos
    idl2v23 = trmodels.IdealToV2V3(v3idlyangle, v2ref, v3ref, vparity)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.0.20210524-py38/lib/python3.8/site-packages/jwst/transforms/models.py", line 970, in __init__
    super(IdealToV2V3, self).__init__(v3idlyangle=v3idlyangle, v2ref=v2ref,
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.0.20210524-py38/lib/python3.8/site-packages/astropy/modeling/core.py", line 719, in __init__
    self._initialize_parameters(args, kwargs)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.0.20210524-py38/lib/python3.8/site-packages/astropy/modeling/core.py", line 2062, in _initialize_parameters
    self._initialize_parameter_value(param_name, None)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.0.20210524-py38/lib/python3.8/site-packages/astropy/modeling/core.py", line 2136, in _initialize_parameter_value
    raise TypeError("{0}.__init__() requires a value for parameter "
TypeError: IdealToV2V3.__init__() requires a value for parameter 'v3idlyangle'

May or not be relevant: exposure has SUBARRAY = SUBS200A2.

[stscijgbot-jp]

Comment by James Davies on JIRA:

Looks like the dataset doesn't have the V3I_YANG keyword set. Is the SIAF for this subarray empty for this?

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

James Davies It looks like the only keywords populated in the WCS parameters are:

V2_REF = -146.05066 / [arcsec] Telescope v2 coordinate of the referen V3_REF = -432.206071 / [arcsec] Telescope v3 coordinate of the referen RA_REF = 0.0 / [deg] Right ascension of the reference point
DEC_REF = 0.0 / [deg] Declination of the reference point
ROLL_REF= 0.0 / [deg] V3 roll angle at the ref point (N over E) VELOSYS = 5160.29 / [m/s] Radial velocity wrt Barycenter

V3I_YANG is not present.

[jdavies-st]

So yeah, I suspect this is an upstream bug in the creation of the _uncal file. So either a SIAF or database problem.

V3I_YANG is absolutely necessary in all exposures that need WCS, as it describes the orientation of the specified aperture relative to the telescope's V3 axis in the focal plane.

Our set_telescope_pointing script should raise an error if this is not found in the data

https://github.com/spacetelescope/jwst/blob/b5fe85dde47bcc6d122464a2ab389bb26b5eae5a/jwst/lib/set_telescope_pointing.py#L440-L443

[hbushouse]

Definitely odd in that V2_REF and V3_REF are populated, which also come from the SIAF. I would expect that either all SIAF values would be available for a given APERNAME or none of them - not some. Check the SIAF entry for that instrument and APERNAME. I note that the pointing values are all zero too - could indicate that set_telescope_pointing could not complete the calculation, perhaps due to missing SIAF values.

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

It is an error occurring in set_telescope_pointing. The SDP log is attached, but it is reproducible with the command set_telescope_pointing.py jw00636019001_04101_00001_nrs1_uncal.fits

producing the error found below. There is a schema mismatch somewhere.

2021-06-04 10:08:02,243 - stpipe - INFO - ------Setting pointing for jw00636019001_04101_00001_nrs1_uncal.fits 2021-06-04 10:08:02,243 - stpipe - INFO - ------Setting pointing for jw00636019001_04101_00001_nrs1_uncal.fits Updating WCS info for file jw00636019001_04101_00001_nrs1_uncal.fits 2021-06-04 10:08:02,243 - stpipe - INFO - Updating WCS info for file jw00636019001_04101_00001_nrs1_uncal.fits Cannot determine pointing information: Column names don't match schema. Schema has {'mt_z_helio', 'moving_target_x', 'moving_target_y', 'moving_target_ra', 'mt_jwst_distance', 'mt_y_jwst', 'mt_x_jwst', 'mt_z_jwst', 'mt_y_helio', 'mt_x_helio', 'moving_target_dec', 'mt_sun_distance', 'phase_angle'}. Data has {'mt_apparent_dec', 'mt_apparent_sun_distance', 'mt_apparent_x_jwst', 'mt_true_x_jwst', 'mt_apparent_jwst_distance', 'mt_detector_x', 'mt_true_x_helio', 'mt_true_y_helio', 'mt_true_z_helio', 'mt_true_y_jwst', 'mt_apparent_ra', 'mt_apparent_y_jwst', 'mt_apparent_y_helio', 'mt_apparent_x_helio', 'mt_apparent_z_helio', 'mt_true_z_jwst', 'apparent_phase_angle', 'mt_detector_y', 'mt_apparent_z_jwst', 'apparent_light_travel_time'} 2021-06-04 10:08:02,826 - stpipe - INFO - Cannot determine pointing information: Column names don't match schema. Schema has {'mt_z_helio', 'moving_target_x', 'moving_target_y', 'moving_target_ra', 'mt_jwst_distance', 'mt_y_jwst', 'mt_x_jwst', 'mt_z_jwst', 'mt_y_helio', 'mt_x_helio', 'moving_target_dec', 'mt_sun_distance', 'phase_angle'}. Data has {'mt_apparent_dec', 'mt_apparent_sun_distance', 'mt_apparent_x_jwst', 'mt_true_x_jwst', 'mt_apparent_jwst_distance', 'mt_detector_x', 'mt_true_x_helio', 'mt_true_y_helio', 'mt_true_z_helio', 'mt_true_y_jwst', 'mt_apparent_ra', 'mt_apparent_y_jwst', 'mt_apparent_y_helio', 'mt_apparent_x_helio', 'mt_apparent_z_helio', 'mt_true_z_jwst', 'apparent_phase_angle', 'mt_detector_y', 'mt_apparent_z_jwst', 'apparent_light_travel_time'}

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

Issue has been found. The moving target schema are not covered by the schema_editor because there is no link to the schema from the top level schema. There will be an issue filed for that.

In the meantime, resolution of this issue will be to update the datamodel schema for moving target.

[hbushouse]

This is likely due to recent SDP updates to the MT position table columns that did not get corresponding updates in our datamodels schema for that table. It should be an easy schema update.

[stscijgbot-jp]

Comment by James Davies on JIRA:

We need to update the script in scripts/set_telescope_pointing.py to not consume the traceback. From the SDP log, it's not clear where the "Column names don't match schema." error message is even coming from, because the traceback has been erased in the exception handler in the script. We need the full traceback.

[stscijgbot-jp]

Comment by James Davies on JIRA:

Here's the traceback. It's happening when the file is being loaded as a Level1bModel.

Traceback (most recent call last):
  File "/home/jdavies/miniconda3/envs/jwst/bin/set_telescope_pointing.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/jdavies/dev/jwst/scripts/set_telescope_pointing.py", line 97, in <module>
    add_wcs(
  File "/home/jdavies/dev/jwst/jwst/lib/set_telescope_pointing.py", line 190, in add_wcs
    with Level1bModel(filename) as model:
  File "/home/jdavies/miniconda3/envs/jwst/lib/python3.8/site-packages/stdatamodels/model_base.py", line 232, in __init__
    asdffile = fits_support.from_fits(
  File "/home/jdavies/miniconda3/envs/jwst/lib/python3.8/site-packages/stdatamodels/fits_support.py", line 590, in from_fits
    known_keywords, known_datas = _load_from_schema(
  File "/home/jdavies/miniconda3/envs/jwst/lib/python3.8/site-packages/stdatamodels/fits_support.py", line 511, in _load_from_schema
    mschema.walk_schema(schema, callback)
  File "/home/jdavies/miniconda3/envs/jwst/lib/python3.8/site-packages/stdatamodels/schema.py", line 149, in walk_schema
    recurse(schema, [], None, ctx)
  File "/home/jdavies/miniconda3/envs/jwst/lib/python3.8/site-packages/stdatamodels/schema.py", line 137, in recurse
    recurse(val, path + [key], combiner, ctx)
  File "/home/jdavies/miniconda3/envs/jwst/lib/python3.8/site-packages/stdatamodels/schema.py", line 124, in recurse
    if callback(schema, path, combiner, ctx, recurse):
  File "/home/jdavies/miniconda3/envs/jwst/lib/python3.8/site-packages/stdatamodels/fits_support.py", line 489, in callback
    result = _fits_array_loader(
  File "/home/jdavies/miniconda3/envs/jwst/lib/python3.8/site-packages/stdatamodels/fits_support.py", line 441, in _fits_array_loader
    return from_fits_hdu(hdu, schema)
  File "/home/jdavies/miniconda3/envs/jwst/lib/python3.8/site-packages/stdatamodels/fits_support.py", line 628, in from_fits_hdu
    data = properties._cast(data, schema)
  File "/home/jdavies/miniconda3/envs/jwst/lib/python3.8/site-packages/stdatamodels/properties.py", line 83, in _cast
    val = util.gentle_asarray(val, dtype)
  File "/home/jdavies/miniconda3/envs/jwst/lib/python3.8/site-packages/stdatamodels/util.py", line 69, in gentle_asarray
    raise ValueError(
ValueError: Column names don't match schema. Schema has {'mt_z_jwst', 'phase_angle', 'mt_z_helio', 'mt_jwst_distance', 'moving_target_ra', 'mt_x_helio', 'mt_y_helio', 'moving_target_y', 'moving_target_dec', 'mt_y_jwst', 'mt_x_jwst', 'moving_target_x', 'mt_sun_distance'}. Data has {'mt_apparent_y_helio', 'apparent_phase_angle', 'mt_apparent_y_jwst', 'mt_apparent_x_jwst', 'mt_true_z_jwst', 'apparent_light_travel_time', 'mt_true_z_helio', 'mt_apparent_z_jwst', 'mt_detector_x', 'mt_apparent_x_helio', 'mt_true_y_helio', 'mt_apparent_z_helio', 'mt_apparent_sun_distance', 'mt_true_x_helio', 'mt_true_x_jwst', 'mt_apparent_ra', 'mt_apparent_jwst_distance', 'mt_apparent_dec', 'mt_detector_y', 'mt_true_y_jwst'}

[stscijgbot-jp]

Comment by James Davies on JIRA:

Finally, we should probably not be allowing exceptions on this error in the the pointing script if the file is not valid (relative to our schemas), it should fail at the beginning. It just so happens that it fails on ValueError in schema validation, probably a different ValueError than what we were intending to handle in the set_telescope_pointing.py script.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed in #6129