tddesjardins
commented
5 months ago Here's an example TVAC file that does not work with DQInitStep and the exception it produces. The file is at /grp/roman/TEST_DATA/RITA_DEV/TVAC_SAMPLES_RDM/TVAC1_NOMOPS_SCIMON_20231015055603_WFI01_uncal.asdf.
2024-04-26 12:07:48,957 - CRDS - ERROR - Error determining best reference for 'pars-dqinitstep' = Unknown reference type 'pars-dqinitstep'
2024-04-26 12:07:48,957 - stpipe.DQInitStep - INFO - DQInitStep instance created.
2024-04-26 12:07:48,992 - stpipe.DQInitStep - INFO - Step DQInitStep running with args (<roman_datamodels.datamodels._datamodels.TvacModel object at 0x133828890>,).
2024-04-26 12:07:48,993 - stpipe.DQInitStep - INFO - Step DQInitStep parameters are:
pre_hooks: []
post_hooks: []
output_file: None
output_dir: None
output_ext: .asdf
output_use_model: False
output_use_index: True
save_results: False
skip: False
suffix: None
search_output_file: True
input_dir: ''
---------------------------------------------------------------------------
ValidationError Traceback (most recent call last)
Cell In[8], line 1
----> 1 dq.call(test)
File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/stpipe/step.py:686, in Step.call(cls, *args, **kwargs)
683 name = config.get("name", None)
684 instance = cls.from_config_section(config, name=name, config_file=config_file)
--> 686 return instance.run(*args)
File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/stpipe/step.py:509, in Step.run(self, *args)
507 self.prefetch(*args)
508 try:
--> 509 step_result = self.process(*args)
510 except TypeError as e:
511 if "process() takes exactly" in str(e):
File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/romancal/dq_init/dq_init_step.py:72, in DQInitStep.process(self, input)
69 input_ramp[key] = input_model.__getattr__(key)
71 # Create model from node
---> 72 output_model = RampModel(input_ramp)
73 else:
74 output_model = input_model
File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/roman_datamodels/datamodels/_datamodels.py:43, in _RomanDataModel.__init__(self, init, **kwargs)
42 def __init__(self, init=None, **kwargs):
---> 43 super().__init__(init, **kwargs)
45 if init is not None:
46 self.meta.model_type = self.__class__.__name__
File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/roman_datamodels/datamodels/_core.py:134, in DataModel.__init__(self, init, **kwargs)
132 af = asdf.AsdfFile()
133 af["roman"] = self._instance
--> 134 af.validate()
135 self._asdf = af
136 return
File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/asdf/_asdf.py:628, in AsdfFile.validate(self)
624 def validate(self):
625 """
626 Validate the current state of the tree against the ASDF schema.
627 """
--> 628 self._validate(self._tree)
File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/asdf/_asdf.py:619, in AsdfFile._validate(self, tree, custom, reading)
614 with self._blocks.options_context():
615 # If we're validating on read then the tree
616 # is already guaranteed to be in tagged form.
617 tagged_tree = tree if reading else yamlutil.custom_tree_to_tagged_tree(tree, self)
--> 619 schema.validate(tagged_tree, self, reading=reading)
620 # Perform secondary validation pass if requested
621 if custom and self._custom_schema:
File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/asdf/schema.py:649, in validate(instance, ctx, schema, validators, reading, *args, **kwargs)
646 ctx = AsdfFile()
648 validator = get_validator({} if schema is None else schema, ctx, validators, None, *args, **kwargs)
--> 649 validator.validate(instance)
651 additional_validators = [_validate_large_literals]
652 if ctx.version >= versioning.RESTRICTED_KEYS_MIN_VERSION:
File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/asdf/_jsonschema/validators.py:312, in create.<locals>.Validator.validate(self, *args, **kwargs)
310 def validate(self, *args, **kwargs):
311 for error in self.iter_errors(*args, **kwargs):
--> 312 raise error
ValidationError: mismatched tags, wanted 'asdf://stsci.edu/datamodels/roman/tags/exposure-1.0.0', got 'asdf://stsci.edu/datamodels/roman/tags/base_exposure-1.0.0'
Failed validating 'tag' in schema['properties']['meta']['allOf'][1]['properties']['exposure']:
{'tag': 'asdf://stsci.edu/datamodels/roman/tags/exposure-1.0.0',
'title': 'Exposure Information'}
On instance['meta']['exposure']:
{'data_problem': False,
'end_time': {'base_format': 'datetime',
'value': '2023-10-15T05:58:50.219'},
'exposure_time': 167.21875,
'frame_divisor': -99,
'frame_time': 3.0403409090909093,
'group_time': 3.0403409090909093,
'groupgap': 0,
'ma_table_name': '71_FULLMA510ALL',
'ma_table_number': 510,
'mid_time': {'base_format': 'datetime',
'value': '2023-10-15T05:57:26.610'},
'nframes': 1,
'ngroups': 56,
'read_pattern': [[1],
[2],
[3],
[4],
[5],
[6],
[7],
[8],
[9],
[10],
[11],
[12],
[13],
[14],
[15],
[16],
[17],
[18],
[19],
[20],
[21],
[22],
[23],
[24],
[25],
[26],
[27],
[28],
[29],
[30],
[31],
[32],
[33],
[34],
[35],
[36],
[37],
[38],
[39],
[40],
[41],
[42],
[43],
[44],
[45],
[46],
[47],
[48],
[49],
[50],
[51],
[52],
[53],
[54],
[55]],
'sca_number': 1,
'start_time': {'base_format': 'datetime',
'value': '2023-10-15T05:56:03.000'},
'type': 'WFI_IMAGE'}
stscijgbot-rstdms
Currently the logic to convert from a L1-like datamodel (e.g., WfiScienceRaw) to the Ramp model is contained in the DQInitStep:
This works for WfiScienceRaw, but not for other L1-like data such as TVAC or FPS datamodels. Either the code in DQInit needs to be updated to accept the other L1-like data (not including GW files), or a suggestion was to modify the datamodel to include a method that handles the transformation to the Ramp model.