Unintuitive log messages if test-required fields violate SDI spec

[AEgbert]

Problem Description

In many places throughout the test harness, we attempt to suppress errors and continue on until something critical fails. One such critical failure is an inability to create a dataclass representation of the object (indicating a name or type error of the message fields in some fashion), if the issue is related to a field used by the actual test in question. Some fields (e.g., vendorExtensions) are not critical spec violations (i.e., the broken field is not required by the test), and should be suppressed. For example:

if self.vendorExtensions is not None:
  with suppress(TypeError):
    self.vendorExtensions = init_from_dicts(self.vendorExtensions, VendorExtension)

will attempt to replace self.vendorExtensions with a list of VendorExtension objects. If an error occurs during the object initialization, the original value for self.vendorExtensions will be left in place. This retains the information in the object for manual inspection, but the extensions cannot be manipulated as VendorExtensions.

However, for a required sub-field, such as .availableSpectrumInquiryResponses, we would want this type of error to prevent execution of the actual test. We check for this condition already in test_main.py (lines 254-260):

logger.debug(f'Parsing received response as a response object...')
      try:
        response_obj = AvailableSpectrumInquiryResponseMessage(**response)
      except TypeError as ex:
        logger.error(f'Exception converting response for comparison: {ex}. Test SKIPPED.\n')
        results.add_result(test_name, TestResult.SKIPPED)
        continue

However, because .availableSpectrumInquiryResponses is subject to similar TypeError suppression as VendorExtensions

with suppress(TypeError):
      self.availableSpectrumInquiryResponses = init_from_dicts(
                                               self.availableSpectrumInquiryResponses,
                                               AvailableSpectrumInquiryResponse)

an invalid AvailableSpectrumInquiryResponse will allow the "successful" creation of the enclosing AvailableSpectrumInquiryResponseMessage, and the try/except block will not catch the issue before proceeding to the test.

Potential Fixes

Selective removal of the suppress(Error) wrappers for "critical" sub-fields would allow the error to propagate upwards and trigger the except clause in test_main.py. However, this would also prevent the partial initialization of any parent object, which is not desired by other code (i.e., the validators).

An alternate fix may be to explicitly validate the presence of the dataclass version of the provided and accessed objects in response_mask_runner.py to ensure field-access errors (using dot-notation to access fields of dicts, etc.) are avoided, and the root issue (failure to create the dataclass representation) is reported clearly.

Acknowledgements

Thanks to Commscope for contributing a test case that reveals this issue.