oasis-tcs / csaf

OASIS CSAF TC: Supporting version control for Work Product artifacts developed by members of TC, including prose specifications and secondary artifacts like meeting minutes and productivity code
https://github.com/oasis-tcs/csaf
Other
147 stars 39 forks source link

Provide an expected failure "code" or "message" in testcases.json #820

Open oxisto opened 1 day ago

oxisto commented 1 day ago

This is a follow-up to the discussion in https://github.com/csaf-sbom/kotlin-csaf/issues/80#issuecomment-2448675774. We make use of the TC test cases JSONs in testing the validation tests in kotlin-csaf. While exploring options to keep tests in sync with the TC, we came up with the idea whether it is possible to include an expected failure "message" in the testcases.json. This way we could align the error messages in our library with the official ones and we could query the testcases.json to automatically assert good and bad test cases even with the appropriate error.

Further discussion points

This would make developing compliant CSAF libraries much more easier, sort of like a Technology Compatibility Kit.

tschmidtb51 commented 1 day ago

We need to take into consideration, what is already existing, e.g. the codes used by the csaf-validator-lib.

The instancePath is crucial as it provides the opportunity for viewers and editors to highlight it message in the right place.