Closed fmigneault closed 1 year ago
We can add another tag for the relevant should_fail: true
tests, like tags: […, schema_valid]
; would that be good enough?
Yes. That would be sufficient. Thank you.
@fmigneault Feel free to send a PR to add the schema_valid
tag to the should_fail: true
tests that you think need it; I'll review it and merge it
As mentioned in the conformance tests, we can use
should_fail
to know the expected outcome: https://github.com/common-workflow-language/cwl-v1.2/blob/d50fad00eedb071ef7372a0f471e6fc1155addcc/CONFORMANCE_TESTS.md?plain=1#L115-L116However, some tests are more of "runtime" nature: https://github.com/common-workflow-language/cwl-v1.2/blob/d50fad00eedb071ef7372a0f471e6fc1155addcc/tests/loadContents/test-index.yaml#L8-L14
While others can be validated beforehand, without even runing the tool, only using the document and schema validation: https://github.com/common-workflow-language/cwl-v1.2/blob/d50fad00eedb071ef7372a0f471e6fc1155addcc/conformance_tests.yaml#L2631-L2636
https://github.com/common-workflow-language/cwl-v1.2/blob/d50fad00eedb071ef7372a0f471e6fc1155addcc/tests/mixed-versions/test-index.yaml#L28-L33
https://github.com/common-workflow-language/cwl-v1.2/blob/d50fad00eedb071ef7372a0f471e6fc1155addcc/tests/mixed-versions/test-index.yaml#L35-L40
It would be useful to provide a flag that allows this distinction. Using the JSON schema validation I'm working on (https://github.com/opengeospatial/ogcapi-processes/pull/329), there is no way to distinguish between the two. I could always filter tests based on a known set of cases, but then, I goes against the use of the configurable test conformance.