common-workflow-language / cwl-v1.2

Released CWL v1.2.1 specification
https://www.commonwl.org/v1.2/
Apache License 2.0
34 stars 22 forks source link

Provide an option to distinguish between runtime and schema-checkable failures #253

Closed fmigneault closed 1 year ago

fmigneault commented 1 year ago

As mentioned in the conformance tests, we can use should_fail to know the expected outcome: https://github.com/common-workflow-language/cwl-v1.2/blob/d50fad00eedb071ef7372a0f471e6fc1155addcc/CONFORMANCE_TESTS.md?plain=1#L115-L116

However, some tests are more of "runtime" nature: https://github.com/common-workflow-language/cwl-v1.2/blob/d50fad00eedb071ef7372a0f471e6fc1155addcc/tests/loadContents/test-index.yaml#L8-L14

While others can be validated beforehand, without even runing the tool, only using the document and schema validation: https://github.com/common-workflow-language/cwl-v1.2/blob/d50fad00eedb071ef7372a0f471e6fc1155addcc/conformance_tests.yaml#L2631-L2636

https://github.com/common-workflow-language/cwl-v1.2/blob/d50fad00eedb071ef7372a0f471e6fc1155addcc/tests/mixed-versions/test-index.yaml#L28-L33

https://github.com/common-workflow-language/cwl-v1.2/blob/d50fad00eedb071ef7372a0f471e6fc1155addcc/tests/mixed-versions/test-index.yaml#L35-L40

It would be useful to provide a flag that allows this distinction. Using the JSON schema validation I'm working on (https://github.com/opengeospatial/ogcapi-processes/pull/329), there is no way to distinguish between the two. I could always filter tests based on a known set of cases, but then, I goes against the use of the configurable test conformance.

mr-c commented 1 year ago

We can add another tag for the relevant should_fail: true tests, like tags: […, schema_valid]; would that be good enough?

fmigneault commented 1 year ago

Yes. That would be sufficient. Thank you.

mr-c commented 1 year ago

@fmigneault Feel free to send a PR to add the schema_valid tag to the should_fail: true tests that you think need it; I'll review it and merge it