Test metadata

[simleo]

Goal: formalize test metadata description using the current RO-Crate examples as a starting point.

The Workflow RO-Crate examples have been developed as a starting point to model how a Workflow RO-Crate can be used to drive the execution of test suites. These examples are based on the Workflow RO-Crate specs, which state that the crate COULD contain a "test" Dataset, without specifying anything on its internal structure.

The sample crates in this repo add all test-related data and metadata under "test", so they are valid Workflow RO-Crates. The current format is as follows:

• A params.jsonld file defines N test cases
• Test data for each test case is contained in a different directory
• In turn, each directory contains an "inputs" and an "outputs" directory

See for instance the test dir in the CWL example. The jsonld file borrows its structure from the RO-Crate one. The test_params module is a first example of library code to parse such a file into a hierarchy of Python objects and derive a Planemo test suite description from them. The top-level check_*.py scripts (e.g., check_cwl.py) serve as an example on how to use all this to drive the tests contained in each crate. For now the focus is on Galaxy and CWL, which can be tested vith Planemo. The Nextflow check example contains more ad-hoc code.

The current version of the params.jsonld file, together with the library code that interacts with it, can be seen as a "version 0" for test data/metadata description. The idea is to use this as a starting point towards something that can be integrated in the Workflow RO-Crate specs. Current issues/questions include:

• The RO-Crate specs describe files, directories and URLs. Test cases are not exactly directories and test parameters are not always files and directories.
  • In particular, the current mapping from parameter names to their values is "name" to "@id". However, parameter values are usually not unique.
• Can the test metadata keep living in their own file in the "test" dir, or should they eventually be moved to the top-level ro-crate-metadata.jsonld?

[ilveroluca]

Maybe our friends @asulis and @svituz are also interested :-)

[simleo]

Current spec draft: https://github.com/crs4/life_monitor/wiki/Test-Metadata-Draft-Spec. Note that the workflow ro-crate examples mentioned here have been moved to interaction_experiments in #14.