I'm trying to update three things at once - version of Galaxy it is rebased against, the versions of CWL we test against, and the version of cwltool we're using.
Unfortunately while hacking on this, I realized the last time I think the "green list" was all green was at the first BioHackathon two years ago. I believe new cwl-1.0 branch has no regressions against the version from last November (the 2019 branch) but it does appear to have some regressions versus some even earlier version of this repo. (There are all sorts of dated branches on this repo so one could probably be pull out the older version.)
In order to track these regressions but still proceed with updated cwl-1.0. I fixed several bugs, and pulled some more tests out the green tests and added them to a new regressions test list that gets merged with the red list.
I also updated the test case generator to create a file of just these tests:
sh run_tests.sh -api lib/galaxy_test/api/cwl/test_cwl_conformance_regressed_v1_0.py
I think the regressions correspond to 4 bugs:
# Regressions
REGRESSIONS = {
'directory_output': "Bug1: Extra files handling changes.",
'wf_compound_doc': "Bug4: No clue what is wrong here.",
'step_input_default_value_nosource': "Bug2: Broken step input defaults?",
"wf_input_default_missing": "Bug2",
"wf_input_default_provided": "Bug3: is not a valid workflow invocation id",
"wf_scatter_dotproduct_twoempty": "Bug3",
"wf_scatter_emptylist": "Bug3",
"wf_step_connect_undeclared_param": "Bug2",
"wf_two_inputfiles_namecollision": "Bug3",
"wf_wc_expressiontool": "Bug3",
"wf_wc_nomultiple": "Bug3",
"wf_wc_scatter_multiple_merge": "Bug3",
"wf_wc_scatter_multiple_flattened": "Bug3",
"wf_wc_scatter": "Bug3",
"wf_wc_parseInt": "Bug3",
"workflow_union_default_input_unspecified": "Bug2",
}
RED_TESTS.update(REGRESSIONS)
Mostly seeming to do with changes in Galaxy core I think.
That is the bad news, the good news I've updated the conformance stuff to allow multiple versions and duplicated the 1.0 green list into a v1.1 green list in branch. Everything that was green appears to still be green in v1.1.
We also need to update cwltool to a version that supports v1.2 to work on conditionals, but that might regress a bunch of stuff which inspired getting these green lists re-generated. I have brought in the conformance tests for 1.2 though at least and created a placeholder for a green list - but haven't filled anything in there yet.
In terms of cwltool however, I've opened a PR to update our dependency from December 2019 to March 2020 (https://github.com/galaxyproject/galaxy/pull/10658). But that is as far as we can go unless we can come up without re-figuring out how to handle GALAXY_SLOTS in the new version.
Actual Done Tasks (especially versus semi-stable cwl-1.0 branch):
[X] Rebase against latest Galaxy.
[X] Update commit that brings in conformance test data to bring 1.1 and 1.2 conformance tests.
[x] Updated the test case generation code to generate news test classes for each version (v1.0, v1.1, and v1.2) corresponding to red and green "required" tests as well as regressions.
[X] Re-run red tests and add 36 news test to green list.
This issue is an informal log of my work to replace the cwl-1.0 branch before this years Biohackathon.
As the result of my work this weekend, I replaced the cwl-1.0 branch (https://github.com/common-workflow-language/galaxy/tree/cwl-1.0) with an "updated" branch. The previous cwl-1.0 branch that was pushed at last years Biohackathon can now be found at https://github.com/common-workflow-language/galaxy/tree/cwl-1.0-2019 - in case there are running apps that depend on this.
I'm trying to update three things at once - version of Galaxy it is rebased against, the versions of CWL we test against, and the version of cwltool we're using.
Unfortunately while hacking on this, I realized the last time I think the "green list" was all green was at the first BioHackathon two years ago. I believe new cwl-1.0 branch has no regressions against the version from last November (the 2019 branch) but it does appear to have some regressions versus some even earlier version of this repo. (There are all sorts of dated branches on this repo so one could probably be pull out the older version.)
In order to track these regressions but still proceed with updated cwl-1.0. I fixed several bugs, and pulled some more tests out the green tests and added them to a new regressions test list that gets merged with the red list.
I also updated the test case generator to create a file of just these tests:
I think the regressions correspond to 4 bugs:
Mostly seeming to do with changes in Galaxy core I think.
That is the bad news, the good news I've updated the conformance stuff to allow multiple versions and duplicated the 1.0 green list into a v1.1 green list in branch. Everything that was green appears to still be green in v1.1.
We also need to update cwltool to a version that supports v1.2 to work on conditionals, but that might regress a bunch of stuff which inspired getting these green lists re-generated. I have brought in the conformance tests for 1.2 though at least and created a placeholder for a green list - but haven't filled anything in there yet.
In terms of cwltool however, I've opened a PR to update our dependency from December 2019 to March 2020 (https://github.com/galaxyproject/galaxy/pull/10658). But that is as far as we can go unless we can come up without re-figuring out how to handle GALAXY_SLOTS in the new version.
Actual Done Tasks (especially versus semi-stable cwl-1.0 branch):