Tracking accomplishments at #BioHackEU20

[mr-c]

PRs

• [x] Annotate CWL 1.2 Conformance Tests #120
• [x] Fix object_store_store_by handling #119
• [x] Fix FieldTypeParameterValue wrapping #118
• [x] Rev cwltool to 3.0.20201109103151 #10658
• [x] Unique labels in string interpolation tests. #55
• [x] expr refactor: better interface #40
• [x] Refactor logic for workflow_step.input_type into model. #10657
• [x] Fix ObjectUploadTarget handling in tool util staging code. #10650
• [x] Late Monday fixes and enhancements https://github.com/common-workflow-language/galaxy/pull/125
  • [x] Fix running recursive conformance tests imported introduced in CWL 1.2 test suite (previous fix #124 had a bug).
  • [x] When pulling workflow output CWL outputs, handle non-data/non-collection datatypes (i.e. simple values). Needed for null and string outputs of a conditional test case.
  • [x] When evaluating expressions for workflows, walk inputs not consumed by the underlying tool also (kind of messy).
  • [x] Add pytest marks for each test tag found in conformance_tests.yaml when generating Python test suites, allows for much more flexible testing combinations.
  • [x] When generating test suites in Python suite, print some common problems so we can fix up the source test specifications.
  • [x] Bring in fixes for the cwl v1.1 conformance tests file (xref https://github.com/common-workflow-language/cwl-v1.1/pull/72).
• [x] Improvements to workflow test monitoring for upstream https://github.com/galaxyproject/galaxy/pull/10668
• [x] Upstream test verifying structure in tool state of optional parameters https://github.com/galaxyproject/galaxy/pull/10669
• [x] Fix regressions related to optional parameters and field types (https://github.com/common-workflow-language/galaxy/pull/127)
• [x] Migrate API tests to GitHub workflow https://github.com/galaxyproject/galaxy/pull/10674
• [x] Re-sync galaxy.tool_util.cwl with CWL branch of Galaxy https://github.com/galaxyproject/galaxy/pull/10704
• [ ] CWL style workflow conditionals
  • [x] Done for CWL in Galaxy: https://github.com/common-workflow-language/galaxy/pull/123
  • [ ] Build up tests and abstractions required to have conditionals for Galaxy workflows - so we can upstream expressions in workflows.
  • [x] Add to format2/ga https://github.com/galaxyproject/gxformat2/pull/48/files
  • [x] gxformat2 plumbing to allow dev releases https://github.com/galaxyproject/gxformat2/pull/49
  • [x] Harden, formalize 'null' handling around boolean tool parameters https://github.com/galaxyproject/galaxy/pull/10694
  • [x] Allow expression tools to output copies of input HDAs https://github.com/galaxyproject/galaxy/pull/10700
  • [x] Null handling and optional values for integers, floats, and text params https://github.com/galaxyproject/galaxy/pull/10703
  • [x] Implement expression tool for picking non-null values for CWL-style merging of conditional branches in native Galaxy workflows https://github.com/galaxyproject/galaxy/pull/10712

New Issues

• [ ] Parse creator and license information from CWL documents. #117
• [x] Migrate Travis Tests to Github Actions #121
• [ ] Figure out plan for pickValue synchronization across steps #126
• [x] Rework resource handling for cwltool>=3.0.20200317142916 #116
• [x] Update CWL Branch #115: @jmchilton
  • [X] Rebase against latest Galaxy.
  • [X] Update commit that brings in conformance test data to bring 1.1 and 1.2 conformance tests.
  • [x] Updated the test case generation code to generate news test classes for each version (v1.0, v1.1, and v1.2) corresponding to red and green "required" tests as well as regressions.
  • [X] Re-run red tests and add 36 news test to green list.
  • [X] Refactor to use new client staging stuff (resulting in https://github.com/galaxyproject/galaxy/pull/10650 merged by Marius)
  • [X] Fix some small issue with a refactoring to cleanup CWL branch, caused nested workflows to break (https://github.com/galaxyproject/galaxy/pull/10657/files).
  • [X] Various fixes for test generation, directory datatype.
  • [X] Updated conformance suite to test case code for newer recursive imports.
  • [X] Categorized the remaining regressions into 4 likely bugs, documented alone side the red/green list code.
  • [x] More correct, robust should_fail handling for conformance tests (https://github.com/common-workflow-language/galaxy/pull/130). Needed for conditional tests.

to do:

• [x] What happened to the code to integrate the CWL Expresion refactoring tool? https://github.com/galaxyproject/galaxy/commit/9a8855d0dd91487e5cecf53b7a3d0ff3d49bb98c https://github.com/galaxyproject/galaxy/compare/dev...common-workflow-language:cwl-1.0-rewrite-expressions-2?expand=1

• [ ] Presenting non-zero (and ideally correct) information on the memory and disk space reservation in the CWL $(runtime) object

• [x] recover tests that used to pass in this branch, but don't currently.

Bonus / semi-related work by team-members

• [x] Full Python dependencies update #10660
• [x] Update mercurial to 5.6 #73

[mr-c]

Since all unchecked items have an issue or a pull request open, and the #BioHackEU20 is over I'm going to close this tracking issue. Many thanks to @jmchilton @mvdbeek @nsoranzo for their contributions this last week!