search

common-workflow-language / cwltool

Common Workflow Language reference implementation
https://cwltool.readthedocs.io/
Apache License 2.0
324 stars 225 forks source link

flaky conformance test with `cwltool`: simple_simple_scatter #2003

Open mr-c opened 1 month ago

mr-c commented 1 month ago

This has been failing randomly for over a year

https://github.com/common-workflow-language/cwltool/actions/runs/9138932694/job/25130670885#step:7:471

Tests are being run with the cwltest plugin for pytest:

PYTEST_EXTRA="--cwl-include simple_simple_scatter" bash conformance-test.sh (possibly with SKIP_INSTALL=1 at the beginning if you are hacking oncwltool` and it is already installed)

which is roughly equivalent to

cwltest --test https://github.com/common-workflow-language/cwl-v1.2/raw/main/conformance_tests.yaml -s simple_simple_scatter -- --parallel

or more directly

cwltool --parallel https://github.com/common-workflow-language/cwl-v1.2/raw/main/tests/scatter/simple-simple-scatter.cwl https://github.com/common-workflow-language/cwl-v1.2/raw/main/tests/scatter/scatter-job.yml

Running any of the above produces a null output for me (which would fail the test) about 1 out of every 4 times.

CWL test execution failed. 
expected: {
    "result": [
        [
            "^1a$",
            "^2a$",
            "^3a$",
            "^4a$"
        ],
        [
            "^1b$",
            "^2b$",
            "^3b$",
            "^4b$"
        ],
        [
            "^1c$",
            "^2c$",
            "^3c$",
            "^4c$"
        ],
        [
            "^1d$",
            "^2d$",
            "^3d$",
            "^4d$"
        ]
    ]
got: [
    [
        "^1a$",
        "^2a$",
        "^3a$",
        "^4a$"
    ],
    [
        "^1b$",
        "^2b$",
        "^3b$",
        "^4b$"
    ],
    [
        "^1c$",
        "^2c$",
        "^3c$",
        "^4c$"
    ],
    null
]
caused by: expected: [
    "^1d$",
    "^2d$",
    "^3d$",
    "^4d$"
]
got: null
Test: job: 
  file:///home/runner/work/cwltool/cwltool/cwl-v1.2-main/tests/scatter/scatter-job.yml
id: 
  file:///home/runner/work/cwltool/cwltool/cwl-v1.2-main/tests/scatter/test-index.yaml#simple_simple_scatter
tool: 
  file:///home/runner/work/cwltool/cwltool/cwl-v1.2-main/tests/scatter/simple-simple-scatter.cwl
output:
  result:
  - - ^1a$
    - ^2a$
    - ^3a$
    - ^4a$
  - - ^1b$
    - ^2b$
    - ^3b$
    - ^4b$
  - - ^1c$
    - ^2c$
    - ^3c$
    - ^4c$
  - - ^1d$
    - ^2d$
    - ^3d$
    - ^4d$
doc: Two level nested scatter
tags:
- workflow
- subworkflow
- scatter
- inline_javascript
- expression_tool