common-workflow-language / cwltool

Common Workflow Language reference implementation
https://cwltool.readthedocs.io/
Apache License 2.0
324 stars 225 forks source link

Fix 2d output arrays with format tag #2010

Open vjaganat90 opened 1 week ago

vjaganat90 commented 1 week ago

The issue is when we print 2d_output arrays which have "format" tag

touch filename.txt cwltool --cachedir foo output_2D_file_format.cwl

("Error collecting output for parameter 'output_array': output_2D_file_format.cwl:14:3: list indices must
                                                                                        be integers or
                                                                                        slices, not str",
                                                                                        {})

The crux of the issue is aslist function, only promotes scalars to 1d arrays. Arbitrary JS can return an arbitrary json, format handling code needs to handle arbitrary json (not just scalars or 1d arrays which is done through aslist function).

With this fix the output should be something like this

jaganathv2$ touch filename.txt
jaganathv2$ cwltool --cachedir foo output_2D_file_format.cwl
INFO /Users/jaganathv2/mambaforge/envs/cwt/bin/cwltool 3.1.20220628170239.dev281+g57619ffd
INFO Resolved 'output_2D_file_format.cwl' to 'file:///Users/jaganathv2/Code/WIC/cwltool/tests/output_2D_file_format.cwl'
INFO [job output_2D_file_format.cwl] Output of job will be cached in /Users/jaganathv2/Code/WIC/cwltool/tests/foo/585a1a734523fb1e13030642dc26ad4a
INFO [job output_2D_file_format.cwl] /Users/jaganathv2/Code/WIC/cwltool/tests/foo/585a1a734523fb1e13030642dc26ad4a$ true
INFO [job output_2D_file_format.cwl] completed success
{
    "output_array": [
        [
            {
                "class": "File",
                "location": "file:///Users/jaganathv2/Code/WIC/cwltool/tests/filename.txt",
                "format": "file:///Users/jaganathv2/Code/WIC/cwltool/tests/output_2D_file_format.cwl#some_format",
                "basename": "filename.txt",
                "size": 0,
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "path": "/Users/jaganathv2/Code/WIC/cwltool/tests/filename.txt"
            },
            {
                "class": "File",
                "location": "file:///Users/jaganathv2/Code/WIC/cwltool/tests/filename.txt",
                "format": "file:///Users/jaganathv2/Code/WIC/cwltool/tests/output_2D_file_format.cwl#some_format",
                "basename": "filename.txt",
                "size": 0,
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "path": "/Users/jaganathv2/Code/WIC/cwltool/tests/filename.txt"
            }
        ],
        [
            {
                "class": "File",
                "location": "file:///Users/jaganathv2/Code/WIC/cwltool/tests/filename.txt",
                "format": "file:///Users/jaganathv2/Code/WIC/cwltool/tests/output_2D_file_format.cwl#some_format",
                "basename": "filename.txt",
                "size": 0,
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "path": "/Users/jaganathv2/Code/WIC/cwltool/tests/filename.txt"
            },
            {
                "class": "File",
                "location": "file:///Users/jaganathv2/Code/WIC/cwltool/tests/filename.txt",
                "format": "file:///Users/jaganathv2/Code/WIC/cwltool/tests/output_2D_file_format.cwl#some_format",
                "basename": "filename.txt",
                "size": 0,
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "path": "/Users/jaganathv2/Code/WIC/cwltool/tests/filename.txt"
            }
        ]
    ]
}INFO Final process status is success
codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 66.66667% with 3 lines in your changes missing coverage. Please review.

Project coverage is 83.96%. Comparing base (9473f13) to head (4700f30). Report is 4 commits behind head on main.

Files Patch % Lines
cwltool/command_line_tool.py 66.66% 1 Missing and 2 partials :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2010 +/- ## ========================================== + Coverage 83.86% 83.96% +0.09% ========================================== Files 46 46 Lines 8244 8251 +7 Branches 2190 2194 +4 ========================================== + Hits 6914 6928 +14 + Misses 853 841 -12 - Partials 477 482 +5 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

mr-c commented 19 hours ago

Thank you @vjaganat90 ! I've made a note to turn your example into a new conformance test for CWL: https://github.com/common-workflow-language/cwl-v1.3/issues/31