search

common-workflow-language / cwltest

Framework for testing CWL tools and workflows
Apache License 2.0
17 stars 18 forks source link

test failed, not sure why #94

Open wgerlach opened 5 years ago

wgerlach commented 5 years ago

Hi, I am having problems passing test 107 with my implementation and do not understand what is wrong. The File object for file a should be what is expected, still cwltest complains. Any ideas?

Test [107/197] Test if a writable input directory is recursively copied and writable
Test 107 failed: /go/bin/awe-cwl-submitter-wrapper.sh --outdir=/tmp/tmp2k39bljr --quiet v1.0/recursive-input-directory.cwl v1.0/recursive-input-directory.yml
Test if a writable input directory is recursively copied and writable
Compare failure expected: {
    "output_dir": {
        "basename": "work_dir",
        "class": "Directory",
        "listing": [
            {
                "basename": "a",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "class": "File",
                "location": "work_dir/a",
                "size": 0
            },
            {
                "basename": "b",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "class": "File",
                "location": "work_dir/b",
                "size": 0
            },
            {
                "basename": "c",
                "class": "Directory",
                "listing": [
                    {
                        "basename": "d",
                        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                        "class": "File",
                        "location": "work_dir/c/d",
                        "size": 0
                    }
                ],
                "location": "work_dir/c"
            },
            {
                "basename": "e",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "class": "File",
                "location": "work_dir/e",
                "size": 0
            }
        ],
        "location": "work_dir"
    },
    "test_result": {
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "class": "File",
        "location": "output.txt",
        "size": 0
    }
}
got: {
    "output_dir": {
        "basename": "work_dir",
        "class": "Directory",
        "listing": [
            {
                "basename": "a",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "class": "File",
                "location": "work_dir/a",
                "size": 0
            },
            {
                "basename": "c",
                "class": "Directory",
                "listing": [
                    {
                        "basename": "d",
                        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                        "class": "File",
                        "location": "work_dir/c/d",
                        "size": 0
                    }
                ],
                "location": "c"
            },
            {
                "basename": "e",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "class": "File",
                "location": "work_dir/e",
                "size": 0
            },
            {
                "basename": "b",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "class": "File",
                "location": "work_dir/b",
                "size": 0
            }
        ],
        "location": "work_dir"
    },
    "test_result": {
        "basename": "output.txt",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "class": "File",
        "location": "input_807229205/output.txt",
        "nameext": ".txt",
        "size": 0
    }
}
caused by: failed comparison for key 'output_dir': expected: {
    "basename": "work_dir",
    "class": "Directory",
    "listing": [
        {
            "basename": "a",
            "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
            "class": "File",
            "location": "work_dir/a",
            "size": 0
        },
        {
            "basename": "b",
            "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
            "class": "File",
            "location": "work_dir/b",
            "size": 0
        },
        {
            "basename": "c",
            "class": "Directory",
            "listing": [
                {
                    "basename": "d",
                    "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                    "class": "File",
                    "location": "work_dir/c/d",
                    "size": 0
                }
            ],
            "location": "work_dir/c"
        },
        {
            "basename": "e",
            "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
            "class": "File",
            "location": "work_dir/e",
            "size": 0
        }
    ],
    "location": "work_dir"
}
got: {
    "basename": "work_dir",
    "class": "Directory",
    "listing": [
        {
            "basename": "a",
            "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
            "class": "File",
            "location": "work_dir/a",
            "size": 0
        },
        {
            "basename": "c",
            "class": "Directory",
            "listing": [
                {
                    "basename": "d",
                    "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                    "class": "File",
                    "location": "work_dir/c/d",
                    "size": 0
                }
            ],
            "location": "c"
        },
        {
            "basename": "e",
            "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
            "class": "File",
            "location": "work_dir/e",
            "size": 0
        },
        {
            "basename": "b",
            "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
            "class": "File",
            "location": "work_dir/b",
            "size": 0
        }
    ],
    "location": "work_dir"
}
caused by: {
    "basename": "a",
    "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
    "class": "File",
    "location": "work_dir/a",
    "size": 0
} not found
0 tests passed, 1 failures, 0 unsupported features

1 tool tests failed

thx

tom-tan commented 5 years ago

How about sorting the file and directory objects in the listing field for work_dir by their names?

I guess the conformance test 107 implicitly expects that they are sorted but such a behavior is not documented in the spec. I am not sure it is mis-documented spec or it accidentally depends on a specific implementation such as cwltool.

However, IMO, the above output object should be accepted by cwltest (if the corresponding files and directories exist) because the spec of v1.1 says:

- Clarify that unspecified details related to execution are left open to the platform.

It is the same in v1.0 because v1.1 has no breaking changes from v1.0, I guess.

What do you think, @mr-c?

Maybe related: common-workflow-language/cwl-v1.2#233

mr-c commented 5 years ago

Hey @wgerlach and @tom-tan

CWL is built upon the POSIX standards. In https://www.commonwl.org/v1.0/CommandLineTool.html#CommandOutputBinding is

glob Find files relative to the output directory, using POSIX glob(3) pathname matching.

POSIX glob(3) states that the result mist be sorted according to the current locale.

I don't know of any locale that sorts the letters "a", "b", "c", "d", "e" in a different order.

tom-tan commented 5 years ago

@mr-c Thanks for the information and sorry for the misleading comment about related issue.

common-workflow-language/cwl-v1.2#233 is related but not the same issue. This issue is about the sorting order of the objects in listing field, not the result of glob field.

As you mentioned, the result of glob should be sorted and it is described in the spec. However there are no descriptions of the sorting order of listing field in directory object.

wgerlach commented 5 years ago

I figured the problem seems to be the cwltool!? Searching for the place in my code where the sort order gets messes up I noticed that it happens when my implementation invokes the cwltool to execute my CommandLineTool. Note that my implementation parses the original CommandLineTool and later, upon execution of a step, writes the CommandLineTool to disk. This CommandLineTool can be different from than the original compliance test, but should be equivalent.

Below are the tool, input and result of running cwltool. While the input is correctly sorted, the output is not. Of course, I could work around this by sorting the results again in my implementation, but it seems to me this should not have happened anyway. Any ideas?

cwl_tool.yaml

class: CommandLineTool
requirements:
- class: InitialWorkDirRequirement
  listing:
  - entry: $(inputs.input_dir)
    entryname: work_dir
    writable: true
- class: ShellCommandRequirement
hints:
- class: ResourceRequirement
  ramMin: 8
inputs:
- id: input_dir
  type:
  - Directory
outputs:
- id: output_dir
  outputBinding:
    glob:
    - work_dir
  type: Directory
- id: test_result
  type: stdout
cwlVersion: v1.0
arguments:
- valueFrom: |
    touch work_dir/e;
    if [ ! -w work_dir ]; then echo work_dir not writable; fi;
    if [ -L work_dir ]; then echo work_dir is a symlink; fi;
    if [ ! -w work_dir/a ]; then echo work_dir/a not writable; fi;
    if [ -L work_dir/a ]; then echo work_dir/a is a symlink; fi;
    if [ ! -w work_dir/c ]; then echo work_dir/c not writable; fi;
    if [ -L work_dir/c ]; then echo work_dir/c is a symlink; fi;
    if [ ! -w work_dir/c/d ]; then echo work_dir/c/d not writable; fi;
    if [ -L work_dir/c/d ]; then echo work_dir/c/d is a symlink; fi;
    if [ ! -w work_dir/e ]; then echo work_dir/e not writable; fi;
    if [ -L work_dir/e ]; then echo work_dir/e is a symlink ; fi;
  shellQuote: false
stdout: output.txt

cwl_job_input.yaml

input_dir:
  class: Directory
  location: testdir
  listing:
  - class: File
    location: testdir/a
    basename: a
    checksum: sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709
    size: 0
  - class: File
    location: testdir/b
    basename: b
    checksum: sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709
    size: 0
  - class: Directory
    basename: c
    listing:
    - class: File
      location: testdir/c/d
      basename: d
      checksum: sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709
      size: 0

cwltool cwl_tool.yaml cwl_job_input.yaml

INFO /usr/bin/cwltool 1.0.20190621173317
INFO Resolved 'cwl_tool.yaml' to 'file:///Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/cwl_tool.yaml'
INFO [job cwl_tool.yaml] /tmp/zfeso6q4$ /bin/sh \
    -c \
    touch work_dir/e;
if [ ! -w work_dir ]; then echo work_dir not writable; fi;
if [ -L work_dir ]; then echo work_dir is a symlink; fi;
if [ ! -w work_dir/a ]; then echo work_dir/a not writable; fi;
if [ -L work_dir/a ]; then echo work_dir/a is a symlink; fi;
if [ ! -w work_dir/c ]; then echo work_dir/c not writable; fi;
if [ -L work_dir/c ]; then echo work_dir/c is a symlink; fi;
if [ ! -w work_dir/c/d ]; then echo work_dir/c/d not writable; fi;
if [ -L work_dir/c/d ]; then echo work_dir/c/d is a symlink; fi;
if [ ! -w work_dir/e ]; then echo work_dir/e not writable; fi;
if [ -L work_dir/e ]; then echo work_dir/e is a symlink ; fi;
 > /tmp/zfeso6q4/output.txt
INFO Could not collect memory usage, job ended before monitoring began.
INFO [job cwl_tool.yaml] completed success
{
    "output_dir": {
        "location": "file:///Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/work_dir",
        "basename": "work_dir",
        "class": "Directory",
        "listing": [
            {
                "class": "File",
                "location": "file:///Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/work_dir/a",
                "basename": "a",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "size": 0,
                "path": "/Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/work_dir/a"
            },
            {
                "class": "Directory",
                "location": "file:///Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/work_dir/c",
                "basename": "c",
                "listing": [
                    {
                        "class": "File",
                        "location": "file:///Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/work_dir/c/d",
                        "basename": "d",
                        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                        "size": 0,
                        "path": "/Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/work_dir/c/d"
                    }
                ],
                "path": "/Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/work_dir/c"
            },
            {
                "class": "File",
                "location": "file:///Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/work_dir/e",
                "basename": "e",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "size": 0,
                "path": "/Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/work_dir/e"
            },
            {
                "class": "File",
                "location": "file:///Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/work_dir/b",
                "basename": "b",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "size": 0,
                "path": "/Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/work_dir/b"
            }
        ],
        "path": "/Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/work_dir"
    },
    "test_result": {
        "location": "file:///Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/output.txt",
        "basename": "output.txt",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "path": "/Users/wolfganggerlach/awe_data/work/e9/f4/38/e9f43857-f502-41d4-8735-c68c4c763f6d__entrypoint_wrapper_step_0/output.txt"
    }
}
INFO Final process status is **success**
tetron commented 5 years ago

I don't think it is an order issue, cwltest already does an order-independent comparison for listings:

https://github.com/common-workflow-language/cwltest/blob/master/cwltest/utils.py#L120

The comparison function is failing to find a, although it looks like it is there. There may be a very subtle difference between "actual" and "expected" that cwltest is not reporting in a useful way (and should probably ignore).

wgerlach commented 5 years ago

I think I found the problem in the compare_location function, but still do not fully understand this:

https://github.com/common-workflow-language/cwltest/blob/master/cwltest/utils.py#L79 The compare_location function compares this field, which is identical in actual and expected:

"location": "work_dir/a",

It seems that if they are identical it is expected that no "/" is supposed to be in the location string of the actual. I am not sure why this is required here, but that seems to be the problem.

The other problem is the missing error message. While an exception is thrown in compare_location, it is catched later and ignored, because the previous function compares all listing elements with each other to find matching pairs. I am not sure if there is a better solution to this. Would it be a safe assumption that the basename field always exists? If so, you could first search for matching basename, then compare the object and throw an exception if needed.