search

common-workflow-language / cwltool

Common Workflow Language reference implementation
https://cwltool.readthedocs.io/
Apache License 2.0
335 stars 230 forks source link

Staging input files in a non-writable sub-directory fails on Linux #1042

Open bogdang989 opened 5 years ago

bogdang989 commented 5 years ago

Expected Behavior

Completed run.

Actual Behavior

Executing on Mac this completes, but on Linux it fails due to trying to delete and not having write permissions for the staged file (discovered by Jenkins, confirmed on a local Ubuntu machine).

Setting the folder to be staged as writable: true gets past this error as it avoids calling os.remove(host_outdir_tgt) ``

Workflow Code

Workflow code can be found here: https://github.com/common-workflow-language/common-workflow-language/pull/820/commits/09a897514c7a9079da8f3cfde858e30f695ef0ae

class: CommandLineTool
cwlVersion: v1.0
baseCommand:
  - ls
  - staged
inputs:
  - id: infiles
    type: File[]
outputs:
  - id: outfile
    type: File
    outputBinding:
      glob: staged/whale.txt
requirements:
  - class: InitialWorkDirRequirement
    listing:
      - entry: >-
          ${ return { 'class': 'Directory', 'listing': inputs.infiles,
          'basename': 'staged'} }
  - class: InlineJavascriptRequirement
hints:
  - class: DockerRequirement
    dockerPull: 'debian:stretch-slim'

Full Traceback

Exception while running job
Traceback (most recent call last):
  File "/home/bogdan/Documents/cwltool/cwltool/job.py", line 329, in _execute
    inplace_update=self.inplace_update)
  File "/home/bogdan/Documents/cwltool/cwltool/job.py", line 148, in relink_initialworkdir
    os.remove(host_outdir_tgt)
PermissionError: [Errno 13] Permission denied: '/tmp/uue6d20d/staged/whale.txt'
[job stage_array_in_dir.cwl] completed permanentFail
{}
Final process status is permanentFail

Your Environment

mr-c commented 5 years ago

Why doesn't it have write permissions?

bogdang989 commented 5 years ago

I am not sure why tbh. It is quite strange, running the same tool on Mac works fine but on Linux it fails (both on the Jenkins run and local Ubuntu). In debug mode, when I stop at the line os.remove(host_outdir_tgt)- https://github.com/common-workflow-language/cwltool/blob/master/cwltool/job.py#L148 and evaluate os.access(host_outdir_tgt, os.W_OK), I get True on Mac and False on Ubuntu. If I am not mistaken, this is docker user's permission for a file on a mounted folder.

alexiswl commented 2 years ago

I think I've figured out the issue here (although I am not sure of the best solution).

tl;dr it's basically docker being silly with nested mount paths. Why it's fine on mac and not linux I don't know.

cwltool version: 3.1.20211107152837

If I run the cwltool below through docker I get the following logs:

INFO /usr/local/bin/cwltool 3.1.20211107152837
INFO Resolved 'whale-tool.cwl' to 'file:///home/ec2-user/whale-tool.cwl'
DEBUG [job whale-tool.cwl] initializing from file:///home/ec2-user/whale-tool.cwl
DEBUG [job whale-tool.cwl] {
    "infiles": [
        {
            "class": "File",
            "location": "file:///home/ec2-user/whale.txt",
            "size": 11,
            "basename": "whale.txt",
            "nameroot": "whale",
            "nameext": ".txt"
        }
    ]
}
DEBUG [job whale-tool.cwl] path mappings is {
    "file:///home/ec2-user/whale.txt": [
        "/home/ec2-user/whale.txt",
        "/eVOEcE/staged/whale.txt",
        "File",
        false
    ]
}
DEBUG [job whale-tool.cwl] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "ls"
    },
    {
        "position": [
            -1000000,
            1
        ],
        "datum": "staged"
    }
]
DEBUG [job whale-tool.cwl] initial work dir {
    "_:3d2fc630-2d3c-4b8d-b572-60d2834c4347": [
        "_:3d2fc630-2d3c-4b8d-b572-60d2834c4347",
        "/eVOEcE/staged",
        "Directory",
        true
    ],
    "file:///home/ec2-user/whale.txt": [
        "/home/ec2-user/whale.txt",
        "/eVOEcE/staged/whale.txt",
        "File",
        true
    ]
}
INFO [job whale-tool.cwl] /tmp/s003k18w$ docker \
    run \
    -i \
    --mount=type=bind,source=/tmp/s003k18w,target=/eVOEcE \
    --mount=type=bind,source=/tmp/c_tvtgb9,target=/tmp \
    --mount=type=bind,source=/home/ec2-user/whale.txt,target=/eVOEcE/staged/whale.txt,readonly \
    --workdir=/eVOEcE \
    --read-only=true \
    --user=1000:1000 \
    --rm \
    --cidfile=/tmp/hb7_imwb/20220112220125-982650.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/eVOEcE \
    debian:stretch-slim \
    ls \
    staged
whale.txt
INFO [job whale-tool.cwl] Max memory used: 0MiB
INFO [job whale-tool.cwl] completed success
DEBUG [job whale-tool.cwl] outputs {
    "outfile": {
        "location": "/home/ec2-user/whale.txt",
        "basename": "whale.txt",
        "nameroot": "whale",
        "nameext": ".txt",
        "class": "File",
        "checksum": "sha1$c75fe8a7be45b1120f8da36d8c1ba7c3ac895185",
        "size": 11,
        "http://commonwl.org/cwltool#generation": 0
    }
}
DEBUG [job whale-tool.cwl] Removing input staging directory /tmp/8e6z44ce
DEBUG [job whale-tool.cwl] Removing temporary directory /tmp/c_tvtgb9
DEBUG Removing intermediate output directory /tmp/s003k18w
INFO Final process status is success

Note that /tmp/442nvwe9 is mounted to /SMiCml inside the container but /home/ec2-user/whale.txt is mounted to //eVOEcE/staged/whale.txt as read-only.

After running the workflow we see that some stagnant empty files are left over.

$ find /tmp/s003k18w/

/tmp/s003k18w/
/tmp/s003k18w/staged
/tmp/s003k18w/staged/whale.txt

If we have a look at the whale.txt file there we see that it's empty and it's owned by root.

$ ls -lsrth /tmp/s003k18w/staged/whale.txt
0 -rwxr-xr-x 1 root root 0 Jan 12 22:01 /tmp/s003k18w/staged/whale.txt

This file cannot be deleted by the user

$ rm  /tmp/s003k18w/staged/whale.txt
rm: remove write-protected regular empty file ‘/tmp/s003k18w/staged/whale.txt’? y
rm: cannot remove ‘/tmp/s003k18w/staged/whale.txt’: Permission denied

Nor does it have the same ionode as the original file

$ stat whale.txt
  File: ‘whale.txt’
  Size: 11              Blocks: 8          IO Block: 4096   regular file
Device: 10301h/66305d   Inode: 1262513     Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/ec2-user)   Gid: ( 1000/ec2-user)
Access: 2022-01-12 22:00:32.657353343 +0000
Modify: 2022-01-12 22:00:32.657353343 +0000
Change: 2022-01-12 22:00:32.657353343 +0000
 Birth: -
$ stat /tmp/s003k18w/staged/whale.txt
  File: ‘/tmp/s003k18w/staged/whale.txt’
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 10301h/66305d   Inode: 4943920     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-01-12 22:01:26.169507599 +0000
Modify: 2022-01-12 22:01:26.169507599 +0000
Change: 2022-01-12 22:01:26.169507599 +0000
 Birth: -

Reproduce with:

CWLTOOL:

class: CommandLineTool
cwlVersion: v1.0
baseCommand:
  - ls
  - staged
inputs:
  - id: infiles
    type: File[]
outputs:
  - id: outfile
    type: File
    outputBinding:
      glob: staged/whale.txt
requirements:
  - class: InitialWorkDirRequirement
    listing:
      - entry: >-
          ${ return { 'class': 'Directory', 'listing': inputs.infiles,
          'basename': 'staged'} }
  - class: InlineJavascriptRequirement
hints:
  - class: DockerRequirement
    dockerPull: 'debian:stretch-slim'

Input:

infiles:
  - class: File
    location: whale.txt

whale.txt

$ cat whale.txt
Free willy