search

ICGC-TCGA-PanCancer / CGP-Somatic-Docker

Dockerised and simplified version of SeqWare-CGP-SomaticCore
GNU Affero General Public License v3.0
14 stars 9 forks source link

ERROR io.dockstore.common.FileProvisioning - Could not link #10

Closed brainstorm closed 7 years ago

brainstorm commented 7 years ago

Following up a bit on https://github.com/ICGC-TCGA-PanCancer/CGP-Somatic-Docker/issues/9, after installing the required dependencies for dockstore as pointed out in the tool's onboarding:

Successfully installed cwl-runner-1.0 cwltool-1.0.20161007181528 ruamel.yaml-0.11.11 schema-salad-1.14.20160708181155

[rvalls@node08 CGP-Somatic-Docker]$ java -version
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

I got hit by this error:

[rvalls@node08 CGP-Somatic-Docker]$ dockstore tool launch --entry Dockstore.cwl --local-entry --json test1.json
Creating directories for run of Dockstore launcher at: ./datastore//launcher-778d4d57-da1a-4d17-860f-8799612550ee
Provisioning your input files to your local machine
Downloading: #bbFrom from https://s3-eu-west-1.amazonaws.com/wtsi-pancancer/reference/GRCh37d5_battenberg.tar.gz into directory: /home/rvalls/dev/CGP-Somatic-Docker/./datastore/launcher-778d4d57-da1a-4d17-860f-8799612550ee/inputs/ca8b6bca-d1d6-404a-a62c-30df36b8922d
[##################################################] 100%
Downloading: #normal from /media/large_volume/CGP-Somatic-Docker-input/HCC1143_ds/HCC1143_ds/HCC1143_BL.bam into directory: /home/rvalls/dev/CGP-Somatic-Docker/./datastore/launcher-778d4d57-da1a-4d17-860f-8799612550ee/inputs/336e0150-0240-492a-8047-432eec69abf9
16:28:37.266 [main] ERROR io.dockstore.common.FileProvisioning - Could not link /media/large_volume/CGP-Somatic-Docker-input/HCC1143_ds/HCC1143_ds/HCC1143_BL.bam to /home/rvalls/dev/CGP-Somatic-Docker/./datastore/launcher-778d4d57-da1a-4d17-860f-8799612550ee/inputs/336e0150-0240-492a-8047-432eec69abf9/HCC1143_BL.bam , copying instead
java.nio.file.NoSuchFileException: /home/rvalls/dev/CGP-Somatic-Docker/./datastore/launcher-778d4d57-da1a-4d17-860f-8799612550ee/inputs/336e0150-0240-492a-8047-432eec69abf9/HCC1143_BL.bam -> /media/large_volume/CGP-Somatic-Docker-input/HCC1143_ds/HCC1143_ds/HCC1143_BL.bam
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
        at java.nio.file.Files.createLink(Files.java:1086)
        at io.dockstore.common.FileProvisioning.provisionInputFile(FileProvisioning.java:246)
        at io.github.collaboratory.LauncherCWL.copyIndividualFile(LauncherCWL.java:669)
        at io.github.collaboratory.LauncherCWL.doProcessFile(LauncherCWL.java:635)
        at io.github.collaboratory.LauncherCWL.pullFilesHelper(LauncherCWL.java:606)
        at io.github.collaboratory.LauncherCWL.pullFiles(LauncherCWL.java:562)
        at io.github.collaboratory.LauncherCWL.run(LauncherCWL.java:184)
        at io.dockstore.client.cli.nested.AbstractEntryClient.launchCwl(AbstractEntryClient.java:897)
        at io.dockstore.client.cli.nested.AbstractEntryClient.checkEntryFile(AbstractEntryClient.java:748)
        at io.dockstore.client.cli.nested.AbstractEntryClient.launch(AbstractEntryClient.java:821)
        at io.dockstore.client.cli.nested.AbstractEntryClient.processEntryCommands(AbstractEntryClient.java:218)
        at io.dockstore.client.cli.Client.run(Client.java:698)
        at io.dockstore.client.cli.Client.main(Client.java:759)
java.lang.RuntimeException: Could not copy /media/large_volume/CGP-Somatic-Docker-input/HCC1143_ds/HCC1143_ds/HCC1143_BL.bam to /home/rvalls/dev/CGP-Somatic-Docker/./datastore/launcher-778d4d57-da1a-4d17-860f-8799612550ee/inputs/336e0150-0240-492a-8047-432eec69abf9/HCC1143_BL.bam

I realized that test1.json must be edited to a local path which has enough disk space, and so did I but still fails as shown above, with the new path:

16:39:13.403 [main] ERROR io.dockstore.common.FileProvisioning - Could not link /scratch/rvalls/CGP-Somatic-Docker-input/HCC1143_ds/HCC1143_ds/HCC1143_BL.bam to /home/rvalls/dev/CGP-Somatic-Docker/./datastore/launcher-0683728d-b4b8-422a-a7c3-500fe29a6ec3/inputs/c7fc2900-8952-4a55-9a6b-30bcef473758/HCC1143_BL.bam , copying instead
java.nio.file.NoSuchFileException: /home/rvalls/dev/CGP-Somatic-Docker/./datastore/launcher-0683728d-b4b8-422a-a7c3-500fe29a6ec3/inputs/c7fc2900-8952-4a55-9a6b-30bcef473758/HCC1143_BL.bam -> /scratch/rvalls/CGP-Somatic-Docker-input/HCC1143_ds/HCC1143_ds/HCC1143_BL.bam

Even when I mkdir -p part of the dir hierarchy manually on the destination storage:

[rvalls@node08 CGP-Somatic-Docker]$ ls /home/rvalls/dev/CGP-Somatic-Docker/./datastore/launcher-0683728d-b4b8-422a-a7c3-500fe29a6ec3/inputs/c7fc2900-8952-4a55-9a6b-30bcef473758
[rvalls@node08 CGP-Somatic-Docker]$ ls /home/rvalls/dev/CGP-Somatic-Docker/./datastore/launcher-0683728d-b4b8-422a-a7c3-500fe29a6ec3/
configs  inputs  logs  outputs  working

Have you encountered this error before, @denis-yuen?

denis-yuen commented 7 years ago

@brainstorm Hi, I think that what is going on here is not a lack of disk space, but I think you're actually missing the input files or they are in the wrong place. https://github.com/ICGC-TCGA-PanCancer/CGP-Somatic-Docker#running-with-the-dockstore-command-line Indicates that you need to download the input files (also available here https://dcc.icgc.org/releases/PCAWG/reference_data/wtsi-pancancer/testdata ), unzip them, and place them in the path that is in test1.json.

i.e. 

java.nio.file.NoSuchFileException: /home/rvalls/dev/CGP-Somatic-Docker/./datastore/launcher-0683728d-b4b8-422a-a7c3-500fe29a6ec3/inputs/c7fc2900-8952-4a55-9a6b-30bcef473758/HCC1143_BL.bam -> /scratch/rvalls/CGP-Somatic-Docker-input/HCC1143_ds/HCC1143_ds/HCC1143_BL.bam

seems to indicate that "/scratch/rvalls/CGP-Somatic-Docker-input/HCC1143_ds/HCC1143_ds/HCC1143_BL.bam" does not exist. Watch out that it looks like you have a doubled up directory name there.

brainstorm commented 7 years ago

Doh, sorry, thanks a ton @denis-yuen, I will examine the docs and paths more closely next time.

Now it works but you might want to catch the following Invalid cross-device link exception? Looks just like a warning because I'm using two different mountpoints (are you using hardlinks by any chance?), but could confuse users, so worth catching it early I guess:

Downloading: #normal from /scratch/rvalls/HCC1143_ds/HCC1143_BL.bam into directory: /home/rvalls/dev/CGP-Somatic-Docker/./datastore/launcher-0aec7ac8-ebd0-4219-8c85-4b4fa9beb58b/inputs/7a32ceea-9c8a-4901-80d9-e46f0c086919
17:02:39.351 [main] ERROR io.dockstore.common.FileProvisioning - Could not link /scratch/rvalls/HCC1143_ds/HCC1143_BL.bam to /home/rvalls/dev/CGP-Somatic-Docker/./datastore/launcher-0aec7ac8-ebd0-4219-8c85-4b4fa9beb58b/inputs/7a32ceea-9c8a-4901-80d9-e46f0c086919/HCC1143_BL.bam , copying instead
java.nio.file.FileSystemException: /home/rvalls/dev/CGP-Somatic-Docker/./datastore/launcher-0aec7ac8-ebd0-4219-8c85-4b4fa9beb58b/inputs/7a32ceea-9c8a-4901-80d9-e46f0c086919/HCC1143_BL.bam -> /scratch/rvalls/HCC1143_ds/HCC1143_BL.bam: Invalid cross-device link
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
        at java.nio.file.Files.createLink(Files.java:1086)
        at io.dockstore.common.FileProvisioning.provisionInputFile(FileProvisioning.java:246)
        at io.github.collaboratory.LauncherCWL.copyIndividualFile(LauncherCWL.java:669)
        at io.github.collaboratory.LauncherCWL.doProcessFile(LauncherCWL.java:635)
        at io.github.collaboratory.LauncherCWL.pullFilesHelper(LauncherCWL.java:606)
        at io.github.collaboratory.LauncherCWL.pullFiles(LauncherCWL.java:562)
        at io.github.collaboratory.LauncherCWL.run(LauncherCWL.java:184)
        at io.dockstore.client.cli.nested.AbstractEntryClient.launchCwl(AbstractEntryClient.java:897)
        at io.dockstore.client.cli.nested.AbstractEntryClient.checkEntryFile(AbstractEntryClient.java:748)
        at io.dockstore.client.cli.nested.AbstractEntryClient.launch(AbstractEntryClient.java:821)
        at io.dockstore.client.cli.nested.AbstractEntryClient.processEntryCommands(AbstractEntryClient.java:218)
        at io.dockstore.client.cli.Client.run(Client.java:698)
        at io.dockstore.client.cli.Client.main(Client.java:759)
denis-yuen commented 7 years ago

@brainstorm Yup, we are using hard links, but we fallback to copying if we cannot.

This message should not have been ERROR level and we've changed it to INFO level so it looks less alarming https://github.com/ga4gh/dockstore/blob/develop/dockstore-launcher/src/main/java/io/dockstore/common/FileProvisioning.java#L248

brainstorm commented 7 years ago

Is this change (in the .jar) deployed in the CLI tool one downloads from dockstore onboarding site? Still seeing this as ERROR :_S

denis-yuen commented 7 years ago

No, it's in develop, so not until the next release unfortunately. We should also probably suppress the stack trace to be less alarming.