search

broadinstitute / cromwell

Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments
http://cromwell.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
1k stars 359 forks source link

mkfifo: cannot create fifo issue in a Docker container on WSL2 #7002

Open ibrahimkurt opened 1 year ago

ibrahimkurt commented 1 year ago

Command: sudo java -jar cromwell-84.jar run ngs-ubuntu-20-04/iletisim/warp/pipelines/broad/dna_seq/germline/single_sample/exome/local_newGCP_ExomeGermlineSingleSample_deneme6_bcftools.wdl -i ngs-ubuntu-20-04/iletisim/json/S736Nr1.json -o ngs-ubuntu-20-04/iletisim/json/options2.json

Platform: Ubuntu 20.04 via WSL2 on Windows 10 pro

Java: openjdk 11.0.17 2022-10-18 OpenJDK Runtime Environment (build 11.0.17+8-post-Ubuntu-1ubuntu220.04) OpenJDK 64-Bit Server VM (build 11.0.17+8-post-Ubuntu-1ubuntu220.04, mixed mode, sharing)

Docker: Docker version 20.10.22, build 3a2c30b Docker desktop v.4.16.3

Inputs & Options JSON: Please find attached in the zip. forgithub.zip

Workflow & Error: I am running the main workflow (WDL attached) named "local_newGCP_ExomeGermlineSingleSample_deneme6_bcftools.wdl". It first successfully finishes the first task from "BamProcessing.wdl" (WDL attached) specifically running the task "GenerateSubsettedContaminationResources" via docker: "us.gcr.io/broad-gotc-prod/bedtools:2.27.1". In the next step it starts the next task from "paired-fastq-to-unmapped-bam.wdl" (WDL attached) via docker: "broadinstitute/gatk:latest". Inspection of the container's log during this step shown below:

2023-02-05 12:55:43 mkfifo: cannot create fifo '/cromwell-executions/ExomeGermlineSingleSample/9053ae04-ca8c-4d23-848d-7a04313af725/call-ConvertPairedFastQsToUnmappedBamWf/ConvertPairedFastQsToUnmappedBamWf/c19284af-355b-4a83-bc7d-5fed437ea8e7/call-PairedFastQsToUnmappedBAM/tmp.8cbe1f4a/out.1': Operation not supported
2023-02-05 12:55:43 mkfifo: cannot create fifo '/cromwell-executions/ExomeGermlineSingleSample/9053ae04-ca8c-4d23-848d-7a04313af725/call-ConvertPairedFastQsToUnmappedBamWf/ConvertPairedFastQsToUnmappedBamWf/c19284af-355b-4a83-bc7d-5fed437ea8e7/call-PairedFastQsToUnmappedBAM/tmp.8cbe1f4a/err.1': Operation not supported
2023-02-05 12:55:43 /cromwell-executions/ExomeGermlineSingleSample/9053ae04-ca8c-4d23-848d-7a04313af725/call-ConvertPairedFastQsToUnmappedBamWf/ConvertPairedFastQsToUnmappedBamWf/c19284af-355b-4a83-bc7d-5fed437ea8e7/call-PairedFastQsToUnmappedBAM/execution/script: line 17: /cromwell-executions/ExomeGermlineSingleSample/9053ae04-ca8c-4d23-848d-7a04313af725/call-ConvertPairedFastQsToUnmappedBamWf/ConvertPairedFastQsToUnmappedBamWf/c19284af-355b-4a83-bc7d-5fed437ea8e7/call-PairedFastQsToUnmappedBAM/tmp.8cbe1f4a/out.1: No such file or directory
2023-02-05 12:55:43 /cromwell-executions/ExomeGermlineSingleSample/9053ae04-ca8c-4d23-848d-7a04313af725/call-ConvertPairedFastQsToUnmappedBamWf/ConvertPairedFastQsToUnmappedBamWf/c19284af-355b-4a83-bc7d-5fed437ea8e7/call-PairedFastQsToUnmappedBAM/execution/script: line 18: /cromwell-executions/ExomeGermlineSingleSample/9053ae04-ca8c-4d23-848d-7a04313af725/call-ConvertPairedFastQsToUnmappedBamWf/ConvertPairedFastQsToUnmappedBamWf/c19284af-355b-4a83-bc7d-5fed437ea8e7/call-PairedFastQsToUnmappedBAM/tmp.8cbe1f4a/err.1: No such file or directory

Any help to overcome this problem would be GREATLY appreciated!

jkmatila commented 1 year ago

Is your cromwell-executions directory on a file system that does not support FIFOs (named pipes)? Windows NTFS file systems mounted to WSL2 do not support FIFOs. For example:

$ mkfifo /mnt/c/Temp/testfifo
mkfifo: cannot create fifo '/mnt/c/Temp/testfifo': Operation not supported

While an ext4 file system does:

$ mkfifo /tmp/testfifo
$ file /tmp/testfifo
/tmp/testfifo: fifo (named pipe)

If this is the problem, then the solution would be to place the cromwell-executions directory on a file system where FIFOs are supported, if running workflows that need them.