Problems with logs and joint VCF file generation in latest dev build

Hello,

After upgrading to the latest development version, the logs and joint VCF file generation don't seem to work properly anymore. Debug messages don't get printed anymore (neither on stdout, nor in the log file), and the bcbio-nextgen-debug.log file is pretty much identical with bcbio-nextgen.log. The only difference is the resource requests messages which appear in the debug log:

[2018-08-02T10:10Z] Resource requests: bwa, sambamba, samtools; memory: 3.00, 3.00, 3.00; cores: 16, 16, 16
[2018-08-02T10:10Z] Configuring 2 jobs to run, using 16 cores each with 48.1g of memory reserved for each job

[2018-08-02T10:10Z] Resource requests: gatk, gatk-haplotype, picard; memory: 3.50, 3.00, 3.00; cores: 1, 16, 16
[2018-08-02T10:10Z] Configuring 32 jobs to run, using 1 cores each with 3.50g of memory reserved for each job

[2018-08-02T10:18Z] Resource requests: bcbio_variation, fastqc, gatk, gatk-vqsr, gemini, kraken, preseq, qsignature, sambamba, samtools; memory: 3.00, 3.00, 3.50, 3.00, 3.00, 3.00, 3.00, 3.00, 3.00, 3.00; cores: 16, 16, 1, 16, 16, 16, 16, 16, 16, 16
[2018-08-02T10:18Z] Configuring 2 jobs to run, using 16 cores each with 56.1g of memory reserved for each job

The multi-sample <batch>-gatk-haplotype-joint-annotated.vcf.gz did not get generated, even though the sample-specific VCF files are where they should be.

Furthermore, bcbio-nextgen-commands.log is completely empty.

To test all of this, I've run a simple variant calling job that worked flawlessly a few days ago, before upgrading Bcbio-nextgen.

Thanks for the report. This is definitely unexpected as we haven't made any changes to logging or anything otherwise extensive that would make everything stop working. What version were you upgrading from? Are you running this in a clean directory or in a pre-existing work directory? Is it possible it's just skipping all the work because the files already exist?

It's a fresh Bcbio install (one day old), upgraded to development. I've had the problem with the log files with a couple of analyses, and they were both done from scratch, in empty or newly-created directories.

This is how I did the installation:

wget https://raw.github.com/chapmanb/bcbio-nextgen/master/scripts/bcbio_nextgen_install.py
python bcbio_nextgen_install.py <bcbio_dir> --tooldir=<bcbio_dir>/tools \
  --genomes sacCer3 --genomes phix --datatarget variation --datatarget rnaseq \
  --datatarget smallrna --datatarget vep --datatarget kraken \
  --aligners bwa --aligners bowtie2 --isolate --cores 32

export PATH=<bcbio_dir>/tools/bin:<bcbio_dir>/anaconda/bin:$PATH

bcbio_nextgen.py upgrade -u development

I tried upgrading to the latest development version again today but it didn't make any difference.

Thanks for following up. I am stumped and now sure what to suggest. What version did you have previously that worked well on your system? Do you get no logging under specific run environments: does it work when running on a single machine and only fail to log on a distributed setup? I'm not exactly sure how best to diagnose this remotely but with more details can keep trying to guess.

The latest version that worked for me was mentioned in this post.

Unfortunately, I don't have that available anymore, because I decided to do a fresh install (after a recent fix) and I deleted all the old installs I had lying around.

I'll make sure to properly test any new install before deleting older (functioning) ones in the future.

Apologies, I'm not sure what to make of this. That looks like a pretty small update distance between two recent development packages. We haven't overhauled any of the logging in that time so I'm not sure why it disappeared on your runs. A couple of questions to help debug:

How are you running bcbio? Single machine? On a cluster?
Did the run environment change between the working and problem runs?
Could you provide the configuration input you're using?

Apologies, but I don't know why you'd lose both logging and joint calling so hoping for some baseline or ideas to reproduce what you're seeing. Thanks again for helping to debug.

I've reinstalled, this time without upgrading to development.

bcbio_conda list | grep bcbio-nextgen bcbio-nextgen 1.1.0 py27_0 bioconda

This time I'm getting file system errors:

[2018-08-03T15:34Z] Timing: alignment
[2018-08-03T15:34Z] multiprocessing: process_alignment
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Traceback (most recent call last):
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/tools/bin/bcbio_nextgen.py", line 242, in <module>
    main(**kwargs)
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/tools/bin/bcbio_nextgen.py", line 47, in main
    run_main(**kwargs)
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 45, in run_main
    fc_dir, run_info_yaml)
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 89, in _run_toplevel
    for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 131, in variant2pipeline
    samples = run_parallel("process_alignment", samples)
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
    return run_multicore(fn, items, config, parallel=parallel)
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
    for data in joblib.Parallel(parallel["num_jobs"], batch_size=1)(joblib.delayed(fn)(x) for x in items):
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 994, in __call__
    self.retrieve()
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 926, in retrieve
    raise exception.unwrap(this_report)
joblib.my_exceptions.JoblibOSError: JoblibOSError

On this page, someone else mentioned getting a similar error and that it was caused by problems with their NFS storage. I'm installing Bcbio-nextgen on an NFS file system, so my issue might be related. No idea what could cause this, though, as I seem to be able to browse the NFS storage without issues and I doubt there was any NFS crash recently.

I'm installing Bcbio on an NFS storage from a VM in an OpenNebula cloud, running Debian Stretch. The working environment did, indeed, change between the working and non-working runs. I was running Ubuntu 14.04 before.

I'm using the following template:

details:
  - analysis: variant2
    genome_build: sacCer3
    algorithm:
      aligner: bwa
      mark_duplicates: true
      recalibrate: false
      realign: false
      effects: false
      tools_off: gemini
      variantcaller: gatk-haplotype
      jointcaller: gatk-haplotype-joint
      #svcaller: [manta]
      ploidy: 1
      variant_regions: ../config/variant_regions.bed

A small update on this issue: I've noticed that the bcbio-nextgen-commands.log created by the bcbio_prepare_samples.py script is not empty. Is there any possible reason for which that particular one would be properly created, while the one created by bcbio_nextgen.py would remain empty?

Hi @amizeranschi

There is not an special reason why that should be different.

I think Brad asked you : How are you running bcbio? Single machine? On a cluster?

Can you tell us how you are running the different scripts?

The common check we do is to run in a single machine (locally) the process to make sure the parallelization in a cluster is (or not) the cause of the issue.

Cheers

Hi @lpantano

Bcbio is installed on an NFS storage and I'm testing it on a single machine running Debian 9.

I'm setting up samples like this:

bcbio_prepare_samples.py --out ${bcbio_exp_name}-merged --csv ${bcbio_exp_name}.csv
bcbio_nextgen.py --separator R -w template ${bcbio_exp_name}-template.yaml ${bcbio_exp_name}-merged.csv ${bcbio_exp_path}/${bcbio_exp_name}-merged/*fastq*

and then running bcbio like this:

cd ${bcbio_exp_path}/${bcbio_exp_name}-merged/work/
bcbio_nextgen.py ../config/${bcbio_exp_name}-merged.yaml -n ${bcbio_cores}

@amizeranschi, were you able to resolve the issue? I seem to have a similar problem: https://github.com/bcbio/bcbio-nextgen/issues/2520 I am running bcbio as a docker install on a single machine and local drive (no NFS).

No, I never managed to solve this issue.

Sorry about these continued problems. We're still not sure how to best debug and fix this issue and I posted some ideas in the discussion in #2520 around running the test suite to see if that also triggers problems. If you're able to follow along and debug, any information about how to replicate on a small sample would help us try and track this down. Thanks much for all the discussion and patience with the issue.

I'm not able to replicate the original problem anymore, because I don't have access to the cluster where I've experienced these issues.

I've installed the latest stable Bcbio version locally, on a Ubuntu machine and things are working fine, as far as I could see, with the latest stable version of Bcbio (1.1.0). The bcbio-nextgen-commands.log and bcbio-nextgen-debug.log are created just as they used to.

However, after upgrading to the most recent development version, deleting the old analysis files/directories and rerunning, the joint VCF file does not get created in the final directory, although it did with the previous (stable) version. I can't tell if the problem is with the creation of the joint VCF file, or if it simply gets omitted during the stage-out process at the end. I am still using the same template as before.

Thanks for following up and glad to hear things are working as expected with the latest version in terms of logging. I am still perplexed as to what is going on here with logging.

For the joint VCF issue, everything looks fine with your template so I'm not sure why you wouldn't see an output VCF file. It's a bit tough to debug remotely from the information we have so far but I can share what it should look like with the bcbio test data. The output directory should have individual sample gVCFs and a joint called shared VCF:

upload/
├── 2018-10-31_project
│   ├── TestBatch1-gatk-haplotype-joint-annotated.vcf.gz
│   └── TestBatch1-gatk-haplotype-joint-annotated.vcf.gz.tbi
├── Test1
│   ├── Test1-gatk-haplotype.vcf.gz
│   ├── Test1-gatk-haplotype.vcf.gz.tbi
└── Test2
    ├── Test2-gatk-haplotype.vcf.gz
    ├── Test2-gatk-haplotype.vcf.gz.tbi

and in the work directory the joint calls should be in joint/gatk-haplotype-joint/your-batch-name/. If you're able to share what you see or error messages we can hopefully identify why you don't see outputs. Hope this helps and thanks again for the work debugging.

The work directory from my previous test seems to contain the joint VCF files:

ls -la /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged__devel_bcbio_01/work/joint/gatk-haplotype-joint/testingVC/
total 156
drwxrwxr-x 19 amizeranschi amizeranschi  4096 Oct 27 11:29 .
drwxrwxr-x  3 amizeranschi amizeranschi  4096 Oct 27 11:05 ..
drwxrwxr-x  6 amizeranschi amizeranschi  4096 Oct 27 11:06 chrI
drwxrwxr-x 14 amizeranschi amizeranschi  4096 Oct 27 11:07 chrII
drwxrwxr-x  8 amizeranschi amizeranschi  4096 Oct 27 11:08 chrIII
drwxrwxr-x 24 amizeranschi amizeranschi  4096 Oct 27 11:11 chrIV
drwxrwxr-x 10 amizeranschi amizeranschi  4096 Oct 27 11:12 chrIX
drwxrwxr-x 11 amizeranschi amizeranschi  4096 Oct 27 11:13 chrV
drwxrwxr-x  5 amizeranschi amizeranschi  4096 Oct 27 11:13 chrVI
drwxrwxr-x 16 amizeranschi amizeranschi  4096 Oct 27 11:15 chrVII
drwxrwxr-x 11 amizeranschi amizeranschi  4096 Oct 27 11:16 chrVIII
drwxrwxr-x 13 amizeranschi amizeranschi  4096 Oct 27 11:18 chrX
drwxrwxr-x 11 amizeranschi amizeranschi  4096 Oct 27 11:19 chrXI
drwxrwxr-x 18 amizeranschi amizeranschi  4096 Oct 27 11:21 chrXII
drwxrwxr-x 16 amizeranschi amizeranschi  4096 Oct 27 11:23 chrXIII
drwxrwxr-x 15 amizeranschi amizeranschi  4096 Oct 27 11:25 chrXIV
drwxrwxr-x 17 amizeranschi amizeranschi  4096 Oct 27 11:27 chrXV
drwxrwxr-x 17 amizeranschi amizeranschi  4096 Oct 27 11:29 chrXVI
drwxrwxr-x  2 amizeranschi amizeranschi  4096 Oct 27 11:05 inprep
-rw-rw-r--  1 amizeranschi amizeranschi  3466 Oct 27 11:29 testingVC-chrI_0_61663-fixheader.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   101 Oct 27 11:29 testingVC-chrI_0_61663-fixheader.vcf.gz.tbi
-rw-rw-r--  1 amizeranschi amizeranschi  4020 Oct 27 11:29 testingVC-joint-combined.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   777 Oct 27 11:29 testingVC-joint-combined.vcf.gz.tbi
-rw-rw-r--  1 amizeranschi amizeranschi  6359 Oct 27 11:29 testingVC-joint-files.list
-rw-rw-r--  1 amizeranschi amizeranschi  4015 Oct 27 11:29 testingVC-joint-indel-filterINDEL.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   260 Oct 27 11:29 testingVC-joint-indel-filterINDEL.vcf.gz.tbi
-rw-rw-r--  1 amizeranschi amizeranschi  3827 Oct 27 11:29 testingVC-joint-indel.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   260 Oct 27 11:29 testingVC-joint-indel.vcf.gz.tbi
-rw-rw-r--  1 amizeranschi amizeranschi  4984 Oct 27 11:29 testingVC-joint-snp-filterSNP.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   810 Oct 27 11:29 testingVC-joint-snp-filterSNP.vcf.gz.tbi
-rw-rw-r--  1 amizeranschi amizeranschi  4779 Oct 27 11:29 testingVC-joint-snp.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   809 Oct 27 11:29 testingVC-joint-snp.vcf.gz.tbi
-rw-rw-r--  1 amizeranschi amizeranschi 11021 Oct 27 11:29 testingVC-joint.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   933 Oct 27 11:29 testingVC-joint.vcf.gz.tbi

In the final directory, the sample-specific VCF files are present, but the main directory (2018-10-27_project) doesn't contain a joint VCF file. This only happens with Bcbio's development version.

After cloning the Bcbio-nexgen repo and executing ./run_tests.sh devel, the test seems to run fine.

========================================================================= 1 passed, 81 deselected in 110.28 seconds ==========================================================================

However, this doesn't appear to run joint variant calling. The main output directory (test_automated_output/upload/2018-10-31_develtest/) contains two VCF files: TestBatch1-precalled-annotated.vcf.gz and TestBatch1-gatk-haplotype-annotated.vcf.gz. And there is only one sample-specific directory called Test1 in the upload directory.

Thanks again for helping debug and providing all the additional information. It does look like things ran through joint calling successfully so I'm not sure why this would be missing from the final output directory. If you re-run bcbio_nextgen.py in the same work directory, does it get transferred? I'm wondering if there was some failure prior to transfer; it just copies these over at the end so if it died for any reason that would prevent that.

If you want to confirm with the tests, ./run_tests.sh joint will run the joint calling integration test. It would be great to know if that successfully replicates what I'm seeing above. That would help us at least confirm bcbio does or doesn't transfer in your environment on the same set of inputs. Thanks again.

./run_tests.sh joint runs fine, the joint VCF files for both HaplotypeCaller and Strelka2 are being copied to the upload directory at the end.

Would it be possible to mention the ./run_tests.sh joint command in the documentation and, possibly, in the header of the run_tests.sh script? It could help others who run into similar issues as mine.

I tried re-running my test in the same work directory and nothing changed. The joint VCF file did not get copied. I also tried running new separate tests using HaplotypeCaller and, respectively, Strelka2 (as I saw this used in the automated tests) and the same thing happened. I don't see any errors in the logs. It seems like the joint VCF file is just not being copied in the end stage, for some reason.

I can't figure out why this only happens during my test, while the "official" Bcbio automatic test runs fine, even with the latest development version. I am using a set of FASTQ files that I created some time ago by sampling 10000 reads from some larger files. I've been using them to test the pipeline for a while and never had any trouble until now.

Finally, I ran my test again with Strelka2 using the stable version of Bcbio and, sure enough, the testingVC-strelka2-joint-annotated.vcf.gz file has been created in the final upload directory, just as it did with the HaplotypeCaller tests, when using the stable Bcbio version. Is there anything that has changed between the last stable version and the development version that could cause this?

The issue persists with the newly-released stable version, as well as the latest development version, as of today.

Running single-threaded did not make a difference.

Sorry you're still having issues and thank you for continuing to test and help debug. If the tests are running cleanly for you, there must be something in your actual run configuration that's causing bcbio to go down a different path and lead to issues. I know you've posted your template above, but I don't see any substantial differences between that and the tests that run fine for you. Is it possible to share your final configuration that doesn't work? Perhaps there is something in the batching we're missing that we could spot by looking at this. Thank you for the help trying to reproduce the problem.

Sure. This is the full config file:

details:
- algorithm:
    aligner: bwa
    effects: false
    jointcaller: strelka2-joint
    mark_duplicates: true
    ploidy: 1
    realign: false
    recalibrate: false
    svcaller:
    - manta
    tools_off: gemini
    variant_regions: ../config/variant_regions.bed
    variantcaller: strelka2
  analysis: variant2
  description: sample1
  files:
  - /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/sample1_R1.fastq.gz
  - /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/sample1_R2.fastq.gz
  genome_build: sacCer3
  metadata:
    batch: testingVC
- algorithm:
    aligner: bwa
    effects: false
    jointcaller: strelka2-joint
    mark_duplicates: true
    ploidy: 1
    realign: false
    recalibrate: false
    svcaller:
    - manta
    tools_off: gemini
    variant_regions: ../config/variant_regions.bed
    variantcaller: strelka2
  analysis: variant2
  description: sample2
  files:
  - /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/sample2_R1.fastq.gz
  - /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/sample2_R2.fastq.gz
  genome_build: sacCer3
  metadata:
    batch: testingVC
fc_name: testingVC-merged
upload:
  dir: ../final

I've had an attempt at applying the logic from Issue #2520. I've installed Docker and bcbio_vm as per documentation and I've pulled several bcbio-vc images from quay.io.

Note: I had to edit the file bcbiovm/docker/install.py as described in this older issue, in order to get bcbio_vm.py to work with a Docker image name of the form image:tag.

My tests run fine with the Docker images bcbio-vc:1.1.0-v1.1.0 and bcbio-vc:1.1.1a-82fb2a0 and the joint VCF file is created in the final directory. However, this stops happening once using the image bcbio-vc:1.1.1a-026700f. I had a look on GitHub (82fb2a0..026700f) for anything that may cause the joint VCF file to not get created but I didn't find anything.

Thanks so much for all the debugging work and help figuring out this issue. I've been struggling to reproduce and understand this for a few days but all your input finally helped me realize the issue. With the changes to do more post-variant annotation, we were implicitly assuming the output joint called VCF would have vcfanno annotations and be uploaded through that path. However, for your genome it likely doesn't get annotated so gets left out of the list of files to upload.

I've added a check for this case that I hope will resolve the issue if you run the latest development version. Please let me know if not and we can investigate further. Thank you again for all your patience and working helping debug this issue.

Thank you, this fixed it alright.

If I may, I also have a couple of last questions about this. Is bcbio_vm still being maintained and will it be in the future? I remember reading somewhere that it's going to get phased out in the future, but I'm not sure whether that was referring to bcbio_vm or something else.

If it's still being maintained, would it be possible to include the following patch? https://github.com/bcbio/bcbio-nextgen-vm/issues/51

This would help quite a bit with reproducibility, allowing one to use any of the old Docker images from quay.io. Alternatively, is there any other (better) way to achieve this?

Thanks a lot for fixing the joint VCF generation bug and for all the great work with bcbio-nextgen and bcbio_vm!

Thank you for following up. I'm glad that fixed the issue for you and thanks again for all the patience as we identified the underlying problem.

For running with Docker containers, we now recommend using the common workflow language (CWL) interface with the Cromwell runner, documented here:

https://bcbio-nextgen.readthedocs.io/en/latest/contents/cwl.html

This still uses bcbio_vm.py for setting up the run environment and launching jobs, but the actually running takes place via external runners to get improved runtime support over what we can develop as part of bcbio.

To lock down to specific tags in your workflows for reproducibility, there is a --add-container-tag for bcbio_vm.py cwl that uses a specific tag from the quay.io releases rather than latest (https://quay.io/repository/bcbio/bcbio-vc?tab=tags). This gets reflected persistently in generated CWL so you have a fully reproducible run tied to a specific instance.

CWL is still a work in progress but improving and being used more often. Hope this works for your use.

I gave this a try but I'm stuck at running bcbio_vm.py cwl because it doesn't find the reference genome files:

bcbio_vm.py cwl   --add-container-tag 1.1.1a-82fb2a0   ../config/${bcbio_exp_name}-merged.yaml
[2018-11-13T18:04Z] INFO: Using input YAML configuration: ../config/testingVC-merged.yaml
[2018-11-13T18:04Z] INFO: Checking sample YAML configuration: ../config/testingVC-merged.yaml
Traceback (most recent call last):
  File "/home/amizeranschi/install/bcbio-vm//anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/main.py", line 11, in run
    world = run_info.organize(dirs, config, run_info_yaml, is_cwl=True, integrations=integrations)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/pipeline/run_info.py", line 77, in organize
    item = add_reference_resources(item, remote_retriever)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/pipeline/run_info.py", line 174, in add_reference_resources
    _check_ref_files(data["reference"], data)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/pipeline/run_info.py", line 231, in _check_ref_files
    raise ValueError(msg + "\n".join(problems) + "\n")
ValueError: 
Problems with input reference file None
Did not find fasta reference file for genome sacCer3.
Check tool-data/*.loc files to ensure paths to reference data are correct.

I have previously saved the datadir (bcbio_vm.py --datadir=~/install/bcbio-vm saveconfig). Nevertheless, I tried adding the --datadir to the command (bcbio_vm.py --datadir=~/install/bcbio-vm cwl --add-container-tag 1.1.1a-82fb2a0 ../config/${bcbio_exp_name}-merged.yaml), but this didn't help.

However, running without CWL works fine:

bcbio_vm.py --datadir=~/install/bcbio-vm run --image quay.io/bcbio/bcbio-vc:1.1.1a-82fb2a0 -n 4 ../config/${bcbio_exp_name}-merged.yaml

The *.loc files are in ~/install/bcbio-vm/galaxy/tool-data/, but the paths inside them seem to point to /usr/local/share, for some reason (e.g. /usr/local/share/bcbio-nextgen/genomes/Scerevisiae/sacCer3/bwa/sacCer3.fa). I'm not sure why this happened. I've installed bcbio_vm with the following command:

bcbio_vm.py --datadir=~/install/bcbio-vm install --data --image quay.io/bcbio/bcbio-vc:1.1.0-v1.1.0 --tools \
  --genomes sacCer3 --aligners bwa

How can I get the correct paths inside the .loc files (without having to edit them manually)? And why isn't this issue affecting bcbio_vm.py when running without CWL?

P.S. Should I open a separate ticket for these issues?

Sorry about the issues and confusion here. The bcbio_vm.py data install is going to be confusing as it works with data both inside and outside the container, which is what leads to the disconnect here. Practically, we've tried to avoid all this by no longer depending on the *.loc files and instead specifying the location of the reference directory in a bcbio_system.yaml file that you pass to bcbio_vm.py cwl:

https://bcbio-nextgen.readthedocs.io/en/latest/contents/cwl.html#generating-cwl-for-input-to-a-tool

This approach should be much cleaner and avoid the need for any custom install tweaks. You point it at the genome directory and it finds all the inputs from that. Once you have the file in place you pass it to the CWL generation command with bcbio_vm.py cwl --systemconfig bcbio_system.yaml (and all your other arguments).

Hope this helps get CWL generation working for you. Thanks for exploring this and all the patience.

Alright, thanks for following up on this. I edited the file bcbio_system.yaml and got that to work, but now bcbio_vm.py cwl gets stuck due to another config (genome_resources__rnaseq__gene_bed). I'm using the same YAML template as previously mentioned on this thread, so I'm not sure why it complains about an rnaseq config, unless this is somehow referring to the sv_regions config that is used for structural VC. However, I don't have this enabled in the template.

I tried adding it and providing a valid BED file, but this didn't make the error go away. Here's the full output that I get:

bcbio_vm.py cwl --systemconfig ~/install/bcbio-vm/galaxy/bcbio_system.yaml --add-container-tag 1.1.1a-82fb2a0   ../config/${bcbio_exp_name}-merged.yaml
[2018-11-14T19:24Z] INFO: Using input YAML configuration: ../config/testingVC-merged.yaml
[2018-11-14T19:24Z] INFO: Checking sample YAML configuration: ../config/testingVC-merged.yaml
Traceback (most recent call last):
  File "/home/amizeranschi/install/bcbio-vm//anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/main.py", line 12, in run
    create.from_world(world, run_info_yaml, integrations=integrations, add_container_tag=args.add_container_tag)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 37, in from_world
    prep_cwl(samples, workflow_fn, out_dir, out_file, integrations, add_container_tag=add_container_tag)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 427, in prep_cwl
    for cur in workflow.generate(variables, steps, wfoutputs):
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/workflow.py", line 64, in generate
    inputs, parallel_ids, nested_inputs = _get_step_inputs(step, file_vs, std_vs, parallel_ids)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/workflow.py", line 170, in _get_step_inputs
    for orig_input in [_get_variable(x, file_vs) for x in _handle_special_inputs(step.inputs, file_vs)]:
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/workflow.py", line 289, in _get_variable
    raise ValueError("Did not find variable %s in \n%s" % (vid, pprint.pformat(variables)))
ValueError: Did not find variable genome_resources__rnaseq__gene_bed in 
[{'id': 'config__algorithm__align_split_size', 'type': ['null', 'string']},
 {'id': 'files', 'type': {'items': 'File', 'type': 'array'}},
 {'id': 'config__algorithm__trim_reads',
  'type': ['string', 'null', 'boolean']},
 {'id': 'reference__fasta__base',
  'secondaryFiles': ['.fai', '^.dict'],
  'type': 'File'},
 {'id': 'config__algorithm__expression_caller', 'type': ['null', 'string']},
 {'id': 'config__algorithm__vcfanno',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'config__algorithm__svprioritize', 'type': ['null', 'string']},
 {'id': 'config__algorithm__variantcaller',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'config__algorithm__adapters',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'config__algorithm__svcaller',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'genome_resources__variation__editing', 'type': ['null', 'string']},
 {'id': 'config__algorithm__coverage_interval', 'type': ['null', 'string']},
 {'id': 'reference__genome_context',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'genome_resources__variation__gc_profile', 'type': ['null', 'string']},
 {'id': 'rgnames__lb', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__clinvar', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__esp', 'type': ['null', 'string']},
 {'id': 'rgnames__rg', 'type': 'string'},
 {'id': 'config__algorithm__realign', 'type': ['string', 'null', 'boolean']},
 {'id': 'metadata__batch', 'type': 'string'},
 {'id': 'rgnames__lane', 'type': 'string'},
 {'id': 'genome_resources__variation__lcr', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__1000g', 'type': ['null', 'string']},
 {'id': 'config__algorithm__min_allele_fraction', 'type': 'double'},
 {'id': 'config__algorithm__nomap_split_targets', 'type': 'long'},
 {'id': 'reference__versions', 'type': ['null', 'string']},
 {'id': 'reference__bwa__indexes',
  'secondaryFiles': ['^.ann', '^.pac', '^.sa', '^.bwt'],
  'type': 'File'},
 {'id': 'config__algorithm__ploidy', 'type': 'long'},
 {'id': 'vrn_file', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__train_hapmap',
  'type': ['null', 'string']},
 {'id': 'config__algorithm__jointcaller',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'config__algorithm__bam_clean', 'type': ['string', 'null', 'boolean']},
 {'id': 'config__algorithm__nomap_split_size', 'type': 'long'},
 {'id': 'config__algorithm__validate', 'type': ['null', 'string']},
 {'id': 'config__algorithm__validate_regions', 'type': ['null', 'string']},
 {'id': 'genome_resources__rnaseq__transcriptome_index__tophat',
  'type': 'string'},
 {'id': 'config__algorithm__aligner', 'type': 'string'},
 {'id': 'rgnames__pl', 'type': 'string'},
 {'id': 'genome_build', 'type': 'string'},
 {'id': 'rgnames__pu', 'type': 'string'},
 {'id': 'genome_resources__variation__germline_het_pon',
  'type': ['null', 'string']},
 {'id': 'genome_resources__variation__exac', 'type': ['null', 'string']},
 {'id': 'config__algorithm__recalibrate',
  'type': ['string', 'null', 'boolean']},
 {'id': 'metadata__phenotype', 'type': 'string'},
 {'id': 'config__algorithm__quality_format', 'type': 'string'},
 {'id': 'config__algorithm__tools_off',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'genome_resources__variation__dbsnp', 'type': ['null', 'string']},
 {'id': 'config__algorithm__mark_duplicates',
  'type': ['string', 'null', 'boolean']},
 {'id': 'reference__twobit', 'type': 'File'},
 {'id': 'genome_resources__variation__polyx', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__encode_blacklist',
  'type': ['null', 'string']},
 {'id': 'reference__viral', 'type': ['null', 'string']},
 {'id': 'genome_resources__version', 'type': 'long'},
 {'id': 'genome_resources__variation__cosmic', 'type': ['null', 'string']},
 {'id': 'config__algorithm__ensemble', 'type': ['null', 'string']},
 {'id': 'config__algorithm__qc', 'type': {'items': 'string', 'type': 'array'}},
 {'id': 'config__algorithm__sv_regions', 'type': 'File'},
 {'id': 'analysis', 'type': 'string'},
 {'id': 'prep_samples/rgnames__sample', 'type': 'string'},
 {'id': 'config__algorithm__tools_on',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'config__algorithm__effects', 'type': ['string', 'null', 'boolean']},
 {'id': 'prep_samples/config__algorithm__variant_regions',
  'type': ['File', 'null']},
 {'id': 'config__algorithm__svvalidate', 'type': ['null', 'string']},
 {'id': 'reference__snpeff__R64-1-1_86', 'type': 'File'},
 {'id': 'genome_resources__aliases__ensembl', 'type': 'string'},
 {'id': 'config__algorithm__exclude_regions',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'reference__rtg', 'type': 'File'},
 {'id': 'genome_resources__variation__train_indels',
  'type': ['null', 'string']},
 {'id': 'genome_resources__aliases__snpeff', 'type': 'string'},
 {'id': 'config__algorithm__archive', 'type': ['null', 'string']},
 {'id': 'alignment_to_rec/alignment_rec',
  'type': {'items': {'fields': [{'name': 'resources', 'type': 'string'},
                                {'name': 'description', 'type': 'string'},
                                {'name': 'config__algorithm__align_split_size',
                                 'type': ['null', 'string']},
                                {'name': 'files',
                                 'type': {'items': 'File',
                                          'type': 'array'}},
                                {'name': 'config__algorithm__trim_reads',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'reference__fasta__base',
                                 'secondaryFiles': ['.fai', '^.dict'],
                                 'type': 'File'},
                                {'name': 'config__algorithm__adapters',
                                 'type': ['null',
                                          {'items': 'null',
                                           'type': 'array'}]},
                                {'name': 'rgnames__lb',
                                 'type': ['null', 'string']},
                                {'name': 'rgnames__rg', 'type': 'string'},
                                {'name': 'rgnames__lane',
                                 'type': 'string'},
                                {'name': 'reference__bwa__indexes',
                                 'secondaryFiles': ['^.ann',
                                                    '^.pac',
                                                    '^.sa',
                                                    '^.bwt'],
                                 'type': 'File'},
                                {'name': 'config__algorithm__bam_clean',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'config__algorithm__aligner',
                                 'type': 'string'},
                                {'name': 'rgnames__pl', 'type': 'string'},
                                {'name': 'rgnames__pu', 'type': 'string'},
                                {'name': 'config__algorithm__mark_duplicates',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'analysis', 'type': 'string'},
                                {'name': 'rgnames__sample',
                                 'type': 'string'},
                                {'name': 'config__algorithm__variant_regions',
                                 'type': 'File'}],
                     'name': 'alignment_rec',
                     'type': 'record'},
           'type': 'array'}},
 {'id': 'alignment/align_bam',
  'secondaryFiles': ['.bai'],
  'type': ['File', 'null']},
 {'id': 'alignment/hla__fastq',
  'type': ['null', {'items': 'File', 'type': 'array'}]},
 {'id': 'alignment/work_bam_plus__disc',
  'secondaryFiles': ['.bai'],
  'type': ['File', 'null']},
 {'id': 'alignment/work_bam_plus__sr',
  'secondaryFiles': ['.bai'],
  'type': ['File', 'null']},
 {'id': 'prep_samples_to_rec/prep_samples_rec',
  'type': {'items': {'fields': [{'name': 'resources', 'type': 'string'},
                                {'name': 'description', 'type': 'string'},
                                {'name': 'reference__fasta__base',
                                 'secondaryFiles': ['.fai', '^.dict'],
                                 'type': 'File'},
                                {'name': 'rgnames__sample',
                                 'type': 'string'},
                                {'name': 'config__algorithm__variant_regions',
                                 'type': 'File'}],
                     'name': 'prep_samples_rec',
                     'type': 'record'},
           'type': 'array'}},
 {'id': 'prep_samples/config__algorithm__variant_regions_merged',
  'type': ['File', 'null']},
 {'id': 'prep_samples/config__algorithm__variant_regions_orig',
  'type': ['File', 'null']},
 {'id': 'prep_samples/config__algorithm__coverage', 'type': ['File', 'null']},
 {'id': 'prep_samples/config__algorithm__coverage_merged',
  'type': ['File', 'null']},
 {'id': 'prep_samples/config__algorithm__coverage_orig',
  'type': ['File', 'null']},
 {'id': 'prep_samples/config__algorithm__seq2c_bed_ready',
  'type': ['File', 'null']}]

Thanks for testing this and apologies about the additional problems. What we're running into here is that bcbio isn't finding transcript files for your sacCer3 genome install, so doesn't have the BED files derived from that which is uses for annotations. In CWL it will fail up front with these type of issues. Are you using the standard bcbio sacCer3 configuration:

https://github.com/bcbio/bcbio-nextgen/blob/master/config/genomes/sacCer3-resources.yaml

or something custom? Do you have the transcript files available in your install? I'd definitely like to get the yeast genome install fixed for you but haven't worked with it in a while so am not entirely sure what might be missing. If you can provide more details I'm happy to help dig into it.

Alright, thanks for looking into this. I'm using the standard configuration. I have installed bcbio-nextgen as such:

python bcbio_nextgen_install.py ${bcbio_path} --tooldir=${bcbio_path}/tools \
  --genomes ${bcbio_genome} --datatarget variation --datatarget rnaseq \
  --datatarget smallrna --datatarget vep --datatarget kraken \
  --aligners bwa --aligners bowtie2 --isolate --cores ${bcbio_cores}

And this is reflected in the file ${bcbio_path}/config/install-params.yaml:

aligners:
- bwa
- bowtie2
- rtg
- twobit
datatarget:
- variation
- rnaseq
- smallrna
- vep
- kraken
genomes:
- sacCer3
isolate: true
tooldir: /home/amizeranschi/bcbio-stuff/bcbio/tools
toolplus: []

However, there is no rnaseq directory in /home/amizeranschi/bcbio-stuff/bcbio/genomes/Scerevisiae/sacCer3/.

ls /home/amizeranschi/bcbio-stuff/bcbio/genomes/Scerevisiae/sacCer3/
bowtie2  bwa  rtg  seq  seq_prep  snpeff  vep

The sacCer3.2bit file is also missing from /home/amizeranschi/bcbio-stuff/bcbio/genomes/Scerevisiae/sacCer3/seq/, although it is mentioned as installable in ${bcbio_path}/config/install-params.yaml:

ls /home/amizeranschi/bcbio-stuff/bcbio/genomes/Scerevisiae/sacCer3/seq/
chromFa.tar.gz  sacCer3.dict  sacCer3.fa  sacCer3.fa.fai  sacCer3-resources.yaml

The contents of the sacCer3-resources.yaml is identical to the one that you've previously linked on GitHub:

version: 16

aliases:
  ensembl: saccharomyces_cerevisiae_vep_94_R64-1-1
  snpeff: R64-1-1.86
rnaseq:
  transcripts: ../rnaseq/ref-transcripts.gtf
  transcripts_mask: ../rnaseq/ref-transcripts-mask.gtf
  transcriptome_index:
    tophat: ../rnaseq/tophat/sacCer3_transcriptome.ver
  dexseq: ../rnaseq/ref-transcripts.dexseq.gff3
  refflat: ../rnaseq/ref-transcripts.refFlat
  rRNA_fa: ../rnaseq/rRNA.fa

The same problem happens with bcbio-vm. I've installed it this way:

bcbio_vm.py --datadir=~/install/bcbio-vm install --data --image quay.io/bcbio/bcbio-vc:1.1.0-v1.1.0 --tools \
  --genomes sacCer3 --aligners bwa

bcbio_vm.py --datadir=~/install/bcbio-vm saveconfig

There are no rnaseq files in /home/amizeranschi/install/bcbio-vm/genomes/Scerevisiae/sacCer3/:

ls /home/amizeranschi/install/bcbio-vm/genomes/Scerevisiae/sacCer3/
bwa  rtg  rtg--sacCer3.sdf-wf.tar.gz  seq  seq_prep  snpeff  snpeff--R64-1-1.86-wf.tar.gz  ucsc-wf.tar.gz  vep

I've installed the hg19 genome to see if the same issues persist, for any reason, and they didn't. The rnaseq directory was created and the hg19.2bit file was created as well.

It looks like whatever is happening is specific to the sacCer3 genome, although I haven't modified the sacCer3-resources.yaml file.

Thanks for checking on this and apologies about sending you down the wrong path. I hadn't realized the sacCer3 genome wasn't up to date with transcripts. I just pushed fixes for this so if you want to do a clean install of the genome it should now have an rnaseq directory and work cleanly:

rm -rf /path/to/bcbio/genomes/Scerevisiae/sacCer3
bcbio_nextgen.py upgrade --data --cwl --genomes sacCer3

We no longer use the 2bit files since GATK has moved away from requiring those, but it should have everything else you need to run the CWL. Please let me know if you run into any issues with the install or run after this. Thanks again for the patience debugging.

Thanks once more for your help. The rnaseq directory has now been created.

One question: is there a way to install the stand-alone bcbio_vm and genome data without having to download the latest Docker image (as it would just waste space)? I have the image already downloaded (quay.io/bcbio/bcbio-vc:1.1.1a-82fb2a0). However, when I try the following:

bcbio_vm.py --datadir=~/install/bcbio-vm install --data --tools --image quay.io/bcbio/bcbio-vc:1.1.1a-82fb2a0 \
  --genomes sacCer3 --aligners bwa

It says it can't find the image locally, as described here: https://github.com/bcbio/bcbio-nextgen-vm/issues/51.

Regarding the previous issues with using CWL, bcbiovm.py now complains about a missing .2bit file:

bcbio_vm.py cwl --systemconfig ~/install/bcbio-vm/galaxy/bcbio_system.yaml --add-container-tag 1.1.1a-82fb2a0   ../config/${bcbio_exp_name}-merged.yaml
[2018-11-20T17:32Z] INFO: Using input YAML configuration: ../config/testingVC-merged.yaml
[2018-11-20T17:32Z] INFO: Checking sample YAML configuration: ../config/testingVC-merged.yaml
Traceback (most recent call last):
  File "/home/amizeranschi/install/bcbio-vm/anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/main.py", line 12, in run
    create.from_world(world, run_info_yaml, integrations=integrations, add_container_tag=args.add_container_tag)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 37, in from_world
    prep_cwl(samples, workflow_fn, out_dir, out_file, integrations, add_container_tag=add_container_tag)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 419, in prep_cwl
    variables, keyvals = _flatten_samples(samples, out_file, get_retriever)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 480, in _flatten_samples
    data["reference"] = _indexes_to_secondary_files(data["reference"], data["genome_build"])
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 515, in _indexes_to_secondary_files
    raise ValueError("Did not find indexes for %s: %s" % (refname, val))
ValueError: Did not find indexes for twobit: {'indexes': []}

My guess is that some checks are still assuming that the .2bit file is in its right spot. And since this file still gets downloaded for e.g. the hg19 genome, this issue doesn't appear when using it.

I've manually downloaded the sacCer3.2bit file into a ucsc directory in the genome location:

mkdir /home/amizeranschi/install/bcbio-vm/genomes/Scerevisiae/sacCer3/ucsc
cd /home/amizeranschi/install/bcbio-vm/genomes/Scerevisiae/sacCer3/ucsc
wget 'https://hgdownload-test.gi.ucsc.edu/goldenPath/sacCer3/bigZips/sacCer3.2bit'

and this got rid of the previous error, but now I get a different one (Did not find variable genome_resources__aliases__human):

bcbio_vm.py cwl --systemconfig ~/install/bcbio-vm/galaxy/bcbio_system.yaml --add-container-tag 1.1.1a-82fb2a0   ../config/${bcbio_exp_name}-merged.yaml
[2018-11-20T17:43Z] INFO: Using input YAML configuration: ../config/testingVC-merged.yaml
[2018-11-20T17:43Z] INFO: Checking sample YAML configuration: ../config/testingVC-merged.yaml
Preparing CWL input tarball: /home/amizeranschi/install/bcbio-vm/genomes/Scerevisiae/sacCer3/rtg--sacCer3.sdf-wf.tar.gz
Preparing CWL input tarball: /home/amizeranschi/install/bcbio-vm/genomes/Scerevisiae/sacCer3/snpeff--R64-1-1.86-wf.tar.gz
Traceback (most recent call last):
  File "/home/amizeranschi/install/bcbio-vm/anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/main.py", line 12, in run
    create.from_world(world, run_info_yaml, integrations=integrations, add_container_tag=args.add_container_tag)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 37, in from_world
    prep_cwl(samples, workflow_fn, out_dir, out_file, integrations, add_container_tag=add_container_tag)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 427, in prep_cwl
    for cur in workflow.generate(variables, steps, wfoutputs):
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/workflow.py", line 64, in generate
    inputs, parallel_ids, nested_inputs = _get_step_inputs(step, file_vs, std_vs, parallel_ids)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/workflow.py", line 170, in _get_step_inputs
    for orig_input in [_get_variable(x, file_vs) for x in _handle_special_inputs(step.inputs, file_vs)]:
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/workflow.py", line 289, in _get_variable
    raise ValueError("Did not find variable %s in \n%s" % (vid, pprint.pformat(variables)))
ValueError: Did not find variable genome_resources__aliases__human in 
[{'id': 'config__algorithm__align_split_size', 'type': ['null', 'string']},
 {'id': 'files', 'type': {'items': 'File', 'type': 'array'}},
 {'id': 'config__algorithm__trim_reads',
  'type': ['string', 'null', 'boolean']},
 {'id': 'reference__fasta__base',
  'secondaryFiles': ['.fai', '^.dict'],
  'type': 'File'},
 {'id': 'config__algorithm__expression_caller', 'type': ['null', 'string']},
 {'id': 'config__algorithm__vcfanno',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'config__algorithm__svprioritize', 'type': ['null', 'string']},
 {'id': 'reference__snpeff', 'type': 'File'},
 {'id': 'config__algorithm__variantcaller',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'config__algorithm__adapters',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'config__algorithm__svcaller',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'genome_resources__rnaseq__transcripts_mask', 'type': 'File'},
 {'id': 'genome_resources__variation__editing', 'type': ['null', 'string']},
 {'id': 'postprocess_alignment/config__algorithm__coverage_interval',
  'type': ['string', 'null']},
 {'id': 'genome_resources__rnaseq__gene_bed', 'type': 'File'},
 {'id': 'reference__genome_context',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'genome_resources__variation__gc_profile', 'type': ['null', 'string']},
 {'id': 'rgnames__lb', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__clinvar', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__esp', 'type': ['null', 'string']},
 {'id': 'rgnames__rg', 'type': 'string'},
 {'id': 'config__algorithm__realign', 'type': ['string', 'null', 'boolean']},
 {'id': 'metadata__batch', 'type': 'string'},
 {'id': 'rgnames__lane', 'type': 'string'},
 {'id': 'genome_resources__variation__lcr', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__1000g', 'type': ['null', 'string']},
 {'id': 'config__algorithm__min_allele_fraction', 'type': 'double'},
 {'id': 'config__algorithm__nomap_split_targets', 'type': 'long'},
 {'id': 'reference__versions', 'type': 'File'},
 {'id': 'reference__bwa__indexes',
  'secondaryFiles': ['^.ann', '^.pac', '^.sa', '^.bwt'],
  'type': 'File'},
 {'id': 'config__algorithm__ploidy', 'type': 'long'},
 {'id': 'vrn_file', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__train_hapmap',
  'type': ['null', 'string']},
 {'id': 'config__algorithm__jointcaller',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'config__algorithm__bam_clean', 'type': ['string', 'null', 'boolean']},
 {'id': 'config__algorithm__nomap_split_size', 'type': 'long'},
 {'id': 'config__algorithm__validate', 'type': ['null', 'string']},
 {'id': 'genome_resources__rnaseq__dexseq', 'type': 'File'},
 {'id': 'config__algorithm__validate_regions', 'type': ['null', 'string']},
 {'id': 'genome_resources__rnaseq__transcriptome_index__tophat',
  'type': 'string'},
 {'id': 'config__algorithm__aligner', 'type': 'string'},
 {'id': 'rgnames__pl', 'type': 'string'},
 {'id': 'genome_build', 'type': 'string'},
 {'id': 'rgnames__pu', 'type': 'string'},
 {'id': 'genome_resources__variation__germline_het_pon',
  'type': ['null', 'string']},
 {'id': 'genome_resources__variation__exac', 'type': ['null', 'string']},
 {'id': 'config__algorithm__recalibrate',
  'type': ['string', 'null', 'boolean']},
 {'id': 'metadata__phenotype', 'type': 'string'},
 {'id': 'genome_resources__rnaseq__transcripts',
  'secondaryFiles': ['.db'],
  'type': 'File'},
 {'id': 'config__algorithm__quality_format', 'type': 'string'},
 {'id': 'config__algorithm__tools_off',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'genome_resources__variation__dbsnp', 'type': ['null', 'string']},
 {'id': 'config__algorithm__mark_duplicates',
  'type': ['string', 'null', 'boolean']},
 {'id': 'reference__twobit', 'type': 'File'},
 {'id': 'genome_resources__variation__polyx', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__encode_blacklist',
  'type': ['null', 'string']},
 {'id': 'reference__viral', 'type': ['null', 'string']},
 {'id': 'genome_resources__version', 'type': 'long'},
 {'id': 'genome_resources__variation__cosmic', 'type': ['null', 'string']},
 {'id': 'genome_resources__rnaseq__refflat', 'type': 'File'},
 {'id': 'config__algorithm__ensemble', 'type': ['null', 'string']},
 {'id': 'config__algorithm__qc', 'type': {'items': 'string', 'type': 'array'}},
 {'id': 'config__algorithm__sv_regions', 'type': 'File'},
 {'id': 'analysis', 'type': 'string'},
 {'id': 'prep_samples/rgnames__sample', 'type': 'string'},
 {'id': 'config__algorithm__tools_on',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'config__algorithm__effects', 'type': ['string', 'null', 'boolean']},
 {'id': 'postprocess_alignment/config__algorithm__variant_regions',
  'type': ['File', 'null']},
 {'id': 'config__algorithm__svvalidate', 'type': ['null', 'string']},
 {'id': 'genome_resources__aliases__ensembl', 'type': 'string'},
 {'id': 'config__algorithm__exclude_regions',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'reference__rtg', 'type': 'File'},
 {'id': 'genome_resources__variation__train_indels',
  'type': ['null', 'string']},
 {'id': 'genome_resources__aliases__snpeff', 'type': 'string'},
 {'id': 'config__algorithm__archive', 'type': ['null', 'string']},
 {'id': 'alignment_to_rec/alignment_rec',
  'type': {'items': {'fields': [{'name': 'resources', 'type': 'string'},
                                {'name': 'description', 'type': 'string'},
                                {'name': 'config__algorithm__align_split_size',
                                 'type': ['null', 'string']},
                                {'name': 'files',
                                 'type': {'items': 'File',
                                          'type': 'array'}},
                                {'name': 'config__algorithm__trim_reads',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'reference__fasta__base',
                                 'secondaryFiles': ['.fai', '^.dict'],
                                 'type': 'File'},
                                {'name': 'config__algorithm__adapters',
                                 'type': ['null',
                                          {'items': 'null',
                                           'type': 'array'}]},
                                {'name': 'rgnames__lb',
                                 'type': ['null', 'string']},
                                {'name': 'rgnames__rg', 'type': 'string'},
                                {'name': 'rgnames__lane',
                                 'type': 'string'},
                                {'name': 'reference__bwa__indexes',
                                 'secondaryFiles': ['^.ann',
                                                    '^.pac',
                                                    '^.sa',
                                                    '^.bwt'],
                                 'type': 'File'},
                                {'name': 'config__algorithm__bam_clean',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'config__algorithm__aligner',
                                 'type': 'string'},
                                {'name': 'rgnames__pl', 'type': 'string'},
                                {'name': 'rgnames__pu', 'type': 'string'},
                                {'name': 'config__algorithm__mark_duplicates',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'analysis', 'type': 'string'},
                                {'name': 'rgnames__sample',
                                 'type': 'string'},
                                {'name': 'config__algorithm__variant_regions',
                                 'type': 'File'}],
                     'name': 'alignment_rec',
                     'type': 'record'},
           'type': 'array'}},
 {'id': 'postprocess_alignment/align_bam',
  'secondaryFiles': ['.bai'],
  'type': ['File', 'null']},
 {'id': 'alignment/hla__fastq',
  'type': ['null', {'items': 'File', 'type': 'array'}]},
 {'id': 'alignment/work_bam_plus__disc',
  'secondaryFiles': ['.bai'],
  'type': ['File', 'null']},
 {'id': 'alignment/work_bam_plus__sr',
  'secondaryFiles': ['.bai'],
  'type': ['File', 'null']},
 {'id': 'prep_samples_to_rec/prep_samples_rec',
  'type': {'items': {'fields': [{'name': 'resources', 'type': 'string'},
                                {'name': 'description', 'type': 'string'},
                                {'name': 'reference__fasta__base',
                                 'secondaryFiles': ['.fai', '^.dict'],
                                 'type': 'File'},
                                {'name': 'rgnames__sample',
                                 'type': 'string'},
                                {'name': 'config__algorithm__variant_regions',
                                 'type': 'File'}],
                     'name': 'prep_samples_rec',
                     'type': 'record'},
           'type': 'array'}},
 {'id': 'postprocess_alignment/config__algorithm__variant_regions_merged',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/config__algorithm__variant_regions_orig',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/config__algorithm__coverage',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/config__algorithm__coverage_merged',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/config__algorithm__coverage_orig',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/config__algorithm__seq2c_bed_ready',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment_to_rec/postprocess_alignment_rec',
  'type': {'items': {'fields': [{'name': 'resources', 'type': 'string'},
                                {'name': 'description', 'type': 'string'},
                                {'name': 'reference__fasta__base',
                                 'secondaryFiles': ['.fai', '^.dict'],
                                 'type': 'File'},
                                {'name': 'config__algorithm__coverage_interval',
                                 'type': ['null', 'string']},
                                {'name': 'genome_resources__rnaseq__gene_bed',
                                 'type': 'File'},
                                {'name': 'genome_resources__variation__lcr',
                                 'type': ['null', 'string']},
                                {'name': 'config__algorithm__recalibrate',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'genome_resources__variation__dbsnp',
                                 'type': ['null', 'string']},
                                {'name': 'genome_resources__variation__polyx',
                                 'type': ['null', 'string']},
                                {'name': 'genome_resources__variation__encode_blacklist',
                                 'type': ['null', 'string']},
                                {'name': 'config__algorithm__tools_on',
                                 'type': ['null',
                                          {'items': 'null',
                                           'type': 'array'}]},
                                {'name': 'config__algorithm__variant_regions',
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__exclude_regions',
                                 'type': ['null',
                                          {'items': 'null',
                                           'type': 'array'}]},
                                {'name': 'align_bam',
                                 'secondaryFiles': ['.bai'],
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__variant_regions_merged',
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__variant_regions_orig',
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__coverage',
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__coverage_merged',
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__coverage_orig',
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__seq2c_bed_ready',
                                 'type': ['File', 'null']}],
                     'name': 'postprocess_alignment_rec',
                     'type': 'record'},
           'type': 'array'}},
 {'id': 'postprocess_alignment/regions__callable', 'type': ['File', 'null']},
 {'id': 'postprocess_alignment/regions__sample_callable',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/regions__nblock', 'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__samtools__stats',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__samtools__idxstats',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__variant_regions__regions',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__variant_regions__dist',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__sv_regions__regions',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__sv_regions__dist',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__coverage__regions',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__coverage__dist',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__coverage__thresholds',
  'type': ['File', 'null']},
 {'id': 'combine_sample_regions/config__algorithm__callable_regions',
  'type': 'File'},
 {'id': 'combine_sample_regions/config__algorithm__non_callable_regions',
  'type': 'File'},
 {'id': 'combine_sample_regions/config__algorithm__callable_count',
  'type': 'int'}]

Thanks so much for all the work testing this and apologies about the stumbling blocks. We need to have better non-human tests to avoid having folks trip over these types of issues. If you update your bcbio and bcbio-vm packages with:

bcbiovm_conda install -c conda-forge -c bioconda bcbio-nextgen bcbio-nextgen-vm

It should now hopefully avoid both the 2bit and alias issues you ran into when generating the CWL. Fingers crossed this will get it all of the way there.

For docker management, we've been trying to move away from having bcbio_vm.py itself as the manager for containers. CWL runners like Cromwell will download images as needed, so we don't need this explicit up front download. Would skipping the --tools argument to bcbio_vm.py and letting Cromwell take care of image download work for you?

Thanks again for all the patience working through these issues.

Things are definitely moving forward and I got bcbio_vm.py cwl to run. Thanks for your help. However, there was another error popping up at the next stage. This is how I'm setting up the CWL run (am I using the option --add-container-tag correctly?):

bcbio_vm.py cwl --add-container-tag 1.1.1a-82fb2a0 --systemconfig ~/install/bcbio-vm/galaxy/bcbio_system.yaml  ../config/${bcbio_exp_name}-merged.yaml
bcbio_vm.py cwlrun cromwell ${bcbio_exp_name}-merged-workflow

And below is the outcome:

[2018-11-21 17:16:50,08] [info] Slf4jLogger started
[2018-11-21 17:16:50,14] [info] Pre Processing Workflow...
[2018-11-21 17:16:50,30] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/main-testingVC-merged.cwl
[2018-11-21 17:16:51,98] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/alignment_to_rec.cwl
[2018-11-21 17:16:54,45] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/wf-alignment.cwl
[2018-11-21 17:16:55,53] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/prep_align_inputs.cwl
[2018-11-21 17:16:57,50] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/process_alignment.cwl
[2018-11-21 17:17:01,57] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/merge_split_alignments.cwl
[2018-11-21 17:17:03,52] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/prep_samples_to_rec.cwl
[2018-11-21 17:17:05,35] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/prep_samples.cwl
[2018-11-21 17:17:07,20] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/postprocess_alignment_to_rec.cwl
[2018-11-21 17:17:09,15] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/postprocess_alignment.cwl
[2018-11-21 17:17:11,43] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/combine_sample_regions.cwl
[2018-11-21 17:17:13,75] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/batch_for_variantcall.cwl
[2018-11-21 17:17:15,86] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/wf-variantcall.cwl
[2018-11-21 17:17:17,01] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/get_parallel_regions.cwl
[2018-11-21 17:17:18,98] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/variantcall_batch_region.cwl
[2018-11-21 17:17:21,54] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/concat_batch_variantcalls.cwl
[2018-11-21 17:17:23,75] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/compare_to_rm.cwl
[2018-11-21 17:17:25,65] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/batch_for_jointvc.cwl
[2018-11-21 17:17:27,62] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/wf-jointcall.cwl
[2018-11-21 17:17:28,79] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/get_parallel_regions_jointvc.cwl
[2018-11-21 17:17:30,65] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/run_jointvc.cwl
[2018-11-21 17:17:32,91] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/concat_batch_variantcalls_jointvc.cwl
[2018-11-21 17:17:35,19] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/postprocess_variants.cwl
[2018-11-21 17:17:37,24] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/finalize_jointvc.cwl
[2018-11-21 17:17:39,20] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/summarize_vc.cwl
[2018-11-21 17:17:41,12] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/calculate_sv_bins.cwl
[2018-11-21 17:17:43,04] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/calculate_sv_coverage.cwl
[2018-11-21 17:17:45,01] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/normalize_sv_coverage.cwl
[2018-11-21 17:17:46,84] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/batch_for_sv.cwl
[2018-11-21 17:17:48,82] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/wf-svcall.cwl
[2018-11-21 17:17:49,97] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/detect_sv.cwl
[2018-11-21 17:17:52,28] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/summarize_sv.cwl
[2018-11-21 17:17:54,22] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/qc_to_rec.cwl
[2018-11-21 17:17:56,57] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/pipeline_summary.cwl
[2018-11-21 17:17:58,81] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/multiqc_summary.cwl
[2018-11-21 17:18:00,78] [info] Pre Processing Inputs...
[2018-11-21 17:18:00,91] [info] Workflow heartbeat configuration:
{
  "cromwellId" : "cromid-b584c34",
  "heartbeatInterval" : "2 minutes",
  "ttl" : "10 minutes",
  "writeBatchSize" : 10000,
  "writeThreshold" : 10000
}
[2018-11-21 17:18:01,05] [info] Metadata summary refreshing every 2 seconds.
[2018-11-21 17:18:01,07] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-11-21 17:18:01,07] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-11-21 17:18:01,10] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
[2018-11-21 17:18:01,53] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds.
[2018-11-21 17:18:01,54] [info] SingleWorkflowRunnerActor: Version 36-unknown-SNAP
[2018-11-21 17:18:01,55] [info] SingleWorkflowRunnerActor: Submitting workflow
[2018-11-21 17:18:01,59] [info] CWL (Unspecified version) workflow 0856301d-b471-4dd7-bd81-029ac3cf304d submitted
[2018-11-21 17:18:01,59] [info] SingleWorkflowRunnerActor: Workflow submitted 0856301d-b471-4dd7-bd81-029ac3cf304d
[2018-11-21 17:18:01,60] [info] 1 new workflows fetched
[2018-11-21 17:18:01,60] [info] WorkflowManagerActor Starting workflow 0856301d-b471-4dd7-bd81-029ac3cf304d
[2018-11-21 17:18:01,60] [info] WorkflowManagerActor Successfully started WorkflowActor-0856301d-b471-4dd7-bd81-029ac3cf304d
[2018-11-21 17:18:01,60] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2018-11-21 17:18:01,61] [warn] SingleWorkflowRunnerActor: received unexpected message: Done in state RunningSwraData
[2018-11-21 17:18:01,62] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
[2018-11-21 17:18:01,68] [info] MaterializeWorkflowDescriptorActor [0856301d]: Parsing workflow as CWL v1.0
[2018-11-21 17:18:03,16] [info] MaterializeWorkflowDescriptorActor [0856301d]: Call-to-Backend assignments: alignment_to_rec -> Local, calculate_sv_coverage -> Local, batch_for_variantcall -> Local, prep_samples -> Local, get_parallel_regions_jointvc -> Local, pipeline_summary -> Local, finalize_jointvc -> Local, postprocess_alignment -> Local, batch_for_jointvc -> Local, batch_for_sv -> Local, compare_to_rm -> Local, concat_batch_variantcalls_jointvc -> Local, process_alignment -> Local, variantcall_batch_region -> Local, calculate_sv_bins -> Local, prep_samples_to_rec -> Local, postprocess_variants -> Local, combine_sample_regions -> Local, get_parallel_regions -> Local, postprocess_alignment_to_rec -> Local, qc_to_rec -> Local, detect_sv -> Local, merge_split_alignments -> Local, summarize_sv -> Local, multiqc_summary -> Local, prep_align_inputs -> Local, normalize_sv_coverage -> Local, run_jointvc -> Local, summarize_vc -> Local, concat_batch_variantcalls -> Local
[2018-11-21 17:18:13,93] [error] WorkflowManagerActor Workflow 0856301d-b471-4dd7-bd81-029ac3cf304d failed (during InitializingWorkflowState): cromwell.core.CromwellFatalException: java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:46)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:37)
    at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:413)
    at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:131)
    at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:127)
    at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:100)
    at cromwell.cloudsupport.gcp.auth.ApplicationDefaultMode$.applicationDefaultCredentials$lzycompute(GoogleAuthMode.scala:276)
    at cromwell.cloudsupport.gcp.auth.ApplicationDefaultMode$.cromwell$cloudsupport$gcp$auth$ApplicationDefaultMode$$applicationDefaultCredentials(GoogleAuthMode.scala:276)
    at cromwell.cloudsupport.gcp.auth.ApplicationDefaultMode.credentials(GoogleAuthMode.scala:282)
    at cromwell.cloudsupport.gcp.auth.ApplicationDefaultMode.credentials(GoogleAuthMode.scala:279)
    at cromwell.cloudsupport.gcp.auth.GoogleAuthMode.pipelinesApiCredentials(GoogleAuthMode.scala:128)
    at cromwell.cloudsupport.gcp.auth.GoogleAuthMode.pipelinesApiCredentials$(GoogleAuthMode.scala:127)
    at cromwell.cloudsupport.gcp.auth.ApplicationDefaultMode.pipelinesApiCredentials(GoogleAuthMode.scala:279)
    at cromwell.filesystems.gcs.GoogleUtil$EnhancedGoogleAuthMode$.credential$1(GoogleUtil.scala:33)
    at cromwell.filesystems.gcs.GoogleUtil$EnhancedGoogleAuthMode$.$anonfun$retryPipelinesApiCredentials$4(GoogleUtil.scala:49)
    at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:654)
    at scala.util.Success.$anonfun$map$1(Try.scala:251)
    at scala.util.Success.map(Try.scala:209)
    at scala.concurrent.Future.$anonfun$map$1(Future.scala:288)
    at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
    at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
    ... 12 more

[2018-11-21 17:18:13,93] [info] WorkflowManagerActor WorkflowActor-0856301d-b471-4dd7-bd81-029ac3cf304d is in a terminal state: WorkflowFailedState
[2018-11-21 17:18:21,95] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2018-11-21 17:18:26,22] [info] SingleWorkflowRunnerActor writing metadata to /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/testingVC-merged-metadata.json
[2018-11-21 17:18:26,25] [info] Workflow polling stopped
[2018-11-21 17:18:26,26] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2018-11-21 17:18:26,27] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2018-11-21 17:18:26,27] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2018-11-21 17:18:26,27] [info] Aborting all running workflows.
[2018-11-21 17:18:26,27] [info] JobExecutionTokenDispenser stopped
[2018-11-21 17:18:26,27] [info] WorkflowStoreActor stopped
[2018-11-21 17:18:26,27] [info] WorkflowLogCopyRouter stopped
[2018-11-21 17:18:26,27] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2018-11-21 17:18:26,28] [info] WorkflowManagerActor All workflows finished
[2018-11-21 17:18:26,28] [info] WorkflowManagerActor stopped
[2018-11-21 17:18:26,28] [info] Connection pools shut down
[2018-11-21 17:18:26,28] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2018-11-21 17:18:26,28] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2018-11-21 17:18:26,28] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2018-11-21 17:18:26,28] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2018-11-21 17:18:26,28] [info] SubWorkflowStoreActor stopped
[2018-11-21 17:18:26,28] [info] JobStoreActor stopped
[2018-11-21 17:18:26,28] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2018-11-21 17:18:26,28] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2018-11-21 17:18:26,28] [info] KvWriteActor Shutting down: 0 queued messages to process
[2018-11-21 17:18:26,28] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2018-11-21 17:18:26,28] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2018-11-21 17:18:26,28] [info] CallCacheWriteActor stopped
[2018-11-21 17:18:26,29] [info] ServiceRegistryActor stopped
[2018-11-21 17:18:26,29] [info] DockerHashActor stopped
[2018-11-21 17:18:26,29] [info] IoProxy stopped
[2018-11-21 17:18:26,30] [info] Database closed
[2018-11-21 17:18:26,30] [info] Stream materializer shut down
[2018-11-21 17:18:26,30] [info] WDL HTTP import resolver closed
Workflow 0856301d-b471-4dd7-bd81-029ac3cf304d transitioned to state Failed
Unable to find image 'quay.io/bcbio/bcbio-base:latest' locally
latest: Pulling from bcbio/bcbio-base
18d680d61657: Pull complete 
0addb6fece63: Pull complete 
78e58219b215: Pull complete 
eb6959a66df2: Pull complete 
f4d326eaeb65: Pull complete 
Digest: sha256:4d4f67015e28437cda1c39e5b40926600eedd41b1b87fe69ffe5445c7bf28f84
Status: Downloaded newer image for quay.io/bcbio/bcbio-base:latest
Failed bcbio Cromwell run
-------------------------

The same thing happens when running the CWL tests:

wget -O test_bcbio_cwl.tar.gz https://github.com/bcbio/test_bcbio_cwl/archive/master.tar.gz
tar -xzvpf test_bcbio_cwl.tar.gz
cd test_bcbio_cwl-master/somatic
bcbio_vm.py cwlrun cromwell somatic-workflow

P.S. Sorry for the confusion, but does the data still need to be downloaded separately, when using Bcbio_vm with CWL, or do the CWL wrappers download the genome data automatically? Since the instructions are to add the data location to the bcbio_system.yaml file, I guess that it still needs to be installed using the bcbio_vm.py script, right?

I'm trying to avoid relying on a full local install of Bcbio-nextgen. However, running the following, without specifying a Docker image, doesn't seem to work:

bcbio_vm.py --datadir=~/bcbio-stuff/bcbio-vm install --data --genomes sacCer3 --aligners bwa
Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/bcbio_vm.py", line 42, in cmd_install
    install.full(args, devel.DOCKER)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbiovm/docker/install.py", line 39, in full
    _check_docker_image(args)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbiovm/docker/install.py", line 136, in _check_docker_image
    raise ValueError("Could not find docker image %s in local repository" % args.image)
ValueError: Could not find docker image bcbio/bcbio in local repository

Sorry you're still hitting issues here, thank you for all the patience. It looks like your CWL run specification is correct and should be adding the right tags to the output CWL. You can check the dockerImageId in any cwl file in yourproject-workflow/steps to confirm that it's added to the specification. What I'm confused about is why it's pulling bcbio-base from latest. I'd thought this was only used in the build step and should be baked into the actual bcbio-vc container you're using but must not be right. I think the issue is that we need to be able to specify the revision in the FROM tag in the Docker container:

https://github.com/bcbio/bcbio_docker/blob/master/Dockerfile.tools#L3

I'll dig to check if this is now possible and see if we can pin these explicitly. For now if you have bcbio/bcbio-base:latest available it'll use your specific docker container for the actual run. Does having that avoid the Cromwell errors?

I'll also work on not needing to have the Docker image present for downloading data. We should be able to avoid that now. Apologies, this needs some cleanup in the transition to CWL and I'll work on that over the next couple of days and follow up here when there is a better process in place.

Thanks again for helping work through these issues.

It actually did download the bcbio/bcbio-base:latest image during the first run, but it still crashed. On a subsequent run, it skipped over the bcbio/bcbio-base:latest part, and then ended with Failed bcbio Cromwell run.

It seems like the source of the error is the following line:

[2018-11-21 17:18:13,93] [error] WorkflowManagerActor Workflow 0856301d-b471-4dd7-bd81-029ac3cf304d failed (during InitializingWorkflowState): cromwell.core.CromwellFatalException: java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:46)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:37)
[...]

This is puzzling to me. Why does it error out due to missing Google cloud credentials, when I'm trying to run this locally?

I checked the dockerImageId value and it is correctly set to quay.io/bcbio/bcbio-vc:1.1.1a-82fb2a0. I didn't find the string "bcbio-base" in any of the files inside the CWL project directory.

I ran the CWL test with toil instead of cromwell (bcbio_vm.py cwlrun toil somatic-workflow) and this also failed. Apologies for the large block of text:

bcbio_vm.py cwlrun toil somatic-workflow
Running: export PATH=/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin:$PATH && cwltoil --jobStore /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/cwltoil_jobstore --logFile /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/somatic-toil.log --workDir /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/tmpdir --linkImports -- /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic.cwl /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic-samples.json
amizeranschi-Predator-PH317-51 2018-11-22 21:58:35,671 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:35,671 MainThread INFO toil.lib.bioio: Logging to file '/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/somatic-toil.log'.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:35,671 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:35,671 MainThread INFO toil.lib.bioio: Logging to file '/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/somatic-toil.log'.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:35,675 MainThread INFO toil.jobStores.abstractJobStore: The workflow ID is: 'ded58332-e978-4cd6-a654-df89ccaff86b'
Resolved '/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic.cwl' to 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic.cwl'
amizeranschi-Predator-PH317-51 2018-11-22 21:58:35,676 MainThread INFO cwltool: Resolved '/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic.cwl' to 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic.cwl'
../somatic-workflow/steps/alignment_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:37,867 MainThread INFO cwltool: ../somatic-workflow/steps/alignment_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/prep_align_inputs.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:37,984 MainThread INFO cwltool: ../somatic-workflow/steps/prep_align_inputs.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/process_alignment.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:39,706 MainThread INFO cwltool: ../somatic-workflow/steps/process_alignment.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/process_alignment.cwl:83:3: Unknown hint http://arvados.org/cwl#APIRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:39,706 MainThread INFO cwltool: ../somatic-workflow/steps/process_alignment.cwl:83:3: Unknown hint http://arvados.org/cwl#APIRequirement
../somatic-workflow/steps/merge_split_alignments.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:39,778 MainThread INFO cwltool: ../somatic-workflow/steps/merge_split_alignments.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/prep_samples_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:39,832 MainThread INFO cwltool: ../somatic-workflow/steps/prep_samples_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/prep_samples.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:39,867 MainThread INFO cwltool: ../somatic-workflow/steps/prep_samples.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/postprocess_alignment_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:39,942 MainThread INFO cwltool: ../somatic-workflow/steps/postprocess_alignment_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/postprocess_alignment.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,015 MainThread INFO cwltool: ../somatic-workflow/steps/postprocess_alignment.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/postprocess_alignment.cwl:52:3: Unknown hint http://arvados.org/cwl#APIRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,015 MainThread INFO cwltool: ../somatic-workflow/steps/postprocess_alignment.cwl:52:3: Unknown hint http://arvados.org/cwl#APIRequirement
../somatic-workflow/steps/combine_sample_regions.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,067 MainThread INFO cwltool: ../somatic-workflow/steps/combine_sample_regions.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/combine_sample_regions.cwl:40:3: Unknown hint http://arvados.org/cwl#APIRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,067 MainThread INFO cwltool: ../somatic-workflow/steps/combine_sample_regions.cwl:40:3: Unknown hint http://arvados.org/cwl#APIRequirement
../somatic-workflow/steps/batch_for_variantcall.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,222 MainThread INFO cwltool: ../somatic-workflow/steps/batch_for_variantcall.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/get_parallel_regions.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,433 MainThread INFO cwltool: ../somatic-workflow/steps/get_parallel_regions.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/variantcall_batch_region.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,943 MainThread INFO cwltool: ../somatic-workflow/steps/variantcall_batch_region.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/variantcall_batch_region.cwl:112:3: Unknown hint http://arvados.org/cwl#APIRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,944 MainThread INFO cwltool: ../somatic-workflow/steps/variantcall_batch_region.cwl:112:3: Unknown hint http://arvados.org/cwl#APIRequirement
../somatic-workflow/steps/variantcall_batch_region.cwl:113:3: Unknown hint http://arvados.org/cwl#RuntimeConstraints
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,944 MainThread INFO cwltool: ../somatic-workflow/steps/variantcall_batch_region.cwl:113:3: Unknown hint http://arvados.org/cwl#RuntimeConstraints
../somatic-workflow/steps/concat_batch_variantcalls.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,030 MainThread INFO cwltool: ../somatic-workflow/steps/concat_batch_variantcalls.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/concat_batch_variantcalls.cwl:40:3: Unknown hint http://arvados.org/cwl#APIRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,030 MainThread INFO cwltool: ../somatic-workflow/steps/concat_batch_variantcalls.cwl:40:3: Unknown hint http://arvados.org/cwl#APIRequirement
../somatic-workflow/steps/postprocess_variants.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,098 MainThread INFO cwltool: ../somatic-workflow/steps/postprocess_variants.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/compare_to_rm.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,208 MainThread INFO cwltool: ../somatic-workflow/steps/compare_to_rm.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/batch_for_ensemble.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,357 MainThread INFO cwltool: ../somatic-workflow/steps/batch_for_ensemble.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/combine_calls.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,426 MainThread INFO cwltool: ../somatic-workflow/steps/combine_calls.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/summarize_vc.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,523 MainThread INFO cwltool: ../somatic-workflow/steps/summarize_vc.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/qc_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,623 MainThread INFO cwltool: ../somatic-workflow/steps/qc_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/pipeline_summary.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,709 MainThread INFO cwltool: ../somatic-workflow/steps/pipeline_summary.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/pipeline_summary.cwl:76:3: Unknown hint http://arvados.org/cwl#RuntimeConstraints
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,709 MainThread INFO cwltool: ../somatic-workflow/steps/pipeline_summary.cwl:76:3: Unknown hint http://arvados.org/cwl#RuntimeConstraints
../somatic-workflow/steps/multiqc_summary.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,760 MainThread INFO cwltool: ../somatic-workflow/steps/multiqc_summary.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,009 MainThread INFO toil.common: Using the single machine batch system
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,009 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxCores to CPU count of system (8).
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,010 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxMemory to physically available memory (8242171904).
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,010 MainThread INFO toil.common: Created the workflow directory at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/tmpdir/toil-ded58332-e978-4cd6-a654-df89ccaff86b-5a229753ce0549c6bf7503d6f103b439
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,010 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxDisk to physically available disk (78591340544).
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,017 MainThread INFO toil.common: User script ModuleDescriptor(dirPath='/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=False) belongs to Toil. No need to hot-deploy it.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,018 MainThread INFO toil.common: No user script to hot-deploy.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,018 MainThread INFO toil.common: Written the environment for the jobs to the environment file
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,018 MainThread INFO toil.common: Caching all jobs in job store
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,019 MainThread INFO toil.common: 0 jobs downloaded.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,162 MainThread INFO toil: Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,163 MainThread INFO toil.realtimeLogger: Real-time logging disabled
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,165 MainThread INFO toil.toilState: (Re)building internal scheduler state
2018-11-22 21:58:42,165 - toil.toilState - INFO - (Re)building internal scheduler state
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,165 MainThread INFO toil.leader: Found 1 jobs to start and 0 jobs with successors to run
2018-11-22 21:58:42,165 - toil.leader - INFO - Found 1 jobs to start and 0 jobs with successors to run
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,165 MainThread INFO toil.leader: Checked batch system has no running jobs and no updated jobs
2018-11-22 21:58:42,165 - toil.leader - INFO - Checked batch system has no running jobs and no updated jobs
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,166 MainThread INFO toil.leader: Starting the main loop
2018-11-22 21:58:42,166 - toil.leader - INFO - Starting the main loop
amizeranschi-Predator-PH317-51 2018-11-22 21:58:43,084 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi with job batch system ID: 1 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 21:58:43,084 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi with job batch system ID: 1 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 21:58:43,087 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with job batch system ID: 2 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 21:58:43,087 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with job batch system ID: 2 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,722 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
2018-11-22 22:10:13,722 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,723 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
2018-11-22 22:10:13,723 - toil.leader - WARNING - The job seems to have left a log file, indicating failure: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,723 MainThread WARNING toil.leader: D/X/job_AlQdQ    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
2018-11-22 22:10:13,723 - toil.leader - WARNING - D/X/job_AlQdQ    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,723 MainThread WARNING toil.leader: D/X/job_AlQdQ    INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
2018-11-22 22:10:13,723 - toil.leader - WARNING - D/X/job_AlQdQ    INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    ['docker', 'pull', 'quay.io/bcbio/bcbio-vc']
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    ['docker', 'pull', 'quay.io/bcbio/bcbio-vc']
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    INFO:cwltool:['docker', 'pull', 'quay.io/bcbio/bcbio-vc']
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    INFO:cwltool:['docker', 'pull', 'quay.io/bcbio/bcbio-vc']
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    Using default tag: latest
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    Using default tag: latest
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    latest: Pulling from bcbio/bcbio-vc
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    latest: Pulling from bcbio/bcbio-vc
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    18d680d61657: Already exists
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    18d680d61657: Already exists
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    0addb6fece63: Already exists
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    0addb6fece63: Already exists
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    78e58219b215: Already exists
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    78e58219b215: Already exists
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    eb6959a66df2: Already exists
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    eb6959a66df2: Already exists
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    f4d326eaeb65: Already exists
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    f4d326eaeb65: Already exists
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    73f63353b610: Pulling fs layer
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    73f63353b610: Pulling fs layer
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    af6367e4755f: Pulling fs layer
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    af6367e4755f: Pulling fs layer
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    73f63353b610: Download complete
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    73f63353b610: Download complete
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    73f63353b610: Pull complete
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    73f63353b610: Pull complete
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    af6367e4755f: Verifying Checksum
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    af6367e4755f: Verifying Checksum
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    af6367e4755f: Download complete
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    af6367e4755f: Download complete
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    af6367e4755f: Pull complete
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    af6367e4755f: Pull complete
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ    Digest: sha256:7748650d81ab0b549172846b467f0281c15e523dbf977b2731bda20fad284711
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ    Digest: sha256:7748650d81ab0b549172846b467f0281c15e523dbf977b2731bda20fad284711
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ    Status: Downloaded newer image for quay.io/bcbio/bcbio-vc:latest
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ    Status: Downloaded newer image for quay.io/bcbio/bcbio-vc:latest
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ    Traceback (most recent call last):
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ    Traceback (most recent call last):
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        returnValues = self._run(jobGraph, fileStore)
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        returnValues = self._run(jobGraph, fileStore)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        return self.run(fileStore)
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        return self.run(fileStore)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        (output, status) = cwltool.main.single_job_executor(**opts)
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        (output, status) = cwltool.main.single_job_executor(**opts)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        r.run(**kwargs)
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        r.run(**kwargs)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        self._setup(kwargs)
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        self._setup(kwargs)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,726 MainThread WARNING toil.leader: D/X/job_AlQdQ    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai) not found or is not a regular file.
2018-11-22 22:10:13,726 - toil.leader - WARNING - D/X/job_AlQdQ    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai) not found or is not a regular file.
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,726 MainThread WARNING toil.leader: D/X/job_AlQdQ    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
2018-11-22 22:10:13,726 - toil.leader - WARNING - D/X/job_AlQdQ    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,726 MainThread WARNING toil.leader: D/X/job_AlQdQ    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ to 1
2018-11-22 22:10:13,726 - toil.leader - WARNING - D/X/job_AlQdQ    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ to 1
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,726 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with job batch system ID: 3 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 22:10:13,726 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with job batch system ID: 3 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,092 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
2018-11-22 22:10:16,092 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
2018-11-22 22:10:16,093 - toil.leader - WARNING - The job seems to have left a log file, indicating failure: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ    INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ    INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ    Traceback (most recent call last):
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ    Traceback (most recent call last):
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ        returnValues = self._run(jobGraph, fileStore)
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ        returnValues = self._run(jobGraph, fileStore)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ        return self.run(fileStore)
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ        return self.run(fileStore)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ        (output, status) = cwltool.main.single_job_executor(**opts)
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ        (output, status) = cwltool.main.single_job_executor(**opts)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ        r.run(**kwargs)
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ        r.run(**kwargs)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ        self._setup(kwargs)
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ        self._setup(kwargs)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai) not found or is not a regular file.
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai) not found or is not a regular file.
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ to 0
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ to 0
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,099 MainThread WARNING toil.leader: Job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ is completely failed
2018-11-22 22:10:16,099 - toil.leader - WARNING - Job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ is completely failed
amizeranschi-Predator-PH317-51 2018-11-22 22:10:22,216 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi
2018-11-22 22:10:22,216 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi
amizeranschi-Predator-PH317-51 2018-11-22 22:10:22,833 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl t/Q/jobYX5NcG with job batch system ID: 5 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 22:10:22,833 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl t/Q/jobYX5NcG with job batch system ID: 5 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 22:10:22,837 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl v/z/jobBKnwKQ with job batch system ID: 6 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 22:10:22,837 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl v/z/jobBKnwKQ with job batch system ID: 6 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 22:10:22,838 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl 5/9/jobg3gTOT with job batch system ID: 7 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 22:10:22,838 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl 5/9/jobg3gTOT with job batch system ID: 7 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 22:10:22,838 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl i/T/jobIa8lhb with job batch system ID: 8 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 22:10:22,838 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl i/T/jobIa8lhb with job batch system ID: 8 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 22:10:32,212 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl 5/9/jobg3gTOT
2018-11-22 22:10:32,212 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl 5/9/jobg3gTOT
amizeranschi-Predator-PH317-51 2018-11-22 22:10:32,527 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl t/Q/jobYX5NcG
2018-11-22 22:10:32,527 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl t/Q/jobYX5NcG
amizeranschi-Predator-PH317-51 2018-11-22 22:10:38,046 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl v/z/jobBKnwKQ
2018-11-22 22:10:38,046 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl v/z/jobBKnwKQ
amizeranschi-Predator-PH317-51 2018-11-22 22:10:40,261 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl i/T/jobIa8lhb
2018-11-22 22:10:40,261 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl i/T/jobIa8lhb
amizeranschi-Predator-PH317-51 2018-11-22 22:10:50,937 MainThread INFO toil.leader: No jobs left to run so exiting.
2018-11-22 22:10:50,937 - toil.leader - INFO - No jobs left to run so exiting.
amizeranschi-Predator-PH317-51 2018-11-22 22:10:50,938 MainThread INFO toil.leader: Finished the main loop
2018-11-22 22:10:50,938 - toil.leader - INFO - Finished the main loop
amizeranschi-Predator-PH317-51 2018-11-22 22:10:50,938 MainThread INFO toil.serviceManager: Waiting for service manager thread to finish ...
2018-11-22 22:10:50,938 - toil.serviceManager - INFO - Waiting for service manager thread to finish ...
amizeranschi-Predator-PH317-51 2018-11-22 22:10:51,346 MainThread INFO toil.serviceManager: ... finished shutting down the service manager. Took 0.406985998154 seconds
2018-11-22 22:10:51,346 - toil.serviceManager - INFO - ... finished shutting down the service manager. Took 0.406985998154 seconds
amizeranschi-Predator-PH317-51 2018-11-22 22:10:51,347 MainThread INFO toil.statsAndLogging: Waiting for stats and logging collator thread to finish ...
2018-11-22 22:10:51,347 - toil.statsAndLogging - INFO - Waiting for stats and logging collator thread to finish ...
amizeranschi-Predator-PH317-51 2018-11-22 22:10:51,464 MainThread INFO toil.statsAndLogging: ... finished collating stats and logs. Took 0.116714000702 seconds
2018-11-22 22:10:51,464 - toil.statsAndLogging - INFO - ... finished collating stats and logs. Took 0.116714000702 seconds
amizeranschi-Predator-PH317-51 2018-11-22 22:10:51,464 MainThread INFO toil.leader: Finished toil run with 5 failed jobs
2018-11-22 22:10:51,464 - toil.leader - INFO - Finished toil run with 5 failed jobs
amizeranschi-Predator-PH317-51 2018-11-22 22:10:51,465 MainThread INFO toil.leader: Failed jobs at end of the run: 'CWLGather' s/0/job8oR0hU 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ 'CWLWorkflow' 9/7/jobPiIsEy 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi 'CWLScatter' u/X/jobMpPwRP
2018-11-22 22:10:51,465 - toil.leader - INFO - Failed jobs at end of the run: 'CWLGather' s/0/job8oR0hU 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ 'CWLWorkflow' 9/7/jobPiIsEy 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi 'CWLScatter' u/X/jobMpPwRP
Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/cwltoil", line 11, in <module>
    load_entry_point('toil==3.14.0', 'console_scripts', 'cwltoil')()
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 990, in main
    outobj = toil.start(wf1)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/common.py", line 743, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/common.py", line 1018, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/leader.py", line 226, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'file:/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/cwltoil_jobstore' contains 5 failed jobs: 'CWLGather' s/0/job8oR0hU, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ, 'CWLWorkflow' 9/7/jobPiIsEy, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi, 'CWLScatter' u/X/jobMpPwRP
=========> Failed job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ 
INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
    return self.run(fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
    (output, status) = cwltool.main.single_job_executor(**opts)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
    r.run(**kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
    self._setup(kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
    "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai) not found or is not a regular file.
ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ to 0
<=========

Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 294, in run
    _TOOLS[args.tool](args)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 121, in _run_toil
    _run_tool(cmd, not args.no_container, work_dir)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 42, in _run_tool
    subprocess.check_call(cmd, shell=True)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'export PATH=/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin:$PATH && cwltoil --jobStore /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/cwltoil_jobstore --logFile /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/somatic-toil.log --workDir /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/tmpdir --linkImports -- /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic.cwl /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic-samples.json' returned non-zero exit status 1

Thanks again for all this testing and your patience in working through the issues.

For the Cromwell problem, there was a bug in how we were specifying the filesystem for local Docker runs which made it look for Google permissions. I pushed a fix for this, so if you update with:

/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/conda install -c conda-forge -c bioconda bcbio-nextgen

and re-run it should hopefully work correctly for all your test cases.

For the Toil run issue, this was a problem with the generated test CWL. If you get the latest from the git repository (https://github.com/bcbio/test_bcbio_cwl) it should correctly account for the missing bai index that is causing it to fail.

Hope this gets these running for you. I'll work more on the bcbio-base tag issue, which will require adjusting the Docker bulld process and also on making it easier to install the data directly. Thanks again for testing these.

Thanks a lot for your help. However, even after upgrading the bcbio-nextgen package and downloading the latest test_bcbio_cwl from GitHub, the two CWL tests still fail for me.

Incoming large blocks of text... First, the Cromwell test -- I'm still running this as follows:

wget -O test_bcbio_cwl.tar.gz https://github.com/bcbio/test_bcbio_cwl/archive/master.tar.gz
tar -xzvpf test_bcbio_cwl.tar.gz
cd test_bcbio_cwl-master/somatic
bcbio_vm.py cwlrun cromwell somatic-workflow

And the outcome:

[...]
[2018-11-24 15:06:42,45] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/756817283/Test2-coverage.mosdepth.region.dist.txt -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-postprocess_alignment/shard-1/execution/coverage/Test2/Test2-coverage.mosdepth.region.dist.txt: Operation not permitted
[2018-11-24 15:06:42,46] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
[2018-11-24 15:06:42,46] [warn] Localization via copy has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
[2018-11-24 15:06:42,46] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
[2018-11-24 15:06:42,46] [error] Failed to hash /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
java.io.FileNotFoundException: Cannot hash file /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv because it can't be found
    at cromwell.backend.impl.sfs.config.ConfigHashingStrategy.usingStandardInitData$1(ConfigHashingStrategy.scala:46)
    at cromwell.backend.impl.sfs.config.ConfigHashingStrategy.getHash(ConfigHashingStrategy.scala:57)
    at cromwell.backend.impl.sfs.config.ConfigBackendFileHashingActor.customHashStrategy(ConfigBackendFileHashingActor.scala:26)
    at cromwell.backend.standard.callcaching.StandardFileHashingActor$$anonfun$fileHashingReceive$1.applyOrElse(StandardFileHashingActor.scala:79)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
    at akka.actor.Actor.aroundReceive(Actor.scala:517)
    at akka.actor.Actor.aroundReceive$(Actor.scala:515)
    at cromwell.backend.standard.callcaching.StandardFileHashingActor.akka$actor$Timers$$super$aroundReceive(StandardFileHashingActor.scala:59)
    at akka.actor.Timers.aroundReceive(Timers.scala:55)
    at akka.actor.Timers.aroundReceive$(Timers.scala:40)
    at cromwell.backend.standard.callcaching.StandardFileHashingActor.aroundReceive(StandardFileHashingActor.scala:59)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
    at akka.actor.ActorCell.invoke(ActorCell.scala:557)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
    at akka.dispatch.Mailbox.run(Mailbox.scala:225)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2018-11-24 15:06:42,46] [warn] Localization via copy has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
[2018-11-24 15:06:42,46] [error] 3c7d418d:qc_to_rec:-1:1: Hash error, disabling call caching for this job.
java.io.FileNotFoundException: Cannot hash file /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv because it can't be found
    at cromwell.backend.impl.sfs.config.ConfigHashingStrategy.usingStandardInitData$1(ConfigHashingStrategy.scala:46)
    at cromwell.backend.impl.sfs.config.ConfigHashingStrategy.getHash(ConfigHashingStrategy.scala:57)
    at cromwell.backend.impl.sfs.config.ConfigBackendFileHashingActor.customHashStrategy(ConfigBackendFileHashingActor.scala:26)
    at cromwell.backend.standard.callcaching.StandardFileHashingActor$$anonfun$fileHashingReceive$1.applyOrElse(StandardFileHashingActor.scala:79)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
    at akka.actor.Actor.aroundReceive(Actor.scala:517)
    at akka.actor.Actor.aroundReceive$(Actor.scala:515)
    at cromwell.backend.standard.callcaching.StandardFileHashingActor.akka$actor$Timers$$super$aroundReceive(StandardFileHashingActor.scala:59)
    at akka.actor.Timers.aroundReceive(Timers.scala:55)
    at akka.actor.Timers.aroundReceive$(Timers.scala:40)
    at cromwell.backend.standard.callcaching.StandardFileHashingActor.aroundReceive(StandardFileHashingActor.scala:59)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
    at akka.actor.ActorCell.invoke(ActorCell.scala:557)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
    at akka.dispatch.Mailbox.run(Mailbox.scala:225)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2018-11-24 15:06:42,46] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
[2018-11-24 15:06:42,46] [warn] Localization via copy has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
[2018-11-24 15:06:42,46] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
[2018-11-24 15:06:42,46] [warn] Localization via copy has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
[2018-11-24 15:06:42,47] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/756817283/Test2-coverage.regions.bed.gz -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-postprocess_alignment/shard-1/execution/coverage/Test2/Test2-coverage.regions.bed.gz: Operation not permitted
[2018-11-24 15:06:42,47] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/Test1-germline-freebayes.vcf.gz -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/Test1-germline-freebayes.vcf.gz: Operation not permitted
[2018-11-24 15:06:42,47] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/Test1-germline-freebayes.vcf.gz.tbi -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/Test1-germline-freebayes.vcf.gz.tbi: Operation not permitted
[2018-11-24 15:06:42,47] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/b1-strelka2.vcf.gz -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/b1-strelka2.vcf.gz: Operation not permitted
[2018-11-24 15:06:42,47] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/b1-strelka2.vcf.gz.tbi -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/b1-strelka2.vcf.gz.tbi: Operation not permitted
[2018-11-24 15:06:42,47] [info] Message [cromwell.backend.standard.callcaching.StandardFileHashingActor$SingleFileHashRequest] from Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-3c7d418d-1198-4f96-b15d-4c12712bc5b8/WorkflowExecutionActor-3c7d418d-1198-4f96-b15d-4c12712bc5b8/3c7d418d-1198-4f96-b15d-4c12712bc5b8-EngineJobExecutionActor-qc_to_rec:NA:1/ejha_for_3c7d418d-1198-4f96-b15d-4c12712bc5b8:BackendJobDescriptorKey_CommandCallNode_qc_to_rec:-1:1/CCHashingJobActor-3c7d418d-qc_to_rec:NA:1#1527628571] to Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-3c7d418d-1198-4f96-b15d-4c12712bc5b8/WorkflowExecutionActor-3c7d418d-1198-4f96-b15d-4c12712bc5b8/3c7d418d-1198-4f96-b15d-4c12712bc5b8-EngineJobExecutionActor-qc_to_rec:NA:1/ejha_for_3c7d418d-1198-4f96-b15d-4c12712bc5b8:BackendJobDescriptorKey_CommandCallNode_qc_to_rec:-1:1/CCHashingJobActor-3c7d418d-qc_to_rec:NA:1/FileHashingActor_for_qc_to_rec:NA:1#1989442513] was not delivered. [1] dead letters encountered, no more dead letters will be logged. If this is not an expected behavior, then [Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-3c7d418d-1198-4f96-b15d-4c12712bc5b8/WorkflowExecutionActor-3c7d418d-1198-4f96-b15d-4c12712bc5b8/3c7d418d-1198-4f96-b15d-4c12712bc5b8-EngineJobExecutionActor-qc_to_rec:NA:1/ejha_for_3c7d418d-1198-4f96-b15d-4c12712bc5b8:BackendJobDescriptorKey_CommandCallNode_qc_to_rec:-1:1/CCHashingJobActor-3c7d418d-qc_to_rec:NA:1/FileHashingActor_for_qc_to_rec:NA:1#1989442513]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2018-11-24 15:06:42,48] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/b1-vardict.vcf.gz -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/b1-vardict.vcf.gz: Operation not permitted
[2018-11-24 15:06:42,48] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/b1-vardict.vcf.gz.tbi -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/b1-vardict.vcf.gz.tbi: Operation not permitted
[2018-11-24 15:06:42,48] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/b1-mutect2.vcf.gz -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/b1-mutect2.vcf.gz: Operation not permitted
[2018-11-24 15:06:42,48] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/b1-mutect2.vcf.gz.tbi -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/b1-mutect2.vcf.gz.tbi: Operation not permitted
[2018-11-24 15:06:42,48] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/b1-ensemble.vcf.gz -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/b1-ensemble.vcf.gz: Operation not permitted
[2018-11-24 15:06:42,48] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/b1-ensemble.vcf.gz.tbi -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/b1-ensemble.vcf.gz.tbi: Operation not permitted
[2018-11-24 15:06:42,48] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/VarOnly-None.vcf.gz -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/VarOnly-None.vcf.gz: Operation not permitted
[2018-11-24 15:06:42,49] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/VarOnly-None.vcf.gz.tbi -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/VarOnly-None.vcf.gz.tbi: Operation not permitted
[2018-11-24 15:06:42,49] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/b2-vardict.vcf.gz -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/b2-vardict.vcf.gz: Operation not permitted
[2018-11-24 15:06:42,49] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/449104517/b2-vardict.vcf.gz.tbi -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-summarize_vc/execution/variants/calls/b2-vardict.vcf.gz.tbi: Operation not permitted
[2018-11-24 15:06:42,49] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/756817283/Test2-variant_regions.regions.bed.gz -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-postprocess_alignment/shard-1/execution/coverage/Test2/Test2-variant_regions.regions.bed.gz: Operation not permitted
[2018-11-24 15:06:42,49] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/787837089/Test1-variant_regions.regions.bed.gz -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-postprocess_alignment/shard-2/execution/coverage/Test1/Test1-variant_regions.regions.bed.gz: Operation not permitted
[2018-11-24 15:06:42,49] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/26451839/Umi1-variant_regions.regions.bed.gz -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-postprocess_alignment/shard-3/execution/coverage/Umi1/Umi1-variant_regions.regions.bed.gz: Operation not permitted
[2018-11-24 15:06:42,49] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/756817283/Test2-coverage.thresholds.bed.gz -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-postprocess_alignment/shard-1/execution/coverage/Test2/Test2-coverage.thresholds.bed.gz: Operation not permitted
[2018-11-24 15:06:42,49] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/1862983550/Test2.txt -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-postprocess_alignment/shard-1/execution/qc/Test2/samtools/Test2.txt: Operation not permitted
[2018-11-24 15:06:42,49] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1542368544/Test1.txt -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-postprocess_alignment/shard-2/execution/qc/Test1/samtools/Test1.txt: Operation not permitted
[2018-11-24 15:06:42,50] [warn] Localization via hard link has failed: /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-752929130/Umi1.txt -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-postprocess_alignment/shard-3/execution/qc/Umi1/samtools/Umi1.txt: Operation not permitted
[2018-11-24 15:06:42,50] [error] BackgroundConfigAsyncJobExecutionActor [3c7d418dqc_to_rec:NA:1]: Error attempting to Execute
java.lang.Exception: Failed command instantiation
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:88)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
    at akka.actor.Actor.aroundReceive(Actor.scala:517)
    at akka.actor.Actor.aroundReceive$(Actor.scala:515)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:193)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
    at akka.actor.ActorCell.invoke(ActorCell.scala:557)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
    at akka.dispatch.Mailbox.run(Mailbox.scala:225)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
    at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
    ... 42 common frames omitted
[2018-11-24 15:06:46,28] [error] BackgroundConfigAsyncJobExecutionActor [3c7d418dqc_to_rec:NA:1]: Error attempting to Execute
java.lang.Exception: Failed command instantiation
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
    at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
    at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
    at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
    ... 35 common frames omitted
[2018-11-24 15:06:48,70] [error] BackgroundConfigAsyncJobExecutionActor [3c7d418dqc_to_rec:NA:1]: Error attempting to Execute
java.lang.Exception: Failed command instantiation
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
    at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
    at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
    at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
    ... 35 common frames omitted
[2018-11-24 15:06:50,74] [error] BackgroundConfigAsyncJobExecutionActor [3c7d418dqc_to_rec:NA:1]: Error attempting to Execute
java.lang.Exception: Failed command instantiation
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
    at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
    at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
    at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
    ... 35 common frames omitted
[2018-11-24 15:06:54,02] [error] BackgroundConfigAsyncJobExecutionActor [3c7d418dqc_to_rec:NA:1]: Error attempting to Execute
java.lang.Exception: Failed command instantiation
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
    at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
    at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
    at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
    ... 35 common frames omitted
[2018-11-24 15:06:57,51] [error] BackgroundConfigAsyncJobExecutionActor [3c7d418dqc_to_rec:NA:1]: Error attempting to Execute
java.lang.Exception: Failed command instantiation
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
    at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
    at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
    at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
    ... 35 common frames omitted
[2018-11-24 15:07:01,86] [error] BackgroundConfigAsyncJobExecutionActor [3c7d418dqc_to_rec:NA:1]: Error attempting to Execute
java.lang.Exception: Failed command instantiation
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
    at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
    at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
    at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
    ... 35 common frames omitted
[2018-11-24 15:07:06,48] [error] BackgroundConfigAsyncJobExecutionActor [3c7d418dqc_to_rec:NA:1]: Error attempting to Execute
java.lang.Exception: Failed command instantiation
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
    at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
    at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
    at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
    ... 35 common frames omitted
[2018-11-24 15:07:12,48] [error] BackgroundConfigAsyncJobExecutionActor [3c7d418dqc_to_rec:NA:1]: Error attempting to Execute
java.lang.Exception: Failed command instantiation
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
    at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
    at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
    at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
    ... 35 common frames omitted
[2018-11-24 15:07:19,23] [error] BackgroundConfigAsyncJobExecutionActor [3c7d418dqc_to_rec:NA:1]: Error attempting to Execute
java.lang.Exception: Failed command instantiation
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
    at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
    at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
    at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
    ... 35 common frames omitted
[2018-11-24 15:07:19,98] [error] WorkflowManagerActor Workflow 3c7d418d-1198-4f96-b15d-4c12712bc5b8 failed (during ExecutingWorkflowState): cromwell.core.CromwellFatalException: java.lang.Exception: Failed command instantiation
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:46)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:37)
    at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:413)
    at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.Exception: Failed command instantiation
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
    at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
    at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
    ... 6 more
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/inputs/-1854814289/versions.csv -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv
    at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
    at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
    ... 35 more

[2018-11-24 15:07:19,98] [info] WorkflowManagerActor WorkflowActor-3c7d418d-1198-4f96-b15d-4c12712bc5b8 is in a terminal state: WorkflowFailedState
[2018-11-24 15:08:36,80] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2018-11-24 15:08:39,43] [info] SingleWorkflowRunnerActor writing metadata to /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/somatic-metadata.json
[2018-11-24 15:08:39,52] [info] Workflow polling stopped
[2018-11-24 15:08:39,53] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2018-11-24 15:08:39,53] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2018-11-24 15:08:39,53] [info] Aborting all running workflows.
[2018-11-24 15:08:39,53] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2018-11-24 15:08:39,54] [info] JobExecutionTokenDispenser stopped
[2018-11-24 15:08:39,54] [info] WorkflowStoreActor stopped
[2018-11-24 15:08:39,54] [info] WorkflowLogCopyRouter stopped
[2018-11-24 15:08:39,54] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2018-11-24 15:08:39,54] [info] WorkflowManagerActor All workflows finished
[2018-11-24 15:08:39,54] [info] WorkflowManagerActor stopped
[2018-11-24 15:08:39,54] [info] Connection pools shut down
[2018-11-24 15:08:39,54] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2018-11-24 15:08:39,54] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2018-11-24 15:08:39,54] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2018-11-24 15:08:39,54] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2018-11-24 15:08:39,54] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2018-11-24 15:08:39,54] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2018-11-24 15:08:39,54] [info] CallCacheWriteActor stopped
[2018-11-24 15:08:39,54] [info] SubWorkflowStoreActor stopped
[2018-11-24 15:08:39,54] [info] JobStoreActor stopped
[2018-11-24 15:08:39,54] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2018-11-24 15:08:39,54] [info] KvWriteActor Shutting down: 0 queued messages to process
[2018-11-24 15:08:39,54] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2018-11-24 15:08:39,54] [info] DockerHashActor stopped
[2018-11-24 15:08:39,54] [info] ServiceRegistryActor stopped
[2018-11-24 15:08:39,54] [info] IoProxy stopped
[2018-11-24 15:08:39,55] [info] Database closed
[2018-11-24 15:08:39,55] [info] Stream materializer shut down
[2018-11-24 15:08:39,55] [info] WDL HTTP import resolver closed
Workflow 3c7d418d-1198-4f96-b15d-4c12712bc5b8 transitioned to state Failed
Failed bcbio Cromwell run
-------------------------
Failure in step: qc_to_rec.0
  bcbio log file     : cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/execution/log/bcbio-nextgen-debug.log
  bcbio commands file: cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec/execution/log/bcbio-nextgen-commands.log
  Cromwell directory : cromwell_work/cromwell-executions/main-somatic.cwl/3c7d418d-1198-4f96-b15d-4c12712bc5b8/call-qc_to_rec

Note: the file /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19/versions.csv does seem to be missing from my setup:

ls -la /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/genomes/hg19
total 204
drwxrwxr-x 9 amizeranschi amizeranschi   4096 Nov 24 15:36 .
drwxrwxr-x 3 amizeranschi amizeranschi   4096 Nov 24 15:36 ..
drwxrwxr-x 2 amizeranschi amizeranschi   4096 Nov 24 15:36 bwa
drwxrwxr-x 3 amizeranschi amizeranschi   4096 Nov 24 15:36 config
drwxrwxr-x 3 amizeranschi amizeranschi   4096 Nov 24 15:36 coverage
-rw-rw-r-- 1 amizeranschi amizeranschi 145031 Nov 24 15:36 hisat2-wf.tar.gz
drwxrwxr-x 2 amizeranschi amizeranschi   4096 Nov 24 15:36 rnaseq
-rw-rw-r-- 1 amizeranschi amizeranschi  15820 Nov 24 15:36 rtg--hg19.sdf-wf.tar.gz
drwxrwxr-x 2 amizeranschi amizeranschi   4096 Nov 24 15:36 seq
-rw-rw-r-- 1 amizeranschi amizeranschi   8174 Nov 24 15:36 snpeff--hg19-wf.tar.gz
drwxrwxr-x 2 amizeranschi amizeranschi   4096 Nov 24 15:36 ucsc
drwxrwxr-x 2 amizeranschi amizeranschi   4096 Nov 24 15:36 variation

Second, the Toil test, which I'm running from the latest GitHub clone:

git clone https://github.com/bcbio/test_bcbio_cwl.git
cd test_bcbio_cwl/somatic
bcbio_vm.py cwlrun toil somatic-workflow

And its outcome:

[...]
amizeranschi-Predator-PH317-51 2018-11-24 15:01:33,097 MainThread WARNING toil.leader: 6/a/jobBneVJu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
2018-11-24 15:01:33,097 - toil.leader - WARNING - 6/a/jobBneVJu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
amizeranschi-Predator-PH317-51 2018-11-24 15:01:33,097 MainThread WARNING toil.leader: 6/a/jobBneVJu        self._setup(kwargs)
2018-11-24 15:01:33,097 - toil.leader - WARNING - 6/a/jobBneVJu        self._setup(kwargs)
amizeranschi-Predator-PH317-51 2018-11-24 15:01:33,097 MainThread WARNING toil.leader: 6/a/jobBneVJu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
2018-11-24 15:01:33,097 - toil.leader - WARNING - 6/a/jobBneVJu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
amizeranschi-Predator-PH317-51 2018-11-24 15:01:33,097 MainThread WARNING toil.leader: 6/a/jobBneVJu        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
2018-11-24 15:01:33,097 - toil.leader - WARNING - 6/a/jobBneVJu        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
amizeranschi-Predator-PH317-51 2018-11-24 15:01:33,097 MainThread WARNING toil.leader: 6/a/jobBneVJu    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv) not found or is not a regular file.
2018-11-24 15:01:33,097 - toil.leader - WARNING - 6/a/jobBneVJu    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv) not found or is not a regular file.
amizeranschi-Predator-PH317-51 2018-11-24 15:01:33,097 MainThread WARNING toil.leader: 6/a/jobBneVJu    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
2018-11-24 15:01:33,097 - toil.leader - WARNING - 6/a/jobBneVJu    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
amizeranschi-Predator-PH317-51 2018-11-24 15:01:33,097 MainThread WARNING toil.leader: 6/a/jobBneVJu    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 6/a/jobBneVJu with ID 6/a/jobBneVJu to 0
2018-11-24 15:01:33,097 - toil.leader - WARNING - 6/a/jobBneVJu    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 6/a/jobBneVJu with ID 6/a/jobBneVJu to 0
amizeranschi-Predator-PH317-51 2018-11-24 15:01:33,098 MainThread WARNING toil.leader: Job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 6/a/jobBneVJu with ID 6/a/jobBneVJu is completely failed
2018-11-24 15:01:33,098 - toil.leader - WARNING - Job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 6/a/jobBneVJu with ID 6/a/jobBneVJu is completely failed
amizeranschi-Predator-PH317-51 2018-11-24 15:02:03,126 MainThread INFO toil.leader: No jobs left to run so exiting.
2018-11-24 15:02:03,126 - toil.leader - INFO - No jobs left to run so exiting.
amizeranschi-Predator-PH317-51 2018-11-24 15:02:03,126 MainThread INFO toil.leader: Finished the main loop
2018-11-24 15:02:03,126 - toil.leader - INFO - Finished the main loop
amizeranschi-Predator-PH317-51 2018-11-24 15:02:03,127 MainThread INFO toil.serviceManager: Waiting for service manager thread to finish ...
2018-11-24 15:02:03,127 - toil.serviceManager - INFO - Waiting for service manager thread to finish ...
amizeranschi-Predator-PH317-51 2018-11-24 15:02:03,656 MainThread INFO toil.serviceManager: ... finished shutting down the service manager. Took 0.529287099838 seconds
2018-11-24 15:02:03,656 - toil.serviceManager - INFO - ... finished shutting down the service manager. Took 0.529287099838 seconds
amizeranschi-Predator-PH317-51 2018-11-24 15:02:03,657 MainThread INFO toil.statsAndLogging: Waiting for stats and logging collator thread to finish ...
2018-11-24 15:02:03,657 - toil.statsAndLogging - INFO - Waiting for stats and logging collator thread to finish ...
amizeranschi-Predator-PH317-51 2018-11-24 15:02:04,095 MainThread INFO toil.statsAndLogging: ... finished collating stats and logs. Took 0.438358783722 seconds
2018-11-24 15:02:04,095 - toil.statsAndLogging - INFO - ... finished collating stats and logs. Took 0.438358783722 seconds
amizeranschi-Predator-PH317-51 2018-11-24 15:02:04,096 MainThread INFO toil.leader: Finished toil run with 18 failed jobs
2018-11-24 15:02:04,096 - toil.leader - INFO - Finished toil run with 18 failed jobs
amizeranschi-Predator-PH317-51 2018-11-24 15:02:04,096 MainThread INFO toil.leader: Failed jobs at end of the run: 'CWLScatter' B/9/jobYDwmJB 'CWLGather' J/x/jobsUEQFX 'CWLWorkflow' f/j/job4t6Ajf 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/summarize_vc.cwl' bcbio_nextgen.py runfn summarize_vc cwl Q/t/jobrMuXha 'CWLGather' d/S/jobf875Ag 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/postprocess_alignment_to_rec.cwl' bcbio_nextgen.py runfn postprocess_alignment_to_rec cwl G/5/jobCBppfQ 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl d/O/jobQP68VV 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl 9/q/job4J49ZC 'CWLGather' t/g/job6kwmee 'CWLGather' m/K/joberSfJx 'CWLScatter' U/V/jobEKpjLO 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 6/a/jobBneVJu 'CWLScatter' S/5/jobCXRZfK 'CWLScatter' K/5/jobV6dbdl 'CWLGather' m/o/jobeGktCV 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/batch_for_variantcall.cwl' bcbio_nextgen.py runfn batch_for_variantcall cwl r/J/joboU0oR4 'CWLScatter' X/W/jobxv2o6L 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/combine_sample_regions.cwl' bcbio_nextgen.py runfn combine_sample_regions cwl 6/A/job0uIlK9
2018-11-24 15:02:04,096 - toil.leader - INFO - Failed jobs at end of the run: 'CWLScatter' B/9/jobYDwmJB 'CWLGather' J/x/jobsUEQFX 'CWLWorkflow' f/j/job4t6Ajf 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/summarize_vc.cwl' bcbio_nextgen.py runfn summarize_vc cwl Q/t/jobrMuXha 'CWLGather' d/S/jobf875Ag 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/postprocess_alignment_to_rec.cwl' bcbio_nextgen.py runfn postprocess_alignment_to_rec cwl G/5/jobCBppfQ 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl d/O/jobQP68VV 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl 9/q/job4J49ZC 'CWLGather' t/g/job6kwmee 'CWLGather' m/K/joberSfJx 'CWLScatter' U/V/jobEKpjLO 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 6/a/jobBneVJu 'CWLScatter' S/5/jobCXRZfK 'CWLScatter' K/5/jobV6dbdl 'CWLGather' m/o/jobeGktCV 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/batch_for_variantcall.cwl' bcbio_nextgen.py runfn batch_for_variantcall cwl r/J/joboU0oR4 'CWLScatter' X/W/jobxv2o6L 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/combine_sample_regions.cwl' bcbio_nextgen.py runfn combine_sample_regions cwl 6/A/job0uIlK9
Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/cwltoil", line 11, in <module>
    load_entry_point('toil==3.14.0', 'console_scripts', 'cwltoil')()
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 990, in main
    outobj = toil.start(wf1)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/common.py", line 743, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/common.py", line 1018, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/leader.py", line 226, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'file:/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/toil_work/cwltoil_jobstore' contains 18 failed jobs: 'CWLScatter' B/9/jobYDwmJB, 'CWLGather' J/x/jobsUEQFX, 'CWLWorkflow' f/j/job4t6Ajf, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/summarize_vc.cwl' bcbio_nextgen.py runfn summarize_vc cwl Q/t/jobrMuXha, 'CWLGather' d/S/jobf875Ag, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/postprocess_alignment_to_rec.cwl' bcbio_nextgen.py runfn postprocess_alignment_to_rec cwl G/5/jobCBppfQ, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl d/O/jobQP68VV, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl 9/q/job4J49ZC, 'CWLGather' t/g/job6kwmee, 'CWLGather' m/K/joberSfJx, 'CWLScatter' U/V/jobEKpjLO, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 6/a/jobBneVJu, 'CWLScatter' S/5/jobCXRZfK, 'CWLScatter' K/5/jobV6dbdl, 'CWLGather' m/o/jobeGktCV, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/batch_for_variantcall.cwl' bcbio_nextgen.py runfn batch_for_variantcall cwl r/J/joboU0oR4, 'CWLScatter' X/W/jobxv2o6L, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/combine_sample_regions.cwl' bcbio_nextgen.py runfn combine_sample_regions cwl 6/A/job0uIlK9
=========> Failed job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 6/a/jobBneVJu 
INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
WARNING:toil.fileStore:Starting job f/j/job4t6Ajf/g/tmp1M_2KE.tmp with less than 10% of disk space remaining.
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
    return self.run(fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
    (output, status) = cwltool.main.single_job_executor(**opts)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
    r.run(**kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
    self._setup(kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
    "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv) not found or is not a regular file.
ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 6/a/jobBneVJu with ID 6/a/jobBneVJu to 0
<=========

Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 294, in run
    _TOOLS[args.tool](args)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 121, in _run_toil
    _run_tool(cmd, not args.no_container, work_dir)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 42, in _run_tool
    subprocess.check_call(cmd, shell=True)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'export PATH=/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin:$PATH && cwltoil --jobStore /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/toil_work/cwltoil_jobstore --logFile /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/toil_work/somatic-toil.log --workDir /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/toil_work/tmpdir --linkImports -- /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/main-somatic.cwl /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/main-somatic-samples.json' returned non-zero exit status 1

This seems to be related to the problem above (missing /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv).

Update: I created a versions.csv file with the following content (copied from the GRCh37 genome install):

seq,broad-20120813
twobit,broad-20120813
GA4GH_problem_regions,20181016
capture_regions,20161202
MIG,20150730
prioritize,20160215

but this didn't get the tests to run.

In the case of Cromwell:

[...]
[2018-11-24 18:25:29,01] [info] BackgroundConfigAsyncJobExecutionActor [8d7f1694prep_samples:0:1]: 'bcbio_nextgen.py' 'runfn' 'prep_samples' 'cwl' 'sentinel_runtime=cores,1,ram,2048' 'sentinel_parallel=multi-parallel' 'sentinel_outputs=rgnames__sample,config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready' 'sentinel_inputs=prep_samples_rec:record' 'run_number=0'
[2018-11-24 18:25:29,02] [info] BackgroundConfigAsyncJobExecutionActor [8d7f1694prep_samples:0:1]: executing: # make sure there is no preexisting Docker CID file
rm -f /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0/execution/docker_cid
# run as in the original configuration without --rm flag (will remove later)
docker run \
  --cidfile /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0/execution/docker_cid \
  -i \
   \
  --entrypoint /bin/bash \
  -v /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0:/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0 \
  quay.io/bcbio/bcbio-vc@sha256:7748650d81ab0b549172846b467f0281c15e523dbf977b2731bda20fad284711 /cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0/execution/script

# get the return code (working even if the container was detached)
rc=$(docker wait cat /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0/execution/docker_cid)

# remove the container after waiting
docker rm cat /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0/execution/docker_cid

# return exit code
exit $rc
[2018-11-24 18:25:32,54] [info] BackgroundConfigAsyncJobExecutionActor [8d7f1694prep_samples:0:1]: job id: 462
[2018-11-24 18:25:32,54] [info] BackgroundConfigAsyncJobExecutionActor [8d7f1694prep_samples:0:1]: Status change from - to WaitingForReturnCodeFile
[2018-11-24 18:25:35,64] [info] BackgroundConfigAsyncJobExecutionActor [8d7f1694prep_samples:0:1]: Status change from WaitingForReturnCodeFile to Done
[2018-11-24 18:25:37,96] [error] WorkflowManagerActor Workflow 8d7f1694-e3c1-441e-ada8-5ac4c5951bc0 failed (during ExecutingWorkflowState): cromwell.core.CromwellFatalException: java.lang.Exception: Failed command instantiation
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:46)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:37)
    at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:413)
    at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.Exception: Failed command instantiation
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
    at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
    at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
    ... 6 more
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
    :
    Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-alignment_to_rec/inputs/66456483/6_100326_FC6107FAAXX.bam.bai:
    /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-alignment_to_rec/inputs/66456483/6_100326_FC6107FAAXX.bam.bai -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai
    at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
    at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
    ... 35 more

[2018-11-24 18:25:37,96] [info] WorkflowManagerActor WorkflowActor-8d7f1694-e3c1-441e-ada8-5ac4c5951bc0 is in a terminal state: WorkflowFailedState
[2018-11-24 18:26:11,37] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2018-11-24 18:26:12,80] [info] SingleWorkflowRunnerActor writing metadata to /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/somatic-metadata.json
[2018-11-24 18:26:12,84] [info] Workflow polling stopped
[2018-11-24 18:26:12,85] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2018-11-24 18:26:12,85] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2018-11-24 18:26:12,86] [info] Aborting all running workflows.
[2018-11-24 18:26:12,86] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2018-11-24 18:26:12,86] [info] JobExecutionTokenDispenser stopped
[2018-11-24 18:26:12,86] [info] WorkflowLogCopyRouter stopped
[2018-11-24 18:26:12,86] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2018-11-24 18:26:12,86] [info] WorkflowStoreActor stopped
[2018-11-24 18:26:12,86] [info] WorkflowManagerActor All workflows finished
[2018-11-24 18:26:12,86] [info] Connection pools shut down
[2018-11-24 18:26:12,86] [info] WorkflowManagerActor stopped
[2018-11-24 18:26:12,86] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2018-11-24 18:26:12,86] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2018-11-24 18:26:12,86] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2018-11-24 18:26:12,86] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2018-11-24 18:26:12,86] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2018-11-24 18:26:12,86] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2018-11-24 18:26:12,86] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2018-11-24 18:26:12,86] [info] CallCacheWriteActor stopped
[2018-11-24 18:26:12,86] [info] JobStoreActor stopped
[2018-11-24 18:26:12,87] [info] SubWorkflowStoreActor stopped
[2018-11-24 18:26:12,87] [info] IoProxy stopped
[2018-11-24 18:26:12,87] [info] DockerHashActor stopped
[2018-11-24 18:26:12,87] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2018-11-24 18:26:12,87] [info] KvWriteActor Shutting down: 0 queued messages to process
[2018-11-24 18:26:12,87] [info] ServiceRegistryActor stopped
[2018-11-24 18:26:12,88] [info] Database closed
[2018-11-24 18:26:12,88] [info] Stream materializer shut down
[2018-11-24 18:26:12,89] [info] WDL HTTP import resolver closed
Workflow 8d7f1694-e3c1-441e-ada8-5ac4c5951bc0 transitioned to state Failed
Failed bcbio Cromwell run
-------------------------
Failure in step: alignment_to_rec.0
  bcbio log file     : cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-alignment_to_rec/execution/log/bcbio-nextgen-debug.log
  bcbio commands file: cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-alignment_to_rec/execution/log/bcbio-nextgen-commands.log
  Cromwell directory : cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-alignment_to_rec

And for Toil:

[...]
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu        returnValues = self._run(jobGraph, fileStore)
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu        returnValues = self._run(jobGraph, fileStore)
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu        return self.run(fileStore)
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu        return self.run(fileStore)
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu        (output, status) = cwltool.main.single_job_executor(**opts)
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu        (output, status) = cwltool.main.single_job_executor(**opts)
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu        r.run(**kwargs)
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu        r.run(**kwargs)
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu        self._setup(kwargs)
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu        self._setup(kwargs)
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,013 MainThread WARNING toil.leader: 8/u/jobZPBjPu        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
2018-11-24 18:32:06,013 - toil.leader - WARNING - 8/u/jobZPBjPu        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,013 MainThread WARNING toil.leader: 8/u/jobZPBjPu    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv) not found or is not a regular file.
2018-11-24 18:32:06,013 - toil.leader - WARNING - 8/u/jobZPBjPu    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv) not found or is not a regular file.
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,013 MainThread WARNING toil.leader: 8/u/jobZPBjPu    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
2018-11-24 18:32:06,013 - toil.leader - WARNING - 8/u/jobZPBjPu    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,013 MainThread WARNING toil.leader: 8/u/jobZPBjPu    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu with ID 8/u/jobZPBjPu to 0
2018-11-24 18:32:06,013 - toil.leader - WARNING - 8/u/jobZPBjPu    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu with ID 8/u/jobZPBjPu to 0
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,014 MainThread WARNING toil.leader: Job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu with ID 8/u/jobZPBjPu is completely failed
2018-11-24 18:32:06,014 - toil.leader - WARNING - Job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu with ID 8/u/jobZPBjPu is completely failed
amizeranschi-Predator-PH317-51 2018-11-24 18:32:36,052 MainThread INFO toil.leader: No jobs left to run so exiting.
2018-11-24 18:32:36,052 - toil.leader - INFO - No jobs left to run so exiting.
amizeranschi-Predator-PH317-51 2018-11-24 18:32:36,052 MainThread INFO toil.leader: Finished the main loop
2018-11-24 18:32:36,052 - toil.leader - INFO - Finished the main loop
amizeranschi-Predator-PH317-51 2018-11-24 18:32:36,053 MainThread INFO toil.serviceManager: Waiting for service manager thread to finish ...
2018-11-24 18:32:36,053 - toil.serviceManager - INFO - Waiting for service manager thread to finish ...
amizeranschi-Predator-PH317-51 2018-11-24 18:32:36,745 MainThread INFO toil.serviceManager: ... finished shutting down the service manager. Took 0.692306041718 seconds
2018-11-24 18:32:36,745 - toil.serviceManager - INFO - ... finished shutting down the service manager. Took 0.692306041718 seconds
amizeranschi-Predator-PH317-51 2018-11-24 18:32:36,746 MainThread INFO toil.statsAndLogging: Waiting for stats and logging collator thread to finish ...
2018-11-24 18:32:36,746 - toil.statsAndLogging - INFO - Waiting for stats and logging collator thread to finish ...
amizeranschi-Predator-PH317-51 2018-11-24 18:32:37,254 MainThread INFO toil.statsAndLogging: ... finished collating stats and logs. Took 0.508193016052 seconds
2018-11-24 18:32:37,254 - toil.statsAndLogging - INFO - ... finished collating stats and logs. Took 0.508193016052 seconds
amizeranschi-Predator-PH317-51 2018-11-24 18:32:37,255 MainThread INFO toil.leader: Finished toil run with 18 failed jobs
2018-11-24 18:32:37,255 - toil.leader - INFO - Finished toil run with 18 failed jobs
amizeranschi-Predator-PH317-51 2018-11-24 18:32:37,258 MainThread INFO toil.leader: Failed jobs at end of the run: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/combine_sample_regions.cwl' bcbio_nextgen.py runfn combine_sample_regions cwl 7/M/jobgvGM1D 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/postprocess_alignment_to_rec.cwl' bcbio_nextgen.py runfn postprocess_alignment_to_rec cwl c/L/jobQ2hf2H 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu 'CWLScatter' K/I/jobOfJUCd 'CWLGather' V/R/jobgdDUnB 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/batch_for_variantcall.cwl' bcbio_nextgen.py runfn batch_for_variantcall cwl S/g/jobPabqDO 'CWLScatter' K/K/jobAxKazj 'CWLScatter' i/X/jobRIQRN6 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl 8/i/job6Ha2qc 'CWLScatter' k/M/jobCNmWhn 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl n/3/jobuSm68l 'CWLScatter' v/6/jobDs9Lxc 'CWLGather' 3/p/jobWjBJTL 'CWLGather' V/l/jobyP87Bu 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/summarize_vc.cwl' bcbio_nextgen.py runfn summarize_vc cwl r/U/jobLYCUiV 'CWLGather' m/P/jobd1Emh6 'CWLGather' m/H/jobwpmje7 'CWLWorkflow' 7/8/jobQHebBQ
2018-11-24 18:32:37,258 - toil.leader - INFO - Failed jobs at end of the run: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/combine_sample_regions.cwl' bcbio_nextgen.py runfn combine_sample_regions cwl 7/M/jobgvGM1D 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/postprocess_alignment_to_rec.cwl' bcbio_nextgen.py runfn postprocess_alignment_to_rec cwl c/L/jobQ2hf2H 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu 'CWLScatter' K/I/jobOfJUCd 'CWLGather' V/R/jobgdDUnB 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/batch_for_variantcall.cwl' bcbio_nextgen.py runfn batch_for_variantcall cwl S/g/jobPabqDO 'CWLScatter' K/K/jobAxKazj 'CWLScatter' i/X/jobRIQRN6 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl 8/i/job6Ha2qc 'CWLScatter' k/M/jobCNmWhn 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl n/3/jobuSm68l 'CWLScatter' v/6/jobDs9Lxc 'CWLGather' 3/p/jobWjBJTL 'CWLGather' V/l/jobyP87Bu 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/summarize_vc.cwl' bcbio_nextgen.py runfn summarize_vc cwl r/U/jobLYCUiV 'CWLGather' m/P/jobd1Emh6 'CWLGather' m/H/jobwpmje7 'CWLWorkflow' 7/8/jobQHebBQ
Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/cwltoil", line 11, in <module>
    load_entry_point('toil==3.14.0', 'console_scripts', 'cwltoil')()
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 990, in main
    outobj = toil.start(wf1)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/common.py", line 743, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/common.py", line 1018, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/leader.py", line 226, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'file:/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/toil_work/cwltoil_jobstore' contains 18 failed jobs: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/combine_sample_regions.cwl' bcbio_nextgen.py runfn combine_sample_regions cwl 7/M/jobgvGM1D, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/postprocess_alignment_to_rec.cwl' bcbio_nextgen.py runfn postprocess_alignment_to_rec cwl c/L/jobQ2hf2H, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu, 'CWLScatter' K/I/jobOfJUCd, 'CWLGather' V/R/jobgdDUnB, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/batch_for_variantcall.cwl' bcbio_nextgen.py runfn batch_for_variantcall cwl S/g/jobPabqDO, 'CWLScatter' K/K/jobAxKazj, 'CWLScatter' i/X/jobRIQRN6, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl 8/i/job6Ha2qc, 'CWLScatter' k/M/jobCNmWhn, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl n/3/jobuSm68l, 'CWLScatter' v/6/jobDs9Lxc, 'CWLGather' 3/p/jobWjBJTL, 'CWLGather' V/l/jobyP87Bu, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/summarize_vc.cwl' bcbio_nextgen.py runfn summarize_vc cwl r/U/jobLYCUiV, 'CWLGather' m/P/jobd1Emh6, 'CWLGather' m/H/jobwpmje7, 'CWLWorkflow' 7/8/jobQHebBQ
=========> Failed job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu 
INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
    return self.run(fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
    (output, status) = cwltool.main.single_job_executor(**opts)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
    r.run(**kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
    self._setup(kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
    "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv) not found or is not a regular file.
ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu with ID 8/u/jobZPBjPu to 0
<=========

Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 294, in run
    _TOOLS[args.tool](args)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 121, in _run_toil
    _run_tool(cmd, not args.no_container, work_dir)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 42, in _run_tool
    subprocess.check_call(cmd, shell=True)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'export PATH=/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin:$PATH && cwltoil --jobStore /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/toil_work/cwltoil_jobstore --logFile /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/toil_work/somatic-toil.log --workDir /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/toil_work/tmpdir --linkImports -- /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/main-somatic.cwl /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/main-somatic-samples.json' returned non-zero exit status 1

Thanks for the help debugging and sorry about the issues. You're exactly right about the problem, I failed to check in the versions.csv file in the data directories. Apologies, I just added this now so things should work if you re-try. I'd suggest removing the existing work directories:

rm -rf cromwell_work toil_work

since I think the errors you're seeing are coming from cached failed runs (missing .bai files for Cromwell and missing versions.csv for toil) which should both be fixed. Hopefully this gets everything running, sorry again about all the back and forth.

Thanks. The two CWL tests are running fine now.

I've also tested on yeast data. I ran into a problem with Toil where it tried to use more CPU cores (16) than what my system actually has available (8). This was due to the default setting (cores: 16) in bcbio_system.yaml.

I remember that the bcbio_nextgen installer automatically detected the CPU cores and available memory and set these up in the system yaml file. Is there a way to do this for bcbio_vm as well?

I tried adding the option -n $cores to bcbio_vm.py cwlrun, but it doesn't seem like this option is supported anymore.

After setting the right number cores (and amount of memory) in bcbio_system.yaml, the Toil run is failing due to problems with one of the intermediary VCF files:

Traceback (most recent call last):
  File "/usr/local/bin/bcbio_nextgen.py", line 227, in <module>
    runfn.process(kwargs["args"])
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/runfn.py", line 58, in process
    out = fn(fnargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 52, in wrapper
    return apply(f, *args, **kwargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 392, in run_jointvc
    return joint.run_jointvc(*args)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 65, in run_jointvc
    joint_out = square_batch_region(data, ready_region, [], [d["vrn_file"] for d in items], out_file)[0]
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 186, in square_batch_region
    merge_gvcfs(data, region, vrn_files, out_file)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 251, in merge_gvcfs
    vcfutils.merge_variant_files([_extract_variants_from_gvcf(f, region, out_file, data) for f in vrn_files],
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 264, in _extract_variants_from_gvcf
    do.run(cmd.format(**locals()), "Extract variants from gVCF %s %s" % (dd.get_sample_name(data), region))
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 23, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 103, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'set -o pipefail; bcftools view -r chrXVI:1-608222 /var/lib/cwl/stg99153a51-3eb9-4b80-b5df-e6c8e87189e8/testingVC.vcf.gz | extract_variants | bgzip -c > /var/spool/cwl/bcbiotx/tmpe2YbZz/testingVC-chrXVI_1-608222-varonly.vcf.gz
ERROR: unexpected number of fields in vcf record:
    line_no: 71
    line: chrXVI    7631    .   T   G   1   LowGQX;LowDepth;NoPassedVariantGTs  SNVHPOL=3;MQ=27 GT:GQ:GQX:DP:DPF:AD:ADF:ADR:SB:FT:PL:AF .:.:.:0:0:0,0:0,0:0,0:0:LowGQX;LowDepth:.:0,0   0/1:22:1:2:0:1,1:0,1:1,0:0:LowGQX;LowDepth:27,0,27:0,0.5
' returned non-zero exit status 1
[job run_jointvc.cwl] Job error:
Error validating output record. * the `vrn_file_region` field is not valid because
    value is a list, expected File or null

It seems that the problem is with handling the multi-sample VCF file. Recall, I am testing joint variant calling using Strelka2 on two samples, using the YAML file posted above.

I've also tested Cromwell. Interestingly, it didn't fail due to the default CPU-core and memory settings from the bcbio_system.yaml. Does Cromwell automatically adjust to the system settings and ignore the values from bcbio_system.yaml?

However, Cromwell did run into the same VCF-related problems as Toil:

Check the content of stderr for potential additional information: /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/39c3beb2-47f0-4fb1-959a-0de7cfaec61d/call-jointcall/shard-0/wf-jointcall.cwl/808e4e98-9688-4256-a0a7-9727322452cb/call-run_jointvc/shard-24/execution/stderr.
 Traceback (most recent call last):
  File "/usr/local/bin/bcbio_nextgen.py", line 227, in <module>
    runfn.process(kwargs["args"])
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/runfn.py", line 58, in process
    out = fn(fnargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 52, in wrapper
    return apply(f, *args, **kwargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 392, in run_jointvc
    return joint.run_jointvc(*args)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 65, in run_jointvc
    joint_out = square_batch_region(data, ready_region, [], [d["vrn_file"] for d in items], out_file)[0]
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 186, in square_batch_region
    merge_gvcfs(data, region, vrn_files, out_file)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 251, in merge_gvcfs
    vcfutils.merge_variant_files([_extract_variants_from_gvcf(f, region, out_file, data) for f in vrn_files],
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 264, in _extract_variants_from_gvcf
    do.run(cmd.format(**locals()), "Extract variants from gVCF %s %s" % (dd.get_sample_name(data), region))
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 23, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 103, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'set -o pipefail; bcftools view -r chrXVI:1-608222 /cromwell-executions/main-testingVC-merged.cwl/39c3beb2-47f0-4fb1-959a-0de7cfaec61d/call-jointcall/shard-0/wf-jointcall.cwl/808e4e98-9688-4256-a0a7-9727322452cb/call-run_jointvc/shard-24/inputs/1823474195/testingVC.vcf.gz | extract_variants | bgzip -c > /cromwell-executions/main-testingVC-merged.cwl/39c3beb2-47f0-4fb1-959a-0de7cfaec61d/call-jointcall/shard-0/wf-jointcall.cwl/808e4e98-9688-4256-a0a7-9727322452cb/call-run_jointvc/shard-24/execution/bcbiotx/tmpQ0z6Ph/testingVC-chrXVI_1-608222-varonly.vcf.gz
ERROR: unexpected number of fields in vcf record:
    line_no: 71
    line: chrXVI    7631    .   T   G   1   LowGQX;LowDepth;NoPassedVariantGTs  SNVHPOL=3;MQ=27 GT:GQ:GQX:DP:DPF:AD:ADF:ADR:SB:FT:PL:AF .:.:.:0:0:0,0:0,0:0,0:0:LowGQX;LowDepth:.:0,0   0/1:22:1:2:0:1,1:0,1:1,0:0:LowGQX;LowDepth:27,0,27:0,0.5
' returned non-zero exit status 1

I noticed that there's a CWL test called gvcf_joint on GitHub (https://github.com/bcbio/test_bcbio_cwl.git). I gave this a try, but it ended up looking for input files in /home/chapmanb/drive/work/cwl:

INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
WARNING:toil.fileStore:Starting job h/T/job2oIVgk/g/tmp88jBdX.tmp with less than 10% of disk space remaining.
Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
    return self.run(fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
    (output, status) = cwltool.main.single_job_executor(**opts)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
    r.run(**kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
    self._setup(kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
    "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
WorkflowException: Input file file:///home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa.fai (at /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa.fai) not found or is not a regular file.

Glad we're making progress and thank you for this detailed feedback and testing on joint calling with your workflow. I believe we're running into an issue with gvcftools during strelka2 joint calling: sequencing/gvcftools#9 Practically, strelka2 joint calling was pretty primitive and outdated and I've moved it to use the updated gvcfgenotyper (https://github.com/Illumina/gvcfgenotyper) approach the Illumina team has been developing. If you re-try the runs with the latest Docker container (1.1.2a-1ffac4b: https://quay.io/repository/bcbio/bcbio-vc?tab=tags) I hope it will work cleanly for you. Also, going forward all new Docker builds will pin to the version of bcbio-base so you'll avoid the issues you saw was pinning versions in CWL and getting bcbio-base:latest.

Finally, I updated the gvcf_joint test which is working cleanly for me now with Cromwell. I'm not sure what is happening with toil in your example as the missing file it's complaining about appears to be present in the repository. I hope the updated version will work cleanly for you.

Thanks again for all this work and debugging and hope this now runs cleanly for you.

Hi, thanks a lot for the update. I'm afraid I don't quite understand what you meant by "all new Docker builds will pin to the version of bcbio-base so you'll avoid the issues you saw was pinning versions in CWL and getting bcbio-base:latest". Will the bcbio-base:latest image still be needed from now on, or will it suffice to have e.g. bcbio_vc:1.1.2a-1ffac4b installed?

I removed all the Docker images from my system (both bcbio_vc and bcbio_base) and got the latest CWL tests from Git. The Cromwell test failed due to wrong input file paths, and then proceeded to download the Docker image bcbio-base:latest:

cromwell.core.CromwellFatalException: java.lang.Exception: Failed command instantiation
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:46)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:37)
    at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:413)
    at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.Exception: Failed command instantiation
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
    at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
    at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
    at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
    ... 6 more
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
    :
    Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1635740323/7_100326_FC6107FAAXX.bam:
    /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1635740323/7_100326_FC6107FAAXX.bam -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam
    File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam
    :
    Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1635740323/6_100326_FC6107FAAXX.bam:
    /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1635740323/6_100326_FC6107FAAXX.bam -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam
    File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam
:
    Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222581580/hg19.fa.amb:
    /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222581580/hg19.fa.amb -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb
    File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb
    Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222581580/hg19.fa.amb:
    /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222581580/hg19.fa.amb -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb
    File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb
:
    Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1471069787/variant_regions-bam.bed:
    /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1471069787/variant_regions-bam.bed -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed
    File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed
    Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1471069787/variant_regions-bam.bed:
    /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1471069787/variant_regions-bam.bed -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed
    File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed
:
    Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222597375/hg19.fa:
    /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222597375/hg19.fa -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa
    File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa
    Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222597375/hg19.fa:
    /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa doesn't exist
    File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222597375/hg19.fa -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa
    File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa
    at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
    at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
    at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
    ... 35 more

[2018-11-26 20:14:19,15] [info] WorkflowManagerActor WorkflowActor-0ca4b313-c127-45ac-ac0b-820d43943dcf is in a terminal state: WorkflowFailedState
[2018-11-26 20:14:32,93] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2018-11-26 20:14:36,09] [info] SingleWorkflowRunnerActor writing metadata to /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/gvcf-joint-metadata.json
[2018-11-26 20:14:36,12] [info] Workflow polling stopped
[2018-11-26 20:14:36,13] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2018-11-26 20:14:36,13] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2018-11-26 20:14:36,14] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2018-11-26 20:14:36,14] [info] Aborting all running workflows.
[2018-11-26 20:14:36,14] [info] JobExecutionTokenDispenser stopped
[2018-11-26 20:14:36,14] [info] WorkflowStoreActor stopped
[2018-11-26 20:14:36,14] [info] WorkflowLogCopyRouter stopped
[2018-11-26 20:14:36,14] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2018-11-26 20:14:36,14] [info] WorkflowManagerActor All workflows finished
[2018-11-26 20:14:36,14] [info] WorkflowManagerActor stopped
[2018-11-26 20:14:36,14] [info] Connection pools shut down
[2018-11-26 20:14:36,14] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2018-11-26 20:14:36,14] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2018-11-26 20:14:36,14] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2018-11-26 20:14:36,14] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2018-11-26 20:14:36,14] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2018-11-26 20:14:36,14] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2018-11-26 20:14:36,14] [info] SubWorkflowStoreActor stopped
[2018-11-26 20:14:36,14] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2018-11-26 20:14:36,14] [info] JobStoreActor stopped
[2018-11-26 20:14:36,14] [info] CallCacheWriteActor stopped
[2018-11-26 20:14:36,14] [info] DockerHashActor stopped
[2018-11-26 20:14:36,14] [info] IoProxy stopped
[2018-11-26 20:14:36,15] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2018-11-26 20:14:36,15] [info] KvWriteActor Shutting down: 0 queued messages to process
[2018-11-26 20:14:36,15] [info] ServiceRegistryActor stopped
[2018-11-26 20:14:36,16] [info] Database closed
[2018-11-26 20:14:36,16] [info] Stream materializer shut down
[2018-11-26 20:14:36,16] [info] WDL HTTP import resolver closed
Workflow 0ca4b313-c127-45ac-ac0b-820d43943dcf transitioned to state Failed
Unable to find image 'quay.io/bcbio/bcbio-base:latest' locally
latest: Pulling from bcbio/bcbio-base
7b8b6451c85f: Pull complete 
ab4d1096d9ba: Pull complete 
e6797d1788ac: Pull complete 
e25c5c290bde: Pull complete 
406293dca006: Pull complete 
Digest: sha256:c397057e303fb5018091149e1c3c4167b8f6847e667e82d6e745a0a95d35093c
Status: Downloaded newer image for quay.io/bcbio/bcbio-base:latest
Failed bcbio Cromwell run
-------------------------
Failure in step: alignment_to_rec.0
  bcbio log file     : cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/execution/log/bcbio-nextgen-debug.log
  bcbio commands file: cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/execution/log/bcbio-nextgen-commands.log
  Cromwell directory : cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec

Failure in step: prep_samples_to_rec.0
  bcbio log file     : cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-prep_samples_to_rec/execution/log/bcbio-nextgen-debug.log
  bcbio commands file: cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-prep_samples_to_rec/execution/log/bcbio-nextgen-commands.log
  Cromwell directory : cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-prep_samples_to_rec

I then removed the bcbio-base:latest image and tried my tests using yeast data, running bcbio_vm.py cwl with the option --add-container-tag 1.1.2a-1ffac4b.

This ended up downloading both the bcbio-vc (with unknown tag) and bcbio-base images:

docker images -a
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
quay.io/bcbio/bcbio-vc     <none>              9c2199cf1749        2 hours ago         7.89GB
quay.io/bcbio/bcbio-base   latest              bdeb18f81e1b        3 hours ago         1.87GB
hello-world                latest              4ab4c602aa5e        2 months ago        1.84kB

Unfortunately, my test also failed. It seems that gvcfgenotyper isn't working properly:

*** Error in gvcfgenotyper': realloc(): invalid pointer: 0x00007fff6b0de6d8 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7ff4240977e5]
/lib/x86_64-linux-gnu/libc.so.6(realloc+0x348)[0x7ff4240a4a08]
gvcfgenotyper[0x46ec81]
gvcfgenotyper[0x41bdbe]
gvcfgenotyper[0x4253c3]
gvcfgenotyper[0x42574d]
gvcfgenotyper[0x425c76]
gvcfgenotyper[0x422e48]
gvcfgenotyper[0x4068c5]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ff424040830]
gvcfgenotyper[0x4099a1]
======= Memory map: ========
00400000-004f6000 r-xp 00000000 fd:00 15473242                           /usr/local/share/bcbio-nextgen/anaconda/bin/gvcfgenotyper
006f5000-006f7000 rw-p 000f5000 fd:00 15473242                           /usr/local/share/bcbio-nextgen/anaconda/bin/gvcfgenotyper
006f7000-006f8000 rw-p 00000000 00:00 0 
01948000-019c1000 rw-p 00000000 00:00 0                                  [heap]
7ff420000000-7ff420021000 rw-p 00000000 00:00 0 
7ff420021000-7ff424000000 ---p 00000000 00:00 0 
7ff424020000-7ff4241e0000 r-xp 00000000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7ff4241e0000-7ff4243e0000 ---p 001c0000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7ff4243e0000-7ff4243e4000 r--p 001c0000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7ff4243e4000-7ff4243e6000 rw-p 001c4000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7ff4243e6000-7ff4243ea000 rw-p 00000000 00:00 0 
7ff4243ea000-7ff4244f2000 r-xp 00000000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7ff4244f2000-7ff4246f1000 ---p 00108000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7ff4246f1000-7ff4246f2000 r--p 00107000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7ff4246f2000-7ff4246f3000 rw-p 00108000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7ff4246f3000-7ff42470b000 r-xp 00000000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7ff42470b000-7ff42490a000 ---p 00018000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7ff42490a000-7ff42490b000 r--p 00017000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7ff42490b000-7ff42490c000 rw-p 00018000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7ff42490c000-7ff424910000 rw-p 00000000 00:00 0 
7ff424910000-7ff424926000 r-xp 00000000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7ff424926000-7ff424b25000 ---p 00016000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7ff424b25000-7ff424b26000 r--p 00015000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7ff424b26000-7ff424b27000 rw-p 00016000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7ff424b27000-7ff424b4d000 r-xp 00000000 fd:00 15343636                   /lib/x86_64-linux-gnu/ld-2.23.so
7ff424ba9000-7ff424bf0000 rw-p 00000000 00:00 0 
7ff424bf0000-7ff424bf3000 r--p 00000000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7ff424bf3000-7ff424c00000 r-xp 00003000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7ff424c00000-7ff424c03000 r--p 00010000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7ff424c03000-7ff424c04000 r--p 00012000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7ff424c04000-7ff424c05000 rw-p 00013000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7ff424c05000-7ff424c06000 rw-p 00000000 00:00 0 
7ff424c06000-7ff424c9a000 r--p 00000000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7ff424c9a000-7ff424cff000 r-xp 00094000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7ff424cff000-7ff424d36000 r--p 000f9000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7ff424d36000-7ff424d40000 r--p 0012f000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7ff424d40000-7ff424d44000 rw-p 00139000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7ff424d44000-7ff424d47000 rw-p 00000000 00:00 0 
7ff424d4a000-7ff424d4c000 rw-p 00000000 00:00 0 
7ff424d4c000-7ff424d4d000 r--p 00025000 fd:00 15343636                   /lib/x86_64-linux-gnu/ld-2.23.so
7ff424d4d000-7ff424d4e000 rw-p 00026000 fd:00 15343636                   /lib/x86_64-linux-gnu/ld-2.23.so
7ff424d4e000-7ff424d4f000 rw-p 00000000 00:00 0 
7fff6b0bf000-7fff6b0e0000 rw-p 00000000 00:00 0                          [stack]
7fff6b1c2000-7fff6b1c5000 r--p 00000000 00:00 0                          [vvar]
7fff6b1c5000-7fff6b1c7000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
' returned non-zero exit status -6

Job run_jointvc:9:1 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-9/execution/stderr.
 Traceback (most recent call last):
  File "/usr/local/bin/bcbio_nextgen.py", line 223, in <module>
    runfn.process(kwargs["args"])
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/runfn.py", line 58, in process
    out = fn(fnargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 52, in wrapper
    return apply(f, *args, **kwargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 396, in run_jointvc
    return joint.run_jointvc(*args)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 64, in run_jointvc
    joint_out = square_batch_region(data, ready_region, [], [d["vrn_file"] for d in items], out_file)[0]
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 185, in square_batch_region
    strelka2.run_gvcfgenotyper(data, region, vrn_files, out_file)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/strelka2.py", line 343, in run_gvcfgenotyper
    for r in regions]
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/strelka2.py", line 358, in _run_gvcfgenotyper
    do.run(cmd, "gvcfgenotyper: %s %s" % (dd.get_sample_name(data), region))
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 23, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 103, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'gvcfgenotyper -f /cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-9/inputs/-1642066049/sacCer3.fa -l /cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-9/execution/bcbiotx/tmpjPGnlh/testingVC-strelka2-chrVII_609522-1090940-chrVII_618649_618650-inputs.txt -r chrVII:618649-618650 -O z -o /cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-9/execution/bcbiotx/tmpjPGnlh/testingVC-strelka2-chrVII_609522-1090940-chrVII_618649_618650.vcf.gz
Logging output to gvcfgenotyper.20181126_185656.80.log
*** Error in gvcfgenotyper': realloc(): invalid pointer: 0x00007ffda7dc0b48 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fc4230937e5]
/lib/x86_64-linux-gnu/libc.so.6(realloc+0x348)[0x7fc4230a0a08]
gvcfgenotyper[0x46ec81]
gvcfgenotyper[0x41bdbe]
gvcfgenotyper[0x4253c3]
gvcfgenotyper[0x42574d]
gvcfgenotyper[0x425c76]
gvcfgenotyper[0x422e48]
gvcfgenotyper[0x4068c5]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fc42303c830]
gvcfgenotyper[0x4099a1]
======= Memory map: ========
00400000-004f6000 r-xp 00000000 fd:00 15473242                           /usr/local/share/bcbio-nextgen/anaconda/bin/gvcfgenotyper
006f5000-006f7000 rw-p 000f5000 fd:00 15473242                           /usr/local/share/bcbio-nextgen/anaconda/bin/gvcfgenotyper
006f7000-006f8000 rw-p 00000000 00:00 0 
01b3c000-01bb5000 rw-p 00000000 00:00 0                                  [heap]
7fc41c000000-7fc41c021000 rw-p 00000000 00:00 0 
7fc41c021000-7fc420000000 ---p 00000000 00:00 0 
7fc42301c000-7fc4231dc000 r-xp 00000000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7fc4231dc000-7fc4233dc000 ---p 001c0000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7fc4233dc000-7fc4233e0000 r--p 001c0000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7fc4233e0000-7fc4233e2000 rw-p 001c4000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7fc4233e2000-7fc4233e6000 rw-p 00000000 00:00 0 
7fc4233e6000-7fc4234ee000 r-xp 00000000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7fc4234ee000-7fc4236ed000 ---p 00108000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7fc4236ed000-7fc4236ee000 r--p 00107000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7fc4236ee000-7fc4236ef000 rw-p 00108000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7fc4236ef000-7fc423707000 r-xp 00000000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7fc423707000-7fc423906000 ---p 00018000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7fc423906000-7fc423907000 r--p 00017000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7fc423907000-7fc423908000 rw-p 00018000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7fc423908000-7fc42390c000 rw-p 00000000 00:00 0 
7fc42390c000-7fc423922000 r-xp 00000000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7fc423922000-7fc423b21000 ---p 00016000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7fc423b21000-7fc423b22000 r--p 00015000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7fc423b22000-7fc423b23000 rw-p 00016000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7fc423b23000-7fc423b49000 r-xp 00000000 fd:00 15343636                   /lib/x86_64-linux-gnu/ld-2.23.so
7fc423ba5000-7fc423bec000 rw-p 00000000 00:00 0 
7fc423bec000-7fc423bef000 r--p 00000000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7fc423bef000-7fc423bfc000 r-xp 00003000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7fc423bfc000-7fc423bff000 r--p 00010000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7fc423bff000-7fc423c00000 r--p 00012000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7fc423c00000-7fc423c01000 rw-p 00013000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7fc423c01000-7fc423c02000 rw-p 00000000 00:00 0 
7fc423c02000-7fc423c96000 r--p 00000000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7fc423c96000-7fc423cfb000 r-xp 00094000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7fc423cfb000-7fc423d32000 r--p 000f9000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7fc423d32000-7fc423d3c000 r--p 0012f000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7fc423d3c000-7fc423d40000 rw-p 00139000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7fc423d40000-7fc423d43000 rw-p 00000000 00:00 0 
7fc423d46000-7fc423d48000 rw-p 00000000 00:00 0 
7fc423d48000-7fc423d49000 r--p 00025000 fd:00 15343636                   /lib/x86_64-linux-gnu/ld-2.23.so
7fc423d49000-7fc423d4a000 rw-p 00026000 fd:00 15343636                   /lib/x86_64-linux-gnu/ld-2.23.so
7fc423d4a000-7fc423d4b000 rw-p 00000000 00:00 0 
7ffda7da2000-7ffda7dc3000 rw-p 00000000 00:00 0                          [stack]
7ffda7dd2000-7ffda7dd5000 r--p 00000000 00:00 0                          [vvar]
7ffda7dd5000-7ffda7dd7000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
' returned non-zero exit status -6

[2018-11-26 20:56:59,18] [info] WorkflowManagerActor WorkflowActor-20f07690-6792-4bf5-80af-52b937112b35 is in a terminal state: WorkflowFailedState
[2018-11-26 20:57:26,44] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2018-11-26 20:57:28,02] [info] SingleWorkflowRunnerActor writing metadata to /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/testingVC-merged-metadata.json
[2018-11-26 20:57:28,09] [info] Workflow polling stopped
[2018-11-26 20:57:28,10] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2018-11-26 20:57:28,10] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2018-11-26 20:57:28,10] [info] Aborting all running workflows.
[2018-11-26 20:57:28,10] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2018-11-26 20:57:28,10] [info] WorkflowStoreActor stopped
[2018-11-26 20:57:28,10] [info] JobExecutionTokenDispenser stopped
[2018-11-26 20:57:28,11] [info] WorkflowLogCopyRouter stopped
[2018-11-26 20:57:28,11] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2018-11-26 20:57:28,11] [info] WorkflowManagerActor All workflows finished
[2018-11-26 20:57:28,11] [info] WorkflowManagerActor stopped
[2018-11-26 20:57:28,11] [info] Connection pools shut down
[2018-11-26 20:57:28,11] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2018-11-26 20:57:28,11] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2018-11-26 20:57:28,11] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2018-11-26 20:57:28,11] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2018-11-26 20:57:28,11] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2018-11-26 20:57:28,11] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2018-11-26 20:57:28,11] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2018-11-26 20:57:28,11] [info] JobStoreActor stopped
[2018-11-26 20:57:28,11] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2018-11-26 20:57:28,11] [info] KvWriteActor Shutting down: 0 queued messages to process
[2018-11-26 20:57:28,11] [info] CallCacheWriteActor stopped
[2018-11-26 20:57:28,11] [info] DockerHashActor stopped
[2018-11-26 20:57:28,11] [info] IoProxy stopped
[2018-11-26 20:57:28,11] [info] ServiceRegistryActor stopped
[2018-11-26 20:57:28,11] [info] SubWorkflowStoreActor stopped
[2018-11-26 20:57:28,12] [info] Database closed
[2018-11-26 20:57:28,12] [info] Stream materializer shut down
[2018-11-26 20:57:28,13] [info] WDL HTTP import resolver closed
Workflow 20f07690-6792-4bf5-80af-52b937112b35 transitioned to state Failed
Unable to find image 'quay.io/bcbio/bcbio-base:latest' locally
latest: Pulling from bcbio/bcbio-base
7b8b6451c85f: Already exists
ab4d1096d9ba: Already exists
e6797d1788ac: Already exists
e25c5c290bde: Already exists
406293dca006: Already exists
Digest: sha256:c397057e303fb5018091149e1c3c4167b8f6847e667e82d6e745a0a95d35093c
Status: Downloaded newer image for quay.io/bcbio/bcbio-base:latest
Running: export PATH=/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin:$PATH && cromwell -Xms1g -Xmx3g run --type CWL -Dconfig.file=/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/bcbio-cromwell.conf --metadata-output /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/testingVC-merged-metadata.json --options /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/testingVC-merged-options.json --inputs /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/main-testingVC-merged-samples.json /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/main-testingVC-merged.cwl 2>&1 | tee -a /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/testingVC-merged-cromwell.log
Failed bcbio Cromwell run
-------------------------
Failure in step: jointcall.0.subWorkflowMetadata.calls.run_jointvc.0
  bcbio log file     : cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-0/execution/log/bcbio-nextgen-debug.log
  bcbio commands file: cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-0/execution/log/bcbio-nextgen-commands.log
  Cromwell directory : cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-0
[...]

Thanks much for the updates and continued testing. I'll try to tackle all 3 of the issue from your last message:

For the Docker images, bcbio-base is still required/expected, but if you pin to a specific version it should pull that tag rather than latest now. So you'll expect bcbio-base and bcbio-vc with a specific tag if you request when generating the CWL.
For the gVCF joint call tests, I pushed a fix which I think will resolve this. I hadn't put relative links for the input files so they were referencing the build on my local machine. You can update and re-run and it should now work cleanly for you.
For the gvcfgenotyper issues to fix your actual analysis, we'll have to try and put together a test case as it seems like something in your inputs is uniquely causing it to fail. If you're able to share the input gVCFs for that region that cause the failure, I'm happy to try and help to put together a bug report and try to get it working for you.

Hope this all helps.

The latest joint call tests are running fine on my system now.

However, regarding my failed tests with yeast data, the error I pasted above doesn't seem to be related to a specific genomic region. There are many similar error blocks, related to all of the analyzed genomic regions, as far as I could tell. I've attached the full output here: https://gist.githubusercontent.com/amizeranschi/48204a751510880b20211cc5e7f07a9d/raw/abb1d685e17a13c6302a4da7cfae3a7b3985aca4/gistfile1.txt.

Unfortunately, I couldn't find any of the gVCF files in the temporary directories mentioned in the gvcfgenotyper errors. All of the temporary directories have been automatically deleted:

ls -la /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-*/execution/bcbiotx/
/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-0/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:53 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:53 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:53 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-10/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-11/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-12/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-13/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-14/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:53 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:53 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:53 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-15/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:53 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:53 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:53 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-16/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-17/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-18/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-19/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-1/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-20/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-21/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-22/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-23/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-24/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-25/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-2/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-3/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-4/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-5/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:53 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:53 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:53 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-6/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:53 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:53 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:53 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-7/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-8/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-9/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

bcbio / bcbio-nextgen

Problems with logs and joint VCF file generation in latest dev build #2473