search

bcbio / bcbio-nextgen

Validated, scalable, community developed variant calling, RNA-seq and small RNA analysis
https://bcbio-nextgen.readthedocs.io
MIT License
986 stars 354 forks source link

Missing `ifne` in docker image quay.io/bcbio/bcbio-vc? #2516

Closed tmooney closed 6 years ago

tmooney commented 6 years ago

I generated CWL using bcbio_vm.py cwl and attempted to run the resulting workflow with cromwell. Things progressed until the pipeline tried to run varscan as seen here: https://github.com/bcbio/bcbio-nextgen/blob/a3473775db06540c10b5f20ddc2043b8cc99d1f8/bcbio/variation/varscan.py#L313

The job failed with a python backtrace in /usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py but the relevant lines from the log seem to be:

CalledProcessError: Command 'set -o pipefail; export PATH=/usr/local/share/bcbio-nextgen/anaconda/bin:$PATH &&  /usr/local/share/bcbio-nextgen/anaconda/bin/samtools mpileup -f /cromwell-executions/main-50729-template.cwl/72269b6d-33c9-4bd5-9fef-5fcd89044aa5/call-variantcall/shard-0/wf-variantcall.cwl/173c9de6-1c4a-42a9-9f3d-0152948639eb/call-variantcall_batch_region/shard-0/inputs/222393154/GRCh37.fa -d 1000 -L 1000 -l /cromwell-executions/main-50729-template.cwl/72269b6d-33c9-4bd5-9fef-5fcd89044aa5/call-variantcall/shard-0/wf-variantcall.cwl/173c9de6-1c4a-42a9-9f3d-0152948639eb/call-variantcall_batch_region/shard-0/execution/varscan/1/50729-1_0_155870419-regions.bed /cromwell-executions/main-50729-template.cwl/72269b6d-33c9-4bd5-9fef-5fcd89044aa5/call-variantcall/shard-0/wf-variantcall.cwl/173c9de6-1c4a-42a9-9f3d-0152948639eb/call-variantcall_batch_region/shard-0/inputs/-193058515/50729-sort.bam | { ifne grep -v -P '\t0\t\t$' || true; } | ifne varscan -Xms909m -Xmx3723m -Duser.language=en -Duser.country=US -XX:+UseSerialGC -Djava.io.tmpdir=/cromwell-executions/main-50729-template.cwl/72269b6d-33c9-4bd5-9fef-5fcd89044aa5/call-variantcall/shard-0/wf-variantcall.cwl/173c9de6-1c4a-42a9-9f3d-0152948639eb/call-variantcall_batch_region/shard-0/execution/bcbiotx/tmpmHGfd5 mpileup2cns --min-coverage 5 --p-value 0.98 --strand-filter 1 --vcf-sample-list /cromwell-executions/main-50729-template.cwl/72269b6d-33c9-4bd5-9fef-5fcd89044aa5/call-variantcall/shard-0/wf-variantcall.cwl/173c9de6-1c4a-42a9-9f3d-0152948639eb/call-variantcall_batch_region/shard-0/execution/bcbiotx/tmpBDTSNM/50729-1_0_155870419-sample_list.txt --min-var-freq 0.01 --output-vcf --variants | /usr/local/share/bcbio-nextgen/anaconda/bin/py -x 'bcbio.variation.vcfutils.add_contig_to_header(x, "/cromwell-executions/main-50729-template.cwl/72269b6d-33c9-4bd5-9fef-5fcd89044aa5/call-variantcall/shard-0/wf-variantcall.cwl/173c9de6-1c4a-42a9-9f3d-0152948639eb/call-variantcall_batch_region/shard-0/inputs/222393154/GRCh37.fa")' | /usr/local/share/bcbio-nextgen/anaconda/bin/py -x 'bcbio.variation.varscan.fix_varscan_output(x)' | awk -F$'\t' -v OFS='\t' '{if ($0 !~ /^#/) gsub(/[KMRYSWBVHDXkmryswbvhdx]/, "N", $4) } {print}' | awk -F$'\t' -v OFS='\t' '{if ($0 !~ /^#/) gsub(/[KMRYSWBVHDXkmryswbvhdx]/, "N", $5) } {print}' | ifne vcfuniqalleles > /cromwell-executions/main-50729-template.cwl/72269b6d-33c9-4bd5-9fef-5fcd89044aa5/call-variantcall/shard-0/wf-variantcall.cwl/173c9de6-1c4a-42a9-9f3d-0152948639eb/call-variantcall_batch_region/shard-0/execution/bcbiotx/tmpBDTSNM/50729-1_0_155870419.vcf
/bin/bash: ifne: command not found
/bin/bash: ifne: command not found
/bin/bash: ifne: command not found

I see ifne in my local bcbio install, but I could not find it using the docker image that the pipeline used, quay.io/bcbio/bcbio-vc@sha256:9b641533116fadb675365e5a8886e3f1db3194337edeeb8c2b66e3beb8c4b06a.

chapmanb commented 6 years ago

Thomas; Thanks so much for the detailed report and for testing this out. Apologies, the moreutils package was missing from the bcbio-vc docker build. I've added this in and re-built the image, so if you pull the latest from quay.io (https://quay.io/repository/bcbio/bcbio-vc?tab=tags) it'll now have this and hopefully work cleanly. Please let us know if you run into any other issues at all.