Newer bcftools does not work with the pipeline

[lbeltrame]

The reason is that -o is -O in current bcftools code.

[chapmanb]

Luca; Has there been a new release candidate for bcftools? I know this changed a bit ago in the code but I was trying to work off release candidates rather than the GitHub head. Thanks much.

[lbeltrame]

Has there been a new release candidate for bcftools? I know this changed a bit ago in the code but I was trying to work off release candidates rather

Not that I know, I'm basing off the latest github code. There's no need to adjust this until there's a new release (I didn't check).

[lbeltrame]

It looks like a new tag has been pushed today (rc6), so this applies.

[chapmanb]

Luca; You must be a mind reader -- great job predicting the new release candidate. I updated the bcftools and htslib build instructions and changed the pipeline to match the new naming schemes and option arguments in 0.2.0-rc6. Thanks again for the heads up on this.

[mjafin]

Hi guys, I'm getting this error when updating my developmental copy:

[localhost] local: /apps/bcbio-nextgen/latest-devel/rhel6-x64/bin/brew tap --repair
Pruned 0 dead formula
Warning: Could not tap chapmanb/cbl/htslib over homebrew/science/htslib
Warning: Could not tap chapmanb/cbl/samtools over homebrew/science/samtools
Tapped 246 formula
[localhost] local: /apps/bcbio-nextgen/latest-devel/rhel6-x64/bin/brew outdated
[localhost] local: /apps/bcbio-nextgen/latest-devel/rhel6-x64/bin/brew info cpanminus
[localhost] local: /apps/bcbio-nextgen/latest-devel/rhel6-x64/bin/brew info expat
[localhost] local: /apps/bcbio-nextgen/latest-devel/rhel6-x64/bin/brew list samtools | grep -c bcftools

Warning: local() encountered an error (return code 1) while executing '/apps/bcbio-nextgen/latest-devel/rhel6-x64/bin/brew list samtools | grep -c bcftools'

[localhost] local: /apps/bcbio-nextgen/latest-devel/rhel6-x64/bin/cpanm -i --notest --local-lib=/apps/bcbio-nextgen/latest-devel/rhel6-x64 'Statistics::Descriptive'
Statistics::Descriptive is up to date. (3.0607)
[localhost] local: export CC=${CC:-`which gcc`} && export CXX=${CXX:-`which g++`} && export PERL5LIB=/apps/bcbio-nextgen/latest-devel/rhel6-x64/lib/perl5:${PERL5LIB} && /apps/bcbio-nextgen/latest-devel/rhel6-x64/bin/brew upgrade --env=inherit bcftools
==> Upgrading 1 outdated package, with result:
bcftools 0.2.0-rc6
Error: You must `brew link htslib' before bcftools can be installed

Fatal error: local() encountered an error (return code 1) while executing 'export CC=${CC:-`which gcc`} && export CXX=${CXX:-`which g++`} && export PERL5LIB=/apps/bcbio-nextgen/latest-devel/rhel6-x64/lib/perl5:${PERL5LIB} && /apps/bcbio-nextgen/latest-devel/rhel6-x64/bin/brew upgrade --env=inherit bcftools'

Aborting.

Further:

[bcbio-devel@ukapdlnx115: ~ ]$ brew link htslib
Error: htslib has multiple installed versions

[chapmanb]

Miika; Ugh, sorry about the issue. It looks like brew is not happily updating the htslib dependency. I added some fixes to avoid this in the future. If you do:

rm -rf tmpbcbio-install
brew uninstall htslib
bcbio_nextgen.py upgrade --tools

It should hopefully clean it up and get you the latest versions. Apologies, using development bcftools inside brew is super tricky. I can't wait to just ship docker containers instead of trying to keep this all sorted. Let us know if you still run into issues.

[mjafin]

Thanks Brad!

[caddymob]

Perhaps again?

IPython.parallel.error.CompositeError: one or more exceptions from call to method: postprocess_variants
[0:apply]: CalledProcessError: Command 'set -o pipefail; /packages/bcbio/LATEST/tooldir/bin/bcftools filter -O z -T /scratch/jcorneveaux/CRDC0102/CRDC0102/work/bedprep/TruSeq_exome_targeted_regions_b37_padded.bed.gz --soft-filter 'FBQualDepth' -e '(AF[0] <= 0.5 && (DP < 4 || (DP < 13 && %QUAL < 10))) || (AF[0] > 0.5 && (DP < 4 && %QUAL < 50)) || (%QUAL < 486.0 && DP > 243 && AF[0] <= 0.5)' -m '+' /scratch/jcorneveaux/CRDC0102/CRDC0102/work/freebayes/CRCD0102-effects.vcf.gz > /scratch/jcorneveaux/CRDC0102/CRDC0102/work/freebayes/tx/tmpoKs8tt/CRCD0102-effects-filter.vcf.gz
filter: invalid option -- 'T'

About:   Apply fixed-threshold filters.
Usage:   bcftools filter [options] <in.bcf>|<in.vcf>|<in.vcf.gz> [region1 [...]]
Options:
    -e, --exclude <expr>          exclude sites for which the expression is true (e.g. '%TYPE="snp" && %QUAL>=10 && (DP4[2]+DP4[3] > 2')
    -g, --SnpGap <int>            filter SNPs within <int> base pairs of an indel
    -G, --IndelGap <int>          filter clusters of indels separated by <int> or fewer base pairs allowing only one to pass
    -i, --include <expr>          include only sites for which the expression is true
    -m, --mode <+|x>              "+": do not replace but add to existing FILTER; "x": reset filters at sites which pass
    -O, --output-type <b|u|z|v>   b: compressed BCF, u: uncompressed BCF, z: compressed VCF, v: uncompressed VCF [v]
    -r, --regions <reg|file>      restrict to comma-separated list of regions or regions listed in a file, see man page for details
    -s, --soft-filter <string>    annotate FILTER column with <string> or unique filter name ("Filter%d") made up by the program ("+")
    -t, --targets <reg|file>      similar to -r but streams rather than index-jumps, see man page for details

Filter expressions may contain:
    - arithmetic operators: +,*,-,/
    - logical operators: && (same as &), || (same as |)
    - comparison operators: == (same as =), >, >=, <=, <, !=
    - parentheses: (, )
    - array subscripts, such as (e.g. AC[0]>=10)
    - double quotes for string values (e.g. %FILTER="PASS")
    - 1 (or 0) for testing the presence (or absence) of a flag (e.g. FlagA=1 && FlagB=0)
    - TAG or INFO/TAG for INFO values (e.g. DP<800 or INFO/DP<800)
    - %QUAL, %FILTER, etc. for column names (note: currently only some columns are supported)
    - %TYPE for variant type, such as %TYPE="indel"|"snp"|"mnp"|"other"
    - %FUNC(TAG) where FUNC is one of MAX, MIN, AVG and TAG is one of the FORMAT fields (e.g. %MIN(DV)>5)

' returned non-zero exit status 1

Seems the -T is really -t ?

[chapmanb]

Jason; The options changed again in the latest bcftools release candidate (rc8), so you need to update tools as well:

bcbio_nextgen.py upgrade --tools

which should get you the latest and greatest bcftools for compatibility. Hope this helps.

[caddymob]

Hey Brad - still doing it... Are these right?

~> cat  provenance/programs.txt
bcbio-nextgen,0.8.0a-eefa1a7
alientrimmer,0.3.2
bamtools,2.3.0
bcbio_variation,0.1.6
bcftools,0.2.0-rc8
bedtools,2.19.1
biobambam,0.0.140
bowtie2,2.1.0
bwa,0.7.9a
cufflinks,2.1.1
cutadapt,1.4.2
fastqc,0.10.1
featurecounts,1.4.4
freebayes,0.9.14-15
gatk,3.1-1-g07a4bf8
gatk-framework,3.1-1
gemini,0.6.5
htseq,0.6.1
mutect,1.1.5
novoalign,3.02.02
novosort,V3.00.02
picard,1.96
platypus-variant,0.5.2
qualimap,0.7.1
rnaseqc,1.1.7
sambamba,0.4.6
samtools,0.1.19
snpeff,3_4
tophat,2.0.9
varscan,2.3.6
vcflib,2014-03-25
vt,2014-04-23
cn.mops,
oncofuse,```

[chapmanb]

Jason; That all looks right. Is it possible you're getting a different older bcftools in your path? What does this report on the failing machine?

$ bcftools --help | grep Version
Version: 0.2.0-rc8 (using htslib 0.2.0-rc8)

You should able to see the new -T option:

$ bcftools filter 2>&1 | grep '\-T'
    -T, --targets-file <file>     similar to -R but streams rather than index-jumps

Sorry for being so picky about this change. The old behavior of using -t for both a file or a region means that new versions will silently fail by removing all of the variants if we kept any of the old code around (since the filename is not a valid selector it just ignores every variant in the file). Hope this helps identify the issue.

[caddymob]

Hmm, weird - bcftools in my path is definitely bcbio's copy, but the version is Version: 0.2.0-rc6+htslib-0.2.0-rc6 - not , 0.2.0-rc8 as listed in my programs.txt ...? Shoudl we just wipe it and run the bcbio_nextgen.py upgrade --tools again?

~> /packages/bcbio/LATEST/tooldir/bin/bcftools

Program: bcftools (Tools for variant calling and manipulating VCFs and BCFs)
Version: 0.2.0-rc6+htslib-0.2.0-rc6

[chapmanb]

Jason; Strange. You should be able to do:

/packages/bcbio/LATEST/tooldir/bin/brew uninstall --force bcftools
/packages/bcbio/LATEST/tooldir/bin/brew uninstall --force htslib
bcbio_nextgen.py upgrade --tools

and hopefully get rc8 after that. Sorry, I'm not sure what caused that but hope this fixes it.

[jpeden1]

Think I have found the problem. In an attempt to have a LATEST and test version of bcbio I copied bcbio from one directory to another. This way we could continue to run human while we worked on canfam. Running updates while other are trying to run jobs didn't seem to be a good idea? I set my path to be certain I am about to run/upgrade the correct bcbio_nextgen. The problem is that in some cases bcbio looks at your current path. In other cases it has configs, several I believe, that point to paths that were there when it was built the first time. I hope I am wrong, but to be certain I have fixed this issue I will need to do a COMPLETE reinstall. Is that correct?

[chapmanb]

Jim; Generally bcbio finds executables on your PATH and java directories via the galaxy/bcbio_system.yaml file. I'm not sure how brew reacts when you move/copy a directory so you're right that is might have symlinks pointing to previous installations from that process. Generally the install directories are not meant to be relocatable -- that was the idea behind the docker and container work -- so a clean reinstall of bcbio and the tools might make the most sense. I'd suggest just keeping a single copy of the data and symlinking to that to avoid needing to re-download/prep that, but the tool build shouldn't be too painful to redo. Hope this helps.

[jpeden1]

We are not using the docker. The problem is that when you run upgrades things like git and brew look at absolute paths. The hard coded path that was there when it was installed. When you run bcbio it looks at relative paths. When I ran the brew uninstall. It uninstalled the items in the current path. When I ran the upgrade brew did in installed base on an absolute path that was different from the ones in my current shell path. The upgrade would finish successfully. When looked for bcftools after the upgrade, they were gone. Not sure how bcbio was reporting it used a binary when that binary was not in its path? I have always thought that using absolute hard coded path inside applications was a bad idea.

So to re-download/prep what are we talking about? Is that just:

python bcbio_nextgen_install.py /path_to_bcbio --tooldir=/path_to_bcbio --nosudo --isolate

We do isolated installs with nosudo. Then I would need to upgrade to the development branch?

Would need the data, but about re-running --aligners ??

[chapmanb]

Jim; That sounds exactly right. I'd suggest linking to the old genome data directory before running, so do:

mkdir -p /path/to/bcbio
ln -s /old/genomes /path/to/bcbio/genomes
ln -s /old/gemini_data /path/to/bcbio/gemini_data
python bcbio_nextgen_install.py /path/to/bcbio --tooldir=/path/to/bcbio --nosudo --isolate --genomes canFam3 --aligners star

This lets you keep your genome data but links in that it's present for all the genomes and aligners you want to support. Hope this helps get things going.

Regarding the relative/local, I believe that's all brew-based. Generally moving directories of executables around is likely to cause pain. bcbio-nextgen gets the installed versions from the manifest directory, which might be the source of the confusion since it things there is one thing installed but it's actually been changed. Thanks again for all the feedback.

[jpeden1]

I checked the manifest directory. None of the yaml files there have any path in them. Ran the install last night. It failed on a lib it was looking for. Restarting.