search

bcbio / bcbio-nextgen

Validated, scalable, community developed variant calling, RNA-seq and small RNA analysis
https://bcbio-nextgen.readthedocs.io
MIT License
986 stars 354 forks source link

Unit Test failure #2947

Closed lubinskibu closed 5 years ago

lubinskibu commented 5 years ago

Just did a clean install of bcbio followed by upgrade to latest-devel and ran the cancermulti test and had that fail.


[2019-09-16T17:13Z] Ensemble intersection calling: PairedBatch
[2019-09-16T17:13Z]                    bcbio.variation.recall.main.main
[2019-09-16T17:13Z]                                                 ...
[2019-09-16T17:13Z]                   bcbio.variation.recall.main/-main            main.clj:   30
[2019-09-16T17:13Z]                   bcbio.variation.recall.main/-main            main.clj:   33
[2019-09-16T17:13Z]                bcbio.variation.recall.main/-main/fn            main.clj:   34
[2019-09-16T17:13Z]                                  clojure.core/apply            core.clj:  665
[2019-09-16T17:13Z]                                                 ...
[2019-09-16T17:13Z]            bcbio.variation.ensemble.intersect/-main       intersect.clj:  117
[2019-09-16T17:13Z]            bcbio.variation.ensemble.intersect/-main       intersect.clj:  141
[2019-09-16T17:13Z]    bcbio.variation.ensemble.intersect/ensemble-vcfs       intersect.clj:   87
[2019-09-16T17:13Z] bcbio.variation.ensemble.intersect/ensemble-vcfs/fn       intersect.clj:   89
[2019-09-16T17:13Z]                                                 ...
[2019-09-16T17:13Z] bcbio.variation.variantcontext/write-vcf-w-template  variantcontext.clj:  167
[2019-09-16T17:13Z] bcbio.variation.variantcontext/write-vcf-w-template  variantcontext.clj:  191
[2019-09-16T17:13Z]                                    clojure.core/seq            core.clj:  137
[2019-09-16T17:13Z]                                                 ...
[2019-09-16T17:13Z]                                 clojure.core/map/fn            core.clj: 2746
[2019-09-16T17:13Z]                                    clojure.core/seq            core.clj:  137
[2019-09-16T17:13Z]                                                 ...
[2019-09-16T17:13Z]                                 clojure.core/map/fn            core.clj: 2755
[2019-09-16T17:13Z]                                clojure.core/comp/fn            core.clj: 2569
[2019-09-16T17:13Z]    bcbio.variation.ensemble.intersect/get-rep-vc/fn       intersect.clj:   59
[2019-09-16T17:13Z] java.lang.Exception: Problem retrieving reference variant for {:chr "chr22", :start 221, :refa "G", :alta ("T"), :end 222, :vc-indic
es (0 5)}: []
[2019-09-16T17:13Z] Uncaught exception occurred
Traceback (most recent call last):
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'set -o pipefail; unset JAVA_HOME && export PATH=/projects/ngs/local/software/bcbio-nextgen.latest_devel/bcbio-de
vel.20190916_152943/anaconda/bin:"$PATH" && /projects/ngs/local/software/bb5D7FAA67/anaconda/bin/bcbio-variation-recall ensemble --cores=1 --numpass 2 -
-names vardict,freebayes,varscan,mutect2,strelka2,octopus --nofiltered /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/
bcbiotx/tmp6nm7xlks/PairedBatch-ensemble.vcf.gz /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/data/genomes/hg19/seq/hg19.fa /projects/ngs/l
ocal/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/ensemble/PairedBatch/vardict/PairedBatch-effects-annotated-damage-ann-noeff-nomultiall
elic.vcf.gz /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/ensemble/PairedBatch/freebayes/PairedBatch-effects-annotate
d-filter-damage-ann-noeff-nomultiallelic.vcf.gz /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/ensemble/PairedBatch/va
rscan/PairedBatch-effects-annotated-damage-ann-noeff-nomultiallelic.vcf.gz /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/ensemble/PairedBatch/mutect2/PairedBatch-annotated-noeff-nomultiallelic.vcf.gz /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_autom
ated_output/ensemble/PairedBatch/strelka2/PairedBatch-effects-annotated-damage-ann-noeff-nomultiallelic.vcf.gz /projects/ngs/local/software/bcbiotest/bc
bio-nextgen/tests/test_automated_output/ensemble/PairedBatch/octopus/PairedBatch-effects-annotated-damage-ann-noeff-nomultiallelic.vcf.gz                                  bcbio.variation.recall.main.main
                                                ...
                  bcbio.variation.recall.main/-main            main.clj:   30
                  bcbio.variation.recall.main/-main            main.clj:   33
               bcbio.variation.recall.main/-main/fn            main.clj:   34
                                 clojure.core/apply            core.clj:  665
                                                ...
           bcbio.variation.ensemble.intersect/-main       intersect.clj:  117
           bcbio.variation.ensemble.intersect/-main       intersect.clj:  141
   bcbio.variation.ensemble.intersect/ensemble-vcfs       intersect.clj:   87
bcbio.variation.ensemble.intersect/ensemble-vcfs/fn       intersect.clj:   89
                                                ...
bcbio.variation.variantcontext/write-vcf-w-template  variantcontext.clj:  167
bcbio.variation.variantcontext/write-vcf-w-template  variantcontext.clj:  191
                                   clojure.core/seq            core.clj:  137
                                                ...
                                clojure.core/map/fn            core.clj: 2746
                                   clojure.core/seq            core.clj:  137
                                                ...                                                                                                                                     clojure.core/map/fn            core.clj: 2755                                                                                                          clojure.core/comp/fn            core.clj: 2569
   bcbio.variation.ensemble.intersect/get-rep-vc/fn       intersect.clj:   59
java.lang.Exception: Problem retrieving reference variant for {:chr "chr22", :start 221, :refa "G", :alta ("T"), :end 222, :vc-indices (0 5)}: []
' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/bin/bcbio_nextgen.py", line 245, in <module>                                                       main(**kwargs)                                                                                                                                        File "/projects/ngs/local/software/bb5D7FAA67/anaconda/bin/bcbio_nextgen.py", line 46, in main
    run_main(**kwargs)
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 50, in run_main
    fc_dir, run_info_yaml)
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 91, in _run_toplevel
    for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 174, in variant2pipeline
    samples = ensemble.combine_calls_parallel(samples, run_parallel)
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/variation/ensemble.py", line 118, in combine_calls_parallel
    processed = run_parallel("combine_calls", ((b, xs, xs[0]) for b, xs in batch_groups.items()))
File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/variation/ensemble.py", line 118, in combine_calls_parallel
    processed = run_parallel("combine_calls", ((b, xs, xs[0]) for b, xs in batch_groups.items()))
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
    return run_multicore(fn, items, config, parallel=parallel)
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
    for data in joblib.Parallel(parallel["num_jobs"], batch_size=1, backend="multiprocessing")(joblib.delayed(fn)(*x) for x in items):
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 921, in __call__
    if self.dispatch_one_batch(iterator):
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 759, in dispatch_one_batch
    self._dispatch(tasks)
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 716, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 182, in apply_async
    result = ImmediateResult(func)
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 549, in __init__
    self.results = batch()
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 225, in __call__
    for func, args, kwargs in self.items]
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 225, in <listcomp>
    for func, args, kwargs in self.items]
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/utils.py", line 55, in wrapper
    return f(*args, **kwargs)
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/distributed/multitasks.py", line 375, in combine_calls
    return ensemble.combine_calls(*args)
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/variation/ensemble.py", line 84, in combine_calls
    callinfo = _run_ensemble_intersection(batch_id, vrn_files, caller_names, base_dir, edata)
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/variation/ensemble.py", line 258, in _run_ensemble_intersection
    do.run(cmd, "Ensemble intersection calling: %s" % (batch_id))
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'set -o pipefail; unset JAVA_HOME && export PATH=/projects/ngs/local/software/bcbio-nextgen.latest_devel/bcbio-devel.20190916_152943/anaconda/bin:"$PATH" && /projects/ngs/local/software/bb5D7FAA67/anaconda/bin/bcbio-variation-recall ensemble --cores=1 --numpass 2 --names vardict,freebayes,varscan,mutect2,strelka2,octopus --nofiltered /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/bcbiotx/tmp6nm7xlks/PairedBatch-ensemble.vcf.gz /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/data/genomes/hg19/seq/hg19.fa /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/ensemble/PairedBatch/vardict/PairedBatch-effects-annotated-damage-ann-noeff-nomultiallelic.vcf.gz /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/ensemble/PairedBatch/freebayes/PairedBatch-effects-annotated-filter-damage-ann-noeff-nomultiallelic.vcf.gz /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/ensemble/PairedBatch/varscan/PairedBatch-effects-annotated-damage-ann-noeff-nomultiallelic.vcf.gz /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/ensemble/PairedBatch/mutect2/PairedBatch-annotated-noeff-nomultiallelic.vcf.gz /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/ensemble/PairedBatch/strelka2/PairedBatch-effects-annotated-damage-ann-noeff-nomultiallelic.vcf.gz /projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/ensemble/PairedBatch/octopus/PairedBatch-effects-annotated-damage-ann-noeff-nomultiallelic.vcf.gz
                   bcbio.variation.recall.main.main
                                                ...
                  bcbio.variation.recall.main/-main            main.clj:   30
                  bcbio.variation.recall.main/-main            main.clj:   33
               bcbio.variation.recall.main/-main/fn            main.clj:   34
                                 clojure.core/apply            core.clj:  665
                                                ...
           bcbio.variation.ensemble.intersect/-main       intersect.clj:  117
           bcbio.variation.ensemble.intersect/-main       intersect.clj:  141
   bcbio.variation.ensemble.intersect/ensemble-vcfs       intersect.clj:   87
bcbio.variation.ensemble.intersect/ensemble-vcfs/fn       intersect.clj:   89
                                                ...
bcbio.variation.variantcontext/write-vcf-w-template  variantcontext.clj:  167
bcbio.variation.variantcontext/write-vcf-w-template  variantcontext.clj:  191
                                   clojure.core/seq            core.clj:  137
                                                ...
                                clojure.core/map/fn            core.clj: 2746
                                   clojure.core/seq            core.clj:  137
                                                ...
                                clojure.core/map/fn            core.clj: 2755
                               clojure.core/comp/fn            core.clj: 2569
   bcbio.variation.ensemble.intersect/get-rep-vc/fn       intersect.clj:   59
java.lang.Exception: Problem retrieving reference variant for {:chr "chr22", :start 221, :refa "G", :alta ("T"), :end 222, :vc-indices (0 5)}: []
' returned non-zero exit status 1.
FAILED

======================================================================= FAILURES =======================================================================
____________________________________________________________________ test_7_cancer _____________________________________________________________________

install_test_files = None, data_dir = '/projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/data/automated'

    @pytest.mark.speed2
    @pytest.mark.cancer
    @pytest.mark.cancermulti
    @pytest.mark.install_required
    def test_7_cancer(install_test_files, data_dir):
        """Test paired tumor-normal calling using multiple
        calling approaches: MuTect, VarScan, FreeBayes.
        """
        with make_workdir() as workdir:
            cl = ["bcbio_nextgen.py",
                  get_post_process_yaml(data_dir, workdir),
                  os.path.join(data_dir, "run_info-cancer.yaml")]
>           subprocess.check_call(cl)

integration/test_automated_analysis.py:243:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['bcbio_nextgen.py', '/projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/bcbio_system.yaml', '/projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/data/automated/run_info-cancer.yaml'],)
kwargs = {}, retcode = 1
cmd = ['bcbio_nextgen.py', '/projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/bcbio_system.yaml', '/projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/data/automated/run_info-cancer.yaml']

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.

        The arguments are the same as for the call function.  Example:

        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['bcbio_nextgen.py', '/projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/test_automated_output/bcbio_system.yaml', '/projects/ngs/local/software/bcbiotest/bcbio-nextgen/tests/data/automated/run_info-cancer.yaml']' returned non-zero exit status 1.

../../../bb5D7FAA67/anaconda/lib/python3.6/subprocess.py:291: CalledProcessError
=================================================================== warnings summary ===================================================================
/projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/Bio/Alphabet/__init__.py:26
  /projects/ngs/local/software/bb5D7FAA67/anaconda/lib/python3.6/site-packages/Bio/Alphabet/__init__.py:26: PendingDeprecationWarning: We intend to remove or replace Bio.Alphabet in 2020, ideally avoid using it explicitly in your code. Please get in touch if you will be adversely affected by this. https://github.com/biopython/biopython/issues/2046
    PendingDeprecationWarning)

/projects/ngs/local/software/bcbiotest/bcbio-nextgen/bcbio/structural/delly.py:83
  /projects/ngs/local/software/bcbiotest/bcbio-nextgen/bcbio/structural/delly.py:83: DeprecationWarning: invalid escape sequence \.
    cmd = ("bcftools view {bcf_file} | sed 's/\.,\.,\././' | bgzip -c > {tx_out_file}")

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=============================================== 1 failed, 81 deselected, 2 warnings in 424.31s (0:07:04) ===============================================```
roryk commented 5 years ago

Thanks, this should be all set now with brad's fix to bcbio-variation-recall. Thanks for reporting it Tristan!