Closed hackinger closed 5 years ago
roryk
commented
5 years ago Thanks, I think, but I'm not sure, that this is because stuff is getting pulled in from pkgs/main. I'm trying to figure out if that is the new normal, or if we can turn it off or otherwise work around it.
Fazulur
commented
5 years ago Hi Rory, I am also getting the same error while trying to install bcbio. Please help me to resolve this issue.
Thanks In Advance Fazulur Rehaman
Fazulur
commented
5 years ago Dear Brad, I re-ran installation again and it is giving same error with R 3.5.1.
Error:
UnsatisfiableError: The following specifications were found to be incompatible with each other:
- r=3.5.1
Traceback (most recent call last):
File "BCBIO/v1.1.5/anaconda/bin/bcbio_nextgen.py", line 221, in
Please help me to resolve this issue.
Thanks In Advance Fazulur Rehaman
chapmanb
commented
5 years ago Thank you all for the report and detailed debugging and apologies about the issues. We've pushed fixes to the install that should resolve this now if you remove your cached cloudbiolinux (rm -rf tmpbcbio-install) and then re-run your install/upgrade command in place. It will pick up where it left off and should resolve and install everything correctly.
The underlying issue was version pinning we had in place for multiqc (no longer needed) and resolving slow install times, which caused the incompatibilites between the base install and initial set of tools. To avoid reverting to slow environment resolution, the fix involves using the in-development mamba solver (https://github.com/QuantStack/mamba) for the initial tool bootstrap, then reverts back to using standard conda for updates. We'll continue to explore mamba going forward as a replacement, but this helps us get through this current install issue.
Please let us know if you run into any other problems.
Fazulur
commented
5 years ago Dear Brad,
Thanks a lot for your suggestions in install. We tried to install again and getting "Mamba" related error now.
Reading packages from BCBIO/v1.1.5/tmpbcbio-install/cloudbiolinux/contrib/flavor/ngs_pipeline_minimal/packages-conda.yaml
BCBIO/v1.1.5/tmpbcbio-install/cloudbiolinux/cloudbio/package/shared.py:11: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
full_data = yaml.load(inhandle)
Checking for problematic or migrated packages in default environment
Initalling initial set of packages for default environment with mamba
/ \ / \ / \ / \
/ \/ \/ \/ \
███████████████/ /██/ /██/ /██/ /████████████████████████
/ / \ / \ / \ / \ ___
/ / _/ _/ _/ \ o \,
/ / ____/ `
|/
███╗ ███╗ █████╗ ███╗ ███╗██████╗ █████╗
████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
██╔████╔██║███████║██╔████╔██║██████╔╝███████║
██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
██║ ╚═╝ ██║██║ ██║██║ ╚═╝ ██║██████╔╝██║ ██║
╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝
Supported by @QuantStack
GitHub: https://github.com/QuantStack/mamba
Twitter: https://twitter.com/QuantStack
█████████████████████████████████████████████████████████████
Getting https://conda.anaconda.org/conda-forge/linux-64
Getting https://conda.anaconda.org/conda-forge/noarch
Getting https://conda.anaconda.org/bioconda/linux-64
Getting https://conda.anaconda.org/bioconda/noarch
Getting https://repo.anaconda.com/pkgs/main/linux-64
Getting https://repo.anaconda.com/pkgs/main/noarch
Getting https://repo.anaconda.com/pkgs/free/linux-64
Getting https://repo.anaconda.com/pkgs/free/noarch
Getting https://repo.anaconda.com/pkgs/r/linux-64
Getting https://repo.anaconda.com/pkgs/r/noarch
Getting https://conda.anaconda.org/r/linux-64
Getting https://conda.anaconda.org/r/noarch
Looking for: ['python 3.', 'age-metasv', 'anaconda-client', 'awscli', 'bamtools 2.4.0.', 'bamutil', 'bbmap', 'bcbio-prioritize', 'bcbio-variation', 'bcbio-variation-recall', 'bcftools', 'bedtools 2.27.1.', 'bio-vcf', 'biobambam', 'bioconductor-annotate', 'bioconductor-biocgenerics', 'bioconductor-biocinstaller', 'bioconductor-biocstyle', 'bioconductor-biostrings', 'bioconductor-biovizbase', 'bioconductor-bsgenome.hsapiens.ucsc.hg19', 'bioconductor-bsgenome.hsapiens.ucsc.hg38', 'bioconductor-bubbletree', 'bioconductor-cn.mops', 'bioconductor-copynumber', 'bioconductor-degreport', 'bioconductor-deseq2', 'bioconductor-dexseq', 'bioconductor-dnacopy', 'bioconductor-genomeinfodbdata', 'bioconductor-genomicranges', 'bioconductor-iranges', 'bioconductor-limma', 'bioconductor-purecn', 'bioconductor-rtracklayer', 'bioconductor-titancna', 'bismark 0.22.1.', 'bowtie', 'bowtie2', 'break-point-inspector', 'bwa', 'bwakit', 'bzip2', 'cage', 'cancerit-allelecount', 'chipseq-greylist', 'cnvkit', 'coincbc', 'cramtools', 'cufflinks', 'cyvcf2', 'deeptools', 'delly', 'duphold >=0.0.9', 'ensembl-vep 96.', 'express', 'extract-sv-reads', 'fastp', 'fastqc >=0.11.8 1', 'fgbio', 'freebayes 1.1.0.46.', 'gatk', 'gatk4', 'geneimpacts', 'genesplicer', 'gffcompare', 'goleft', 'grabix', 'gridss', 'gsort', 'gvcfgenotyper', 'h5py', 'hmftools-amber', 'hmftools-cobalt', 'hmftools-purple', 'hmmlearn', 'hts-nim-tools', 'htslib', 'impute2', 'kallisto >=0.43.1', 'kraken', 'lofreq', 'maxentscan', 'mbuffer', 'minimap2', 'mintmap', 'mirdeep2 2.0.0.7.', 'mirtop', 'moreutils', 'mosdepth', 'multiqc', 'multiqc-bcbio', 'ncurses', 'ngs-disambiguate', 'nodejs', 'novoalign', 'octopus >=0.5.1b', 'oncofuse', 'p7zip', 'parallel', 'pbgzip', 'perl-app-cpanminus', 'perl-archive-extract', 'perl-archive-zip', 'perl-bio-db-sam', 'perl-cgi', 'perl-dbi', 'perl-encode-locale', 'perl-file-fetch', 'perl-file-sharedir', 'perl-file-sharedir-install', 'perl-ipc-system-simple', 'perl-lwp-protocol-https', 'perl-lwp-simple', 'perl-sanger-cgp-battenberg', 'perl-statistics-descriptive', 'perl-time-hires', 'perl-vcftools-vcf', 'picard', 'pindel', 'pizzly', 'preseq', 'pyloh', 'pysam >=0.14.0', 'pythonpy', 'qsignature', 'qualimap', 'r-base', 'r-basejump 0.7.2.', 'r-bcbiornaseq >=0.2.7', 'r-cghflasso', 'r-chbutils', 'r-devtools', 'r-dplyr', 'r-dt', 'r-ggdendro', 'r-ggplot2', 'r-ggrepel >=0.7', 'r-gplots', 'r-gsalib', 'r-knitr', 'r-pheatmap', 'r-plyr', 'r-pscbs', 'r-reshape', 'r-rmarkdown', 'r-rsqlite', 'r-sleuth', 'r-snow', 'r-stringi', 'r-viridis >=0.5', 'r-wasabi', 'r 3.5.1.', 'rapmap', 'razers3 3.5.0.', 'readline', 'rtg-tools', 's3gof3r', 'sailfish', 'salmon', 'sambamba', 'samblaster', 'samtools', 'scalpel', 'seq2c', 'seqbuster', 'seqcluster', 'seqtk', 'sickle-trim', 'simple_sv_annotation', 'singlecell-barcodes', 'snap-aligner 1.0dev.97.', 'snpeff 4.3.1t.', 'solvebio', 'spades', 'star 2.6.1d.', 'stringtie', 'subread', 'survivor', 'tdrmapper', 'tophat-recondition', 'trim-galore 0.6.2.', 'ucsc-bedgraphtobigwig', 'ucsc-bedtobigbed', 'ucsc-bigbedinfo', 'ucsc-bigbedsummary', 'ucsc-bigbedtobed', 'ucsc-bigwiginfo', 'ucsc-bigwigsummary', 'ucsc-bigwigtobedgraph', 'ucsc-bigwigtowig', 'ucsc-fatotwobit', 'ucsc-gtftogenepred', 'ucsc-liftover', 'ucsc-wigtobigwig', 'vardict', 'vardict-java', 'variantbam', 'varscan', 'vcfanno', 'vcflib', 'verifybamid2', 'viennarna', 'vqsr_cnn', 'vt', 'wham', 'xorg-libxt', 'xz']
Traceback (most recent call last):
File "BCBIO/v1.1.5/anaconda/bin/bcbio_nextgen.py", line 221, in
Please suggest me how can I proceed further.
Thanks & Regards Fazulur Rehaman
chapmanb
commented
5 years ago Fazulur; Sorry about the additional issues. It looks like the machine you're running on is killing the mamba process
died with <Signals.SIGILL: 4>It is more resource intensive than standard conda so might need additional resources or a different machine to run. I'm trying to reproduce but it's running fine on Ubuntu machines I've tested on so I'm not entirely positive what would cause it. Is it reproducible and also happens across multiple machines in your cluster? We might need to escalate the question to the mamba repository if we can isolate details about the machines and system types this happens on. Thanks for the help debugging.
Fazulur
commented
5 years ago Dear Brad,
We tried to install in two different nodes but still we are facing the same issue. Please suggest me how can I proceed further.
Thanks In Advance Fazulur Rehaman
chapmanb
commented
5 years ago Fazulur; Sorry for the continued problems. At this point our best bet is to try to understand why it fails on your systems and provide a workaround and hopefully helpful report upstream to the mamba team. Can you provide more details about the operating system of the machines you're installing on where it's running into this problems? Thanks much for the help debugging.
Fazulur
commented
5 years ago Dear Brad,
Thanks for your response. Please find below the details of our system
$ cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.5 (Maipo) $ uname -a Linux HOSTNAME 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux $ cat /etc/os-release NAME="Red Hat Enterprise Linux Server" VERSION="7.5 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.5" PRETTY_NAME="Red Hat Enterprise Linux" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.5:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.5 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.5"
Please suggest me how can I proceed further.
Thanks In Advance Fazulur Rehaman
Keith1mex
commented
5 years ago Hi,
I am getting a similar error. I was wondering if I had this problem because I am running Ubuntu 18.04.2 LTS instead of 16
This problem (Illegal instruction (core dumped)) has been solved by downgrading tensorflow to 1.5. Which I tried and did not work.
Thank you
Kether
GitHub: https://github.com/QuantStack/mamba
Twitter: https://twitter.com/QuantStack█████████████████████████████████████████████████████████████
Getting https://conda.anaconda.org/conda-forge/linux-64 Getting https://conda.anaconda.org/conda-forge/noarch Getting https://conda.anaconda.org/bioconda/linux-64 Getting https://conda.anaconda.org/bioconda/noarch Getting https://repo.anaconda.com/pkgs/main/linux-64 Getting https://repo.anaconda.com/pkgs/main/noarch Getting https://repo.anaconda.com/pkgs/free/linux-64 Getting https://repo.anaconda.com/pkgs/free/noarch Getting https://repo.anaconda.com/pkgs/r/linux-64 Getting https://repo.anaconda.com/pkgs/r/noarch
Looking for: ['python 3.', 'age-metasv', 'anaconda-client', 'awscli', 'bamtools 2.4.0.', 'bamutil', 'bbmap', 'bcbio-prioritize', 'bcbio-variation', 'bcbio-variation-recall', 'bcftools', 'bedtools 2.27.1.', 'bio-vcf', 'biobambam', 'bioconductor-annotate', 'bioconductor-biocgenerics', 'bioconductor-biocinstaller', 'bioconductor-biocstyle', 'bioconductor-biostrings', 'bioconductor-biovizbase', 'bioconductor-bsgenome.hsapiens.ucsc.hg19', 'bioconductor-bsgenome.hsapiens.ucsc.hg38', 'bioconductor-bubbletree', 'bioconductor-cn.mops', 'bioconductor-copynumber', 'bioconductor-degreport', 'bioconductor-deseq2', 'bioconductor-dexseq', 'bioconductor-dnacopy', 'bioconductor-genomeinfodbdata', 'bioconductor-genomicranges', 'bioconductor-iranges', 'bioconductor-limma', 'bioconductor-purecn', 'bioconductor-rtracklayer', 'bioconductor-titancna', 'bismark 0.22.1.', 'bowtie', 'bowtie2', 'break-point-inspector', 'bwa', 'bwakit', 'bzip2', 'cage', 'cancerit-allelecount', 'chipseq-greylist', 'cnvkit', 'coincbc', 'cramtools', 'cufflinks', 'cyvcf2', 'deeptools', 'delly', 'duphold >=0.0.9', 'ensembl-vep 96.', 'express', 'extract-sv-reads', 'fastp', 'fastqc >=0.11.8 1', 'fgbio', 'freebayes 1.1.0.46.', 'gatk', 'gatk4', 'geneimpacts', 'genesplicer', 'gffcompare', 'goleft', 'grabix', 'gridss', 'gsort', 'gvcfgenotyper', 'h5py', 'hmftools-amber', 'hmftools-cobalt', 'hmftools-purple', 'hmmlearn', 'hts-nim-tools', 'htslib', 'impute2', 'kallisto >=0.43.1', 'kraken', 'lofreq', 'maxentscan', 'mbuffer', 'minimap2', 'mintmap', 'mirdeep2 2.0.0.7.', 'mirtop', 'moreutils', 'mosdepth', 'multiqc', 'multiqc-bcbio', 'ncurses', 'ngs-disambiguate', 'nodejs', 'novoalign', 'octopus >=0.5.1b', 'oncofuse', 'p7zip', 'parallel', 'pbgzip', 'perl-app-cpanminus', 'perl-archive-extract', 'perl-archive-zip', 'perl-bio-db-sam', 'perl-cgi', 'perl-dbi', 'perl-encode-locale', 'perl-file-fetch', 'perl-file-sharedir', 'perl-file-sharedir-install', 'perl-ipc-system-simple', 'perl-lwp-protocol-https', 'perl-lwp-simple', 'perl-sanger-cgp-battenberg', 'perl-statistics-descriptive', 'perl-time-hires', 'perl-vcftools-vcf', 'picard', 'pindel', 'pizzly', 'preseq', 'pyloh', 'pysam >=0.14.0', 'pythonpy', 'qsignature', 'qualimap', 'r-base', 'r-basejump 0.7.2.', 'r-bcbiornaseq >=0.2.7', 'r-cghflasso', 'r-chbutils', 'r-devtools', 'r-dplyr', 'r-dt', 'r-ggdendro', 'r-ggplot2', 'r-ggrepel >=0.7', 'r-gplots', 'r-gsalib', 'r-knitr', 'r-pheatmap', 'r-plyr', 'r-pscbs', 'r-reshape', 'r-rmarkdown', 'r-rsqlite', 'r-sleuth', 'r-snow', 'r-stringi', 'r-viridis >=0.5', 'r-wasabi', 'r 3.5.1.', 'rapmap', 'razers3 3.5.0.', 'readline', 'rtg-tools', 's3gof3r', 'sailfish', 'salmon', 'sambamba', 'samblaster', 'samtools', 'scalpel', 'seq2c', 'seqbuster', 'seqcluster', 'seqtk', 'sickle-trim', 'simple_sv_annotation', 'singlecell-barcodes', 'snap-aligner 1.0dev.97.', 'snpeff 4.3.1t.', 'solvebio', 'spades', 'star 2.6.1d.', 'stringtie', 'subread', 'survivor', 'tdrmapper', 'tophat-recondition', 'trim-galore 0.6.2.', 'ucsc-bedgraphtobigwig', 'ucsc-bedtobigbed', 'ucsc-bigbedinfo', 'ucsc-bigbedsummary', 'ucsc-bigbedtobed', 'ucsc-bigwiginfo', 'ucsc-bigwigsummary', 'ucsc-bigwigtobedgraph', 'ucsc-bigwigtowig', 'ucsc-fatotwobit', 'ucsc-gtftogenepred', 'ucsc-liftover', 'ucsc-wigtobigwig', 'vardict', 'vardict-java', 'variantbam', 'varscan', 'vcfanno', 'vcflib', 'verifybamid2', 'viennarna', 'vqsr_cnn', 'vt', 'wham', 'xorg-libxt', 'xz']
Illegal instruction (core dumped)
Traceback (most recent call last):
File "/usr/local/share/bcbio/anaconda/bin/bcbio_nextgen.py", line 221, in
chapmanb
commented
5 years ago Thank you both for the additional information and apologies about the mamba issues. I've been trying to reproduce to file a report upstream but can't do so, so for now have added in a fix to cloudbiolinux to catch and fall back to standard conda installs. I'm hoping without mamba this will resolve the initial problem and still be able to install in a timely manner. I'll continue to try and reproduce to find a minimal example to report but I hope this get you unstuck on the install. Thanks again for the patience debugging.
Fazulur
commented
5 years ago Dear Brad,
Thanks a lot for resolving issues with installation. Bcbio installed without issues & I tested gatk-variant & rna-seq pipelines as well.
As we are using LSF as scheduler, we submitted bcbio main job to lsf. Pipeline generated results in final directory. Engines & control job were stopped, but main bcbio job is still in RUN state and not stopped.
Here is the standard output
Sending a shutdown signal to the controller and engines.
And here is the ipython cluster log
2019-07-29 07:06:38.021 [IPClusterStart] Stopping Engines... 2019-07-29 07:06:38.033 [IPClusterStart] Process 'bsub' stopped: {'job_id': '629141', 'output': 'Job <629141>: Operation is in progress\n'}
Could you please help me to resolve this issue.
Thanks In Advance Fazulur Rehaman
roryk
commented
5 years ago Hi Fazulur,
Did the run finish? The bcbio-nextgen-debug.log file should say finished at the bottom, the last step that bcbio runs is a single-threaded copying of files to where they should be. This can take a long time if you have a lot of files and they are big. Is it doing that step or did it say Finished?
If the run was finished, and the main job still is running, this can happen sometimes, unfortunately. It looks like bcbio tried to kill the job but the scheduler hasn't killed it yet or it didn't work, there isn't too much we can do about it without a huge engineering effort specifically for each scheduler.
Fazulur
commented
5 years ago Hi Rory,
Thanks a lot for your quick response.
Yes, run finished moving files from working directory to final directory. bcbio-nextgen-debug.log also has "Timing: finished" message. But still main job is running. You are right bcbio tried to kill the job & from LSF side it's not working. I killed manually.
In addition for some samples engine jobs were not started eventhough we have enough resources. I can see main & control jobs but not engjne jobs. In ipython cluster log, It was showing "Engines appear to have started successfully". But not started.
Could you please hep me to resolve this issue.
Thanks & Regards Fazulur Rehaman
roryk
commented
5 years ago Hi Fazulur,
If you are talking about for some samples in the same bcbio run, it is tough to say without knowing what the job step was, it is normal behavior for bcbio to spin up a cluster for a set of steps and spin it back down to get a different set of resources and spin it back up for new set of steps. You may have caught it in the middle of one of those spin-downs and ups. This isn't an issue, it's a feature.
If you are talking about different bcbio runs where the engines never started at all, then that is different. Usually what is happening is because you are working on a scheduler, and the main bcbio job and the controller user very little resources, they spin up very fast whereas the engine job is pending for a long time since it is resource intensive. Sometimes it can pend long enough that bcbio gives up on it ever starting-- the time it waits before it gives up is controlled by the --timeout parameter. If this is happening a lot, set it to a value like 3000, which will wait for 3000 minutes before giving up.
Hi there,
we're trying to install bcbio with the below command: python bcbio_nextgen_install.py /shared/pipeline-user/bcbio --tooldir=/shared/pipeline-user/tools/local --genomes hg38 --aligners bwa --datatarget variation --datatarget gemini
We've installed bcbio with the same command on another cluster a week ago and it worked fine, but now we're getting an incompatibility error for several packages:
UnsatisfiableError: The following specifications were found to be incompatible with each other:
bcbio_install_crash.log
Any help solving this would be greatly appreciated!