Closed panic0918 closed 2 years ago
Hi @panic0918 !
Thanks for reporting and sorry about the issue!
It seems that this recipe is failing: https://github.com/chapmanb/cloudbiolinux/blob/master/ggd-recipes/hg38/dbsnp.yaml
If you could test the recipe in more detail, that might help:
Here is your tmp installation dir:
/home/ngs/bcbio/genomes/Hsapiens/hg38/txtmp
Here is the recipe:
/home/ngs/bcbio/genomes/Hsapiens/hg38/txtmp/ggd-run.sh
Could you please check, that https://github.com/chapmanb/cloudbiolinux/blob/master/ggd-recipes/hg38/dbsnp.yaml#L21-L22 these files are downloaded successfully? (if no - downloading issue, please try to load them from the command line)
If so, could you please try running the next commands one by one to see where it fails exactly https://github.com/chapmanb/cloudbiolinux/blob/master/ggd-recipes/hg38/dbsnp.yaml#L23-L26
Sergey
As you said, ggd-run.I changed the sh, but it's the same situation.
ngs@basiclab01:~/bcbio/anaconda/bin$ ./bcbio_nextgen.py upgrade -u skip --genomes hg38 --aligners bwa Upgrading bcbio Upgrading bcbio-nextgen data files List of genomes to get (from the config file at '{'genomes': [{'dbkey': 'hg38', 'name': 'Human (hg38) full', 'indexes': ['seq', 'twobit', 'bwa', 'hisat2'], 'annotations': ['ccds', 'capture_regions', 'coverage', 'prioritize', 'dbsnp', 'hapmap_snps', '1000g_omni_snps', 'ACMG56_genes', '1000g_snps', 'mills_indels', '1000g_indels', 'clinvar', 'qsignature', 'genesplicer', 'effects_transcripts', 'varpon', 'vcfanno', 'viral', 'purecn_mappability', 'simple_repeat', 'af_only_gnomad', 'transcripts', 'RADAR', 'rmsk', 'salmon-decoys', 'fusion-blacklist', 'mirbase'], 'validation': ['giab-NA12878', 'giab-NA24385', 'giab-NA24631', 'platinum-genome-NA12878', 'giab-NA12878-remap', 'giab-NA12878-crossmap', 'dream-syn4-crossmap', 'dream-syn3-crossmap', 'giab-NA12878-NA24385-somatic', 'giab-NA24143', 'giab-NA24149', 'giab-NA24694', 'giab-NA24695']}], 'genome_indexes': ['bwa', 'rtg'], 'install_liftover': False, 'install_uniref': False}'): Human (hg38) full Running GGD recipe: hg38 dbsnp 154-20210112 URL transformed to HTTPS due to an HSTS policy --2021-12-07 09:04:41-- https://ftp.ncbi.nih.gov/snp/archive/b154/VCF/GCF_000001405.38.gz Resolving ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)... 130.14.250.7, 130.14.250.10, 2607:f220:41e:250::7, ... Connecting to ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)|130.14.250.7|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable
The file is already fully retrieved; nothing to do.
URL transformed to HTTPS due to an HSTS policy --2021-12-07 09:04:43-- https://ftp.ncbi.nih.gov/snp/archive/b154/VCF/GCF_000001405.38.gz.tbi Resolving ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)... 130.14.250.10, 130.14.250.7, 2607:f220:41e:250::13, ... Connecting to ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)|130.14.250.10|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable
The file is already fully retrieved; nothing to do.
Writing to .qQZiUh
[buf_flush] Error: cannot write to .qQZiUh/00083.bcf
Cleaning
tbx_index_build failed: dbsnp-154.vcf.gz
Traceback (most recent call last):
File "./bcbio_nextgen.py", line 228, in
dbsnp-154-orig.vcf.gz dbsnp-154-orig.vcf.gz.Tbi exists, so I entered the next command myself.
The program 'bcftools' can be found in the following packages:
Can I download one of the two separately? Thank you for your sincere answer.
I see, the installer can't find bcftools.
Please make sure it is available after you installed bcbio:
echo $PATH
# the output should contain [bcbio]/anaconda/bin:[bcbio]/tools/bin
$ which bcftools
[bcbio]/anaconda/bin/bcftools on my system
If there is no bcftools in which bcftools
please make sure you have set up your PATH properly (look in your ~/.bashrc or ~/.bash_profile)
Sergey
I'm so sorry for the continuing questions. I'm done with the settings you mentioned earlier. I entered the command line one by one. The results are like this.
ngs@basiclab01:~/bcbio/genomes/Hsapiens/hg38/txtmp/variation$ [[ -f dbsnp-154.vcf.gz ]] || bcftools annotate -Ou --rename-chrs https://gist.githubusercontent.com/matthdsm/f833aedd2d67e28013ff1d171c70f4ee/raw/442a45ed3ddc6e85c66c5e58e0fa78e16a0821c8/refseq2ucsc.tsv dbsnp-154-orig.vcf.gz |\
bcftools sort -m 1G -Oz -T . -o dbsnp-154.vcf.gz && \ tabix -f -p vcf -C dbsnp-154.vcf.gz Writing to .BPxTgX [buf_flush] Error: cannot write to .BPxTgX/00039.bcf Cleaning
It's already a great help, but I hope you give me another piece of advice......:(
no worries, let us continue to debug!
could you please show the output of
which -a bcftools
which -a tabix
Thank you so much for your reply. I was going to go to bed, but I'm glad I didn't....
(base) ngs@basiclab01:~$ PATH=$PATH:~/bcbio/tools/bin (base) ngs@basiclab01:~$ PATH=$PATH:~/bcbio/anaconda/bin (base) ngs@basiclab01:~$ which -a bcftools /home/ngs/bcbio/tools/bin/bcftools /home/ngs/bcbio/anaconda/bin/bcftools (base) ngs@basiclab01:~$ which -a tabix /home/ngs/bcbio/anaconda/bin/tabix
here!!
I don't see any issues here, however, I have some minor comments:
it is better to set up your PATH in ~/.bash_profile this way:
export PATH=/home/ngs/bcbio/anaconda/bin:/home/ngs/bcbio/tools/bin:$PATH
Now, please try
$ ls -l /home/ngs/bcbio/tools/bin/bcftools
$ bcftools
$ tabix
(just to make sure your bcftools and tabix are working)
Then make sure you have enough space in your /home/ngs
$ df -h
(some servers/clusters have limited space in /home/ but a large volume mounted in /share or elsewhere).
SN
Thank you for sincerely answering even the rudimentary questions every time. You're the only one I can trust right now, so I'm more desperate. I hope you give me more advice. plz... :(
(base) ngs@basiclab01:~$ bcftools
Program: bcftools (Tools for variant calling and manipulating VCFs and BCFs) License: GNU GPLv3+, due to use of the GNU Scientific Library Version: 1.13 (using htslib 1.13)
Usage: bcftools [--version|--version-only] [--help]
Commands:
-- Indexing index index VCF/BCF files
-- VCF/BCF manipulation annotate annotate and edit VCF/BCF files concat concatenate VCF/BCF files from the same set of samples convert convert VCF/BCF files to different formats and back isec intersections of VCF/BCF files merge merge VCF/BCF files files from non-overlapping sample sets norm left-align and normalize indels plugin user-defined plugins query transform VCF/BCF into user-defined formats reheader modify VCF/BCF header, change sample names sort sort VCF/BCF file view VCF/BCF conversion, view, subset and filter VCF/BCF files
-- VCF/BCF analysis call SNP/indel calling consensus create consensus sequence by applying VCF variants cnv HMM CNV calling csq call variation consequences filter filter VCF/BCF files using fixed thresholds gtcheck check sample concordance, detect sample swaps and contamination mpileup multi-way pileup producing genotype likelihoods polysomy detect number of chromosomal copies roh identify runs of autozygosity (HMM) stats produce VCF/BCF stats
Most commands accept VCF, bgzipped VCF, and BCF with the file type detected automatically even when streaming from a pipe. Indexed VCF and BCF will work in all situations. Un-indexed VCF and BCF and streams will work in most but not all situations.
(base) ngs@basiclab01:~$ tabix
Version: 1.13 Usage: tabix [OPTIONS] [FILE] [REGION [...]]
Indexing Options: -0, --zero-based coordinates are zero-based -b, --begin INT column number for region start [4] -c, --comment CHAR skip comment lines starting with CHAR [null] -C, --csi generate CSI index for VCF (default is TBI) -e, --end INT column number for region end (if no end, set INT to -b) [5] -f, --force overwrite existing index without asking -m, --min-shift INT set minimal interval size for CSI indices to 2^INT [14] -p, --preset STR gff, bed, sam, vcf -s, --sequence INT column number for sequence names (suppressed by -p) [1] -S, --skip-lines INT skip first INT lines [0]
Querying and other options: -h, --print-header print also the header lines -H, --only-header print only the header lines -l, --list-chroms list chromosome names -r, --reheader FILE replace the header with the content of FILE -R, --regions FILE restrict to regions listed in the file -T, --targets FILE similar to -R but streams rather than index-jumps -D do not download the index file --cache INT set cache size to INT megabytes (0 disables) [10] --separate-regions separate the output by corresponding regions --verbosity INT set verbosity [3]
(base) ngs@basiclab01:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 126G 0 126G 0% /dev
tmpfs 26G 2.6G 23G 11% /run
/dev/nvme0n1p3 92G 32G 56G 37% /
tmpfs 126G 216K 126G 1% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 126G 0 126G 0% /sys/fs/cgroup
/dev/nvme0n1p2 945M 207M 674M 24% /boot
/dev/nvme0n1p4 819G 762G 16G 99% /home
/dev/sda1 37T 4.1T 33T 12% /exdisk/sda1
tmpfs 26G 96K 26G 1% /run/user/1002
tmpfs 26G 0 26G 0% /run/user/1004
tmpfs 26G 0 26G 0% /run/user/1007
I see - you have just 16G of disk space left in
/dev/nvme0n1p4 819G 762G 16G 99% /home
dbsnp is a big file - almost 17G and you need more free space to process it, see in our installation:
-rw-r--r-- 1 bcbio bcbio 17G Jan 13 2021 dbsnp-154.vcf.gz
-rw-r--r-- 1 bcbio bcbio 2.4M Jan 13 2021 dbsnp-154.vcf.gz.csi
-rw-r--r-- 1 bcbio bcbio 2.9M Jan 13 2021 dbsnp-154.vcf.gz.tbi
Some bcbio installation scripts (gnomad for example) require up to 1T of temporary space.
So make sure you have enough space in your partition.
You have such partition in your system:
/dev/sda1 37T 4.1T 33T 12% /exdisk/sda1
Try to install bcbio in
/exdisk/sda1/bcbio
instead of /home/ngs/bcbio/
Thank you always for your good answers. After explaining the situation, I am asking for understanding from the professor and installing it in a new directory. But errors that didn't come out in the last directory are coming up.**
ngs@basiclab01:/exdisk/sda1$ python3 bcbio_nextgen_install.py /exdisk/sda1/bcbio --tooldir=/exdisk/sda1/bcbio/tools --mamba Checking required dependencies Installing isolated base python installation Installing {conda_bin} Collecting package metadata (current_repodata.json): done Solving environment: done
Installing conda-build
__ __ __ __
/ \ / \ / \ / \
/ \/ \/ \/ \
███████████████/ /██/ /██/ /██/ /████████████████████████ / / \ / \ / \ / \ _ / / _/ _/ _/ \ o _, / _/ ___/ ` |/ ███╗ ███╗ █████╗ ███╗ ███╗██████╗ █████╗ ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗ ██╔████╔██║███████║██╔████╔██║██████╔╝███████║ ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║ ██║ ╚═╝ ██║██║ ██║██║ ╚═╝ ██║██████╔╝██║ ██║ ╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝
mamba (0.19.0) supported by @QuantStack
GitHub: https://github.com/mamba-org/mamba
Twitter: https://twitter.com/QuantStack
█████████████████████████████████████████████████████████████
Looking for: ['conda-build']
pkgs/main/linux-64 [====================] (00m:00s) No change pkgs/main/noarch [====================] (00m:00s) No change bioconda/linux-64 [====================] (00m:00s) No change pkgs/r/noarch [====================] (00m:00s) No change bioconda/noarch [====================] (00m:00s) No change pkgs/r/linux-64 [====================] (00m:00s) No change conda-forge/noarch [====================] (00m:01s) Done conda-forge/linux-64 [====================] (00m:07s) Done
Pinned packages:
Transaction
Prefix: /exdisk/sda1/bcbio/anaconda
All requested packages already installed
Installing bcbio-nextgen --2021-12-08 23:58:28-- https://raw.githubusercontent.com/bcbio/bcbio-nextgen/master/requirements-conda.txt Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.109.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 20 [text/plain] Saving to: ‘requirements-conda.txt’
requirements-conda.txt 100%[=======================================================>] 20 --.-KB/s in 0s
2021-12-08 23:58:28 (986 KB/s) - ‘requirements-conda.txt’ saved [20/20]
__ __ __ __
/ \ / \ / \ / \
/ \/ \/ \/ \
███████████████/ /██/ /██/ /██/ /████████████████████████ / / \ / \ / \ / \ _ / / _/ _/ _/ \ o _, / _/ ___/ ` |/ ███╗ ███╗ █████╗ ███╗ ███╗██████╗ █████╗ ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗ ██╔████╔██║███████║██╔████╔██║██████╔╝███████║ ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║ ██║ ╚═╝ ██║██║ ██║██║ ╚═╝ ██║██████╔╝██║ ██║ ╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝
mamba (0.19.0) supported by @QuantStack
GitHub: https://github.com/mamba-org/mamba
Twitter: https://twitter.com/QuantStack
█████████████████████████████████████████████████████████████
Looking for: ['bcbio-nextgen']
conda-forge/linux-64 Using cache conda-forge/noarch Using cache bioconda/linux-64 Using cache bioconda/noarch Using cache pkgs/main/linux-64 Using cache pkgs/main/noarch Using cache pkgs/r/linux-64 Using cache pkgs/r/noarch Using cache
Pinned packages:
Transaction
Prefix: /exdisk/sda1/bcbio/anaconda
All requested packages already installed
Collecting package metadata (current_repodata.json): done Solving environment: done
Installing data and third party dependencies Upgrading bcbio Upgrading third party tools to latest versions --2021-12-08 23:58:48-- https://github.com/chapmanb/cloudbiolinux/archive/master.tar.gz Resolving github.com (github.com)... 52.78.231.108 Connecting to github.com (github.com)|52.78.231.108|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://codeload.github.com/chapmanb/cloudbiolinux/tar.gz/master [following] --2021-12-08 23:58:48-- https://codeload.github.com/chapmanb/cloudbiolinux/tar.gz/master Resolving codeload.github.com (codeload.github.com)... 13.124.75.135 Connecting to codeload.github.com (codeload.github.com)|13.124.75.135|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/x-gzip] Saving to: ‘STDOUT’
0K ........ ........ ........ ........ ........ ........ 10.8M
3072K ........ ........ ........ ....... 13.9M=0.4s
2021-12-08 23:58:49 (11.8 MB/s) - written to stdout [5193868]
Reading packages from /exdisk/sda1/tmpbcbio-install/cloudbiolinux/contrib/flavor/ngs_pipeline_minimal/packages-conda.yaml Checking for problematic or migrated packages in default environment Initalling initial set of packages for default environment with mamba
Encountered problems while solving:
Collecting package metadata (current_repodata.json): ...working... done Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): ...working...
After 30 minutes, errors keep popping up. What's wrong with me? Compared to the help you give me, I think I'm so incompetent.
Thanks for the testing, you are close!
Sorry for this issue - this one was introduced just yesterday, and I've not yet fixed that. It is due to purecn upgrade >=2.0.1 which requires bioconductor-rhdf5=2.38 while we have it pinned to 2.36 due to wasabi issue.
Now I've relaxed the pinning, so it should be installing ok, please try.
There might be a wasabi issue after that in the RNA-seq pipeline - I am working to eliminate it without downgrading bioconductor-rhdf5.
You're so quick to fix your makeup! Thank you so much. I've been doing it since then, but it's too long, so I'll skip the middle part.
/exdisk/sda1$ bcbio_nextgen.py upgrade -u skip --genomes hg38 Upgrading bcbio Upgrading bcbio-nextgen data files List of genomes to get (from the config file at '{'genomes': [{'dbkey': 'hg38', 'name': 'Human (hg38) full', 'indexes': ['seq', 'twobit', 'bwa', 'hisat2'], 'annotations': ['ccds', 'capture_regions', 'coverage', 'prioritize', 'dbsnp', 'hapmap_snps', '1000g_omni_snps', 'ACMG56_genes', '1000g_snps', 'mills_indels', '1000g_indels', 'clinvar', 'qsignature', 'genesplicer', 'effects_transcripts', 'varpon', 'vcfanno', 'viral', 'purecn_mappability', 'simple_repeat', 'af_only_gnomad', 'transcripts', 'RADAR', 'rmsk', 'salmon-decoys', 'fusion-blacklist', 'mirbase'], 'validation': ['giab-NA12878', 'giab-NA24385', 'giab-NA24631', 'platinum-genome-NA12878', 'giab-NA12878-remap', 'giab-NA12878-crossmap', 'dream-syn4-crossmap', 'dream-syn3-crossmap', 'giab-NA12878-NA24385-somatic', 'giab-NA24143', 'giab-NA24149', 'giab-NA24694', 'giab-NA24695']}], 'genome_indexes': ['rtg'], 'install_liftover': False, 'install_uniref': False}'): Human (hg38) full Installing snpEff database GRCh38.99 in /exdisk/sda1/bcbio/genomes/Hsapiens/hg38/snpeff --2021-12-09 12:58:43-- http://downloads.sourceforge.net/project/snpeff/databases/v5_0/snpEff_v5_0_GRCh38.99.zip Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2021-12-09 12:58:44 ERROR 404: Not Found.
Traceback (most recent call last):
File "/exdisk/sda1/bcbio/tools/bin/bcbio_nextgen.py", line 228, in
Is it really ending now? I'm sorry for asking too many questions......:(
I saw this comment. https://github.com/bcbio/bcbio-nextgen/issues/3570
ngs@basiclab01:/exdisk/sda1/bcbio/genomes/Hsapiens$ snpEff -version SnpEff 5.0e 2021-03-09 ngs@basiclab01:/exdisk/sda1$ ls bcbio/genomes/Hsapiens/snpeff/hg38k ls: cannot access 'bcbio/genomes/Hsapiens/snpeff/hg38k': No such file or directory ngs@basiclab01:/exdisk/sda1$ ls bcbio/genomes/Hsapiens/snpeff/ ls: cannot access 'bcbio/genomes/Hsapiens/snpeff/': No such file or directory ngs@basiclab01:/exdisk/sda1$ cd bcbio/genomes/Hsapiens ngs@basiclab01:/exdisk/sda1/bcbio/genomes/Hsapiens$ ls hg38
what should I do?!
you have the right snpEff binary, now you need to install snpeff5.0 databases. To do so, you need to upgrade bcbio code as well:
bcbio_nextgen.py upgrade -u development --genomes hg38
then your URL should change from sourceforge.net to the microsoft cloud, see here: https://github.com/bcbio/bcbio-nextgen/blob/master/bcbio/install.py#L38
I tried it yesterday, and it worked fine for me.
Thank you for your kind answers. Why doesn't my address change?
--2021-12-10 12:14:00-- http://downloads.sourceforge.net/project/snpeff/databases/v5_0/snpEff_v5_0_GRCh38.99.zip Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2021-12-10 12:14:01 ERROR 404: Not Found.
Traceback (most recent call last):
File "/exdisk/sda1/bcbio/anaconda/bin/bcbio_nextgen.py", line 228, in
try first to upgrade bcbio code without data?
bcbio_nextgen.py upgrade -u development
I commanded that code..
ngs@basiclab01:/exdisk/sda1$ bcbio_nextgen.py upgrade -u development Upgrading bcbio Detected 1.2.8 as latest version of bcbio-nextgen on bioconda. bcbio version 1.2.8 is newer than the conda version 1.2.8, skipping upgrade from conda Upgrading bcbio-nextgen to latest development version Collecting bcbio-nextgen Cloning https://github.com/bcbio/bcbio-nextgen.git to /tmp/pip-install-yvpvm93z/bcbio-nextgen_966db296f13c428a806a77fc31427020 Running command git clone -q https://github.com/bcbio/bcbio-nextgen.git /tmp/pip-install-yvpvm93z/bcbio-nextgen_966db296f13c428a806a77fc31427020 Upgrade of bcbio-nextgen development code complete. Upgrading third party tools to latest versions Reading packages from /exdisk/sda1/tmpbcbio-install/cloudbiolinux/contrib/flavor/ngs_pipeline_minimal/packages-conda.yaml Checking for problematic or migrated packages in default environment Found packages that moved from default environment: cancerit-allelecount, r-tidyverse
__ __ __ __
/ \ / \ / \ / \
/ \/ \/ \/ \
███████████████/ /██/ /██/ /██/ /████████████████████████ / / \ / \ / \ / \ _ / / _/ _/ _/ \ o _, / _/ ___/ ` |/ ███╗ ███╗ █████╗ ███╗ ███╗██████╗ █████╗ ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗ ██╔████╔██║███████║██╔████╔██║██████╔╝███████║ ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║ ██║ ╚═╝ ██║██║ ██║██║ ╚═╝ ██║██████╔╝██║ ██║ ╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝
mamba (0.19.0) supported by @QuantStack
GitHub: https://github.com/mamba-org/mamba
Twitter: https://twitter.com/QuantStack
█████████████████████████████████████████████████████████████
Removing specs: ['cancerit-allelecount', 'r-tidyverse'] Transaction
Prefix: /exdisk/sda1/bcbio/anaconda
Removing specs:
r-tidyverse
Package Version Build Channel Size ──────────────────────────────────────────────────────────────────────────────────── Remove: ────────────────────────────────────────────────────────────────────────────────────
Summary:
Remove: 52 packages
Total download: 0 B
────────────────────────────────────────────────────────────────────────────────────
Preparing transaction: done Verifying transaction: done Executing transaction: done
Package Version Build Channel Size ───────────────────────────────────────────────────────────────────────────────────────────────── Install: ─────────────────────────────────────────────────────────────────────────────────────────────────
r-vroom 1.5.7 r41h03ef668_0 conda-forge/linux-64 Cached
Summary:
Install: 52 packages
Total download: 0 B
─────────────────────────────────────────────────────────────────────────────────────────────────
Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done
Creating manifest of installed packages in /exdisk/sda1/bcbio/manifest Third party tools upgrade complete. Upgrade completed successfully. ngs@basiclab01:/exdisk/sda1$ bcbio_nextgen.py upgrade -u skip --genomes hg38 --aligners bwa Upgrading bcbio Upgrading bcbio-nextgen data files --2021-12-10 13:17:25-- https://github.com/chapmanb/cloudbiolinux/archive/master.tar.gz Resolving github.com (github.com)... 15.164.81.167 Connecting to github.com (github.com)|15.164.81.167|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://codeload.github.com/chapmanb/cloudbiolinux/tar.gz/master [following] --2021-12-10 13:17:25-- https://codeload.github.com/chapmanb/cloudbiolinux/tar.gz/master Resolving codeload.github.com (codeload.github.com)... 13.124.75.135 Connecting to codeload.github.com (codeload.github.com)|13.124.75.135|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/x-gzip] Saving to: ‘STDOUT’
0K ........ ........ ........ ........ ........ ........ 2.24M
3072K ........ ........ ........ ....... 9.54M=1.5s
2021-12-10 13:17:27 (3.21 MB/s) - written to stdout [5194255]
List of genomes to get (from the config file at '{'genomes': [{'dbkey': 'hg38', 'name': 'Human (hg38) full', 'indexes': ['seq', 'twobit', 'bwa', 'hisat2'], 'annotations': ['ccds', 'capture_regions', 'coverage', 'prioritize', 'dbsnp', 'hapmap_snps', '1000g_omni_snps', 'ACMG56_genes', '1000g_snps', 'mills_indels', '1000g_indels', 'clinvar', 'qsignature', 'genesplicer', 'effects_transcripts', 'varpon', 'vcfanno', 'viral', 'purecn_mappability', 'simple_repeat', 'af_only_gnomad', 'transcripts', 'RADAR', 'rmsk', 'salmon-decoys', 'fusion-blacklist', 'mirbase'], 'validation': ['giab-NA12878', 'giab-NA24385', 'giab-NA24631', 'platinum-genome-NA12878', 'giab-NA12878-remap', 'giab-NA12878-crossmap', 'dream-syn4-crossmap', 'dream-syn3-crossmap', 'giab-NA12878-NA24385-somatic', 'giab-NA24143', 'giab-NA24149', 'giab-NA24694', 'giab-NA24695']}], 'genome_indexes': ['bwa', 'rtg'], 'install_liftover': False, 'install_uniref': False}'): Human (hg38) full Installing snpEff database GRCh38.99 in /exdisk/sda1/bcbio/genomes/Hsapiens/hg38/snpeff --2021-12-10 13:17:28-- http://downloads.sourceforge.net/project/snpeff/databases/v5_0/snpEff_v5_0_GRCh38.99.zip Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2021-12-10 13:17:28 ERROR 404: Not Found.
Traceback (most recent call last):
File "/exdisk/sda1/bcbio/anaconda/bin/bcbio_nextgen.py", line 228, in
interesting, it looks like pip is acting out - you did not get the update. you may try: https://bcbio-nextgen.readthedocs.io/en/latest/contents/development.html#injecting-bcbio-code-into-bcbio-installation
# delete all starting with bcbio in
# /exdisk/sda1/bcbio/anaconda/lib/python3.7/site-packages/bcbio*
# clone bcbio:
cd
git clone https://github.com/bcbio/bcbio-nextgen.git
cd bcbio-nextgen
# make sure your current python is bcbio python
which python
# inject bcbio
python setup.py install
# run the data upgrade
bcbio_nextgen.py upgrade -u skip --genomes hg38
the update process is improved a bit: https://github.com/bcbio/bcbio-nextgen/pull/3583
Thank you for taking care of me! I think the installation was completed well.
Now https://bcbio-nextgen.readthedocs.io/en/latest/contents/bulk_rnaseq.html
I'm practicing bulk_rnaseq based on and I'll ask you another question. Thank you so much.
I am a student who started my internship at graduate school and is building a new server.
I'm going to install bcbio according to the link recommended by the professor, but I run into the same problem for 4 days.
I know you're busy, but can you give me some advice?
ngs@basiclab01:~$ ./bcbio_nextgen.py upgrade -u skip --genomes hg38 Upgrading bcbio Upgrading bcbio-nextgen data files List of genomes to get (from the config file at '{'genomes': [{'dbkey': 'hg38', 'name': 'Human (hg38) full', 'indexes': ['seq', 'twobit', 'bwa', 'hisat2'], 'annotations': ['ccds', 'capture_regions', 'coverage', 'prioritize', 'dbsnp', 'hapmap_snps', '1000g_omni_snps', 'ACMG56_genes', '1000g_snps', 'mills_indels', '1000g_indels', 'clinvar', 'qsignature', 'genesplicer', 'effects_transcripts', 'varpon', 'vcfanno', 'viral', 'purecn_mappability', 'simple_repeat', 'af_only_gnomad', 'transcripts', 'RADAR', 'rmsk', 'salmon-decoys', 'fusion-blacklist', 'mirbase'], 'validation': ['giab-NA12878', 'giab-NA24385', 'giab-NA24631', 'platinum-genome-NA12878', 'giab-NA12878-remap', 'giab-NA12878-crossmap', 'dream-syn4-crossmap', 'dream-syn3-crossmap', 'giab-NA12878-NA24385-somatic', 'giab-NA24143', 'giab-NA24149', 'giab-NA24694', 'giab-NA24695']}], 'genome_indexes': ['bwa', 'rtg'], 'install_liftover': False, 'install_uniref': False}'): Human (hg38) full Running GGD recipe: hg38 dbsnp 154-20210112 URL transformed to HTTPS due to an HSTS policy --2021-12-06 16:49:29-- https://ftp.ncbi.nih.gov/snp/archive/b154/VCF/GCF_000001405.38.gz Resolving ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)... 165.112.9.230, 165.112.9.229, 2607:f220:41e:250::11, ... Connecting to ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)|165.112.9.230|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable
URL transformed to HTTPS due to an HSTS policy --2021-12-06 16:49:30-- https://ftp.ncbi.nih.gov/snp/archive/b154/VCF/GCF_000001405.38.gz.tbi Resolving ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)... 165.112.9.230, 165.112.9.229, 2607:f220:41f:250::229, ... Connecting to ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)|165.112.9.230|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable
Writing to .HszlH8 [buf_flush] Error: cannot write to .HszlH8/00097.bcf Cleaning tbx_index_build failed: dbsnp-154.vcf.gz Traceback (most recent call last): File "./bcbio_nextgen.py", line 228, in
install.upgrade_bcbio(kwargs["args"])
File "/home/ngs/bcbio/anaconda/lib/python3.7/site-packages/bcbio/install.py", line 107, in upgrade_bcbio
upgrade_bcbio_data(args, REMOTES)
File "/home/ngs/bcbio/anaconda/lib/python3.7/site-packages/bcbio/install.py", line 359, in upgrade_bcbio_data
args.cores, ["ggd", "s3", "raw"])
File "/home/ngs/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/genomes.py", line 354, in install_data_local
_prep_genomes(env, genomes, genome_indexes, ready_approaches, data_filedir)
File "/home/ngs/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/genomes.py", line 480, in _prep_genomes
retrieve_fn(env, manager, gid, idx)
File "/home/ngs/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/genomes.py", line 875, in _install_with_ggd
ggd.install_recipe(os.getcwd(), env.system_install, recipe_file, gid)
File "/home/ngs/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/ggd.py", line 30, in install_recipe
recipe["recipe"]["full"]["recipe_type"], system_install)
File "/home/ngs/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/ggd.py", line 62, in _run_recipe
subprocess.check_output(["bash", run_file])
File "/home/ngs/bcbio/anaconda/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/home/ngs/bcbio/anaconda/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['bash', '/home/ngs/bcbio/genomes/Hsapiens/hg38/txtmp/ggd-run.sh']' returned non-zero exit status 1.
I always get stuck in this part.