bcbio 1.2.8 and 1.2.9 installation problem. snpeff returns 404 with or without dev flag

[lento2002]

Version info bcbio 1.2.8 and 1.2.9.
CentOS Linux release 7.6.1810 (Core)

To Reproduce python3 bcbio_nextgen_install.py /mnt/fsx/share/bcbio --tooldir=/mnt/fsx/usr/local --genomes hg38 --aligners star --cores 32

also tried...

python3 bcbio_nextgen_install.py --upgrade="development" /mnt/fsx/share/bcbio --tooldir=/mnt/fsx/usr/local --genomes hg38 --aligners star --cores 32

Resultant output with installer stopping is below:

List of genomes to get (from the config file at '{'genomes': [{'dbkey': 'hg38', 'name': 'Human (hg38) full', 'indexes': ['seq', 'twobit', 'bwa', 'hisat2'], 'annotations': ['ccds', 'capture_regions', 'coverage', 'prioritize', 'dbsnp', 'hapmap_snps', '1000g_omni_snps', 'ACMG56_genes', '1000g_snps', 'mills_indels', '1000g_indels', 'clinvar', 'qsignature', 'genesplicer', 'effects_transcripts', 'varpon', 'vcfanno', 'viral', 'purecn_mappability', 'simple_repeat', 'af_only_gnomad', 'transcripts', 'RADAR', 'rmsk', 'salmon-decoys', 'fusion-blacklist', 'mirbase'], 'validation': ['giab-NA12878', 'giab-NA24385', 'giab-NA24631', 'platinum-genome-NA12878', 'giab-NA12878-remap', 'giab-NA12878-crossmap', 'dream-syn4-crossmap', 'dream-syn3-crossmap', 'giab-NA12878-NA24385-somatic', 'giab-NA24143', 'giab-NA24149', 'giab-NA24694', 'giab-NA24695']}], 'genome_indexes': ['star', 'rtg'], 'install_liftover': False, 'install_uniref': False}'): Human (hg38) full Installing snpEff database GRCh38.99 in /mnt/fsx/share/bcbio/genomes/Hsapiens/hg38/snpeff --2021-12-09 06:20:20-- http://downloads.sourceforge.net/project/snpeff/databases/v5_0/snpEff_v5_0_GRCh38.99.zip Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2021-12-09 06:20:20 ERROR 404: Not Found.

Traceback (most recent call last): File "/mnt/fsx/share/bcbio/anaconda/bin/bcbio_nextgen.py", line 228, in install.upgrade_bcbio(kwargs["args"]) File "/mnt/fsx/share/bcbio/anaconda/lib/python3.7/site-packages/bcbio/install.py", line 107, in upgrade_bcbio upgrade_bcbio_data(args, REMOTES) File "/mnt/fsx/share/bcbio/anaconda/lib/python3.7/site-packages/bcbio/install.py", line 361, in upgrade_bcbio_data _upgrade_snpeff_data(galaxy_home, args, remotes) File "/mnt/fsx/share/bcbio/anaconda/lib/python3.7/site-packages/bcbio/install.py", line 439, in _upgrade_snpeff_data subprocess.check_call(["wget", "--no-check-certificate", "-c", "-O", dl_file, dl_url]) File "/mnt/fsx/share/bcbio/anaconda/lib/python3.7/subprocess.py", line 363, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['wget', '--no-check-certificate', '-c', '-O', 'snpEff_v5_0_GRCh38.99.zip', 'http://downloads.sourceforge.net/project/snpeff/databases/v5_0/snpEff_v5_0_GRCh38.99.zip']' returned non-zero exit status 8. Traceback (most recent call last): File "bcbio_nextgen_install.py", line 304, in main(parser.parse_args(), sys.argv[1:]) File "bcbio_nextgen_install.py", line 55, in main subprocess.check_call([bcbio, "upgrade"] + _clean_args(sys_argv, args)) File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/mnt/fsx/share/bcbio/anaconda/bin/bcbio_nextgen.py', 'upgrade', '--upgrade=development', '--tooldir=/mnt/fsx/usr/local', '--genomes', 'hg38', '--aligners', 'star', '--cores', '32', '--data']' returned non-zero exit status 1.

Greatly appreciate any help insight. It's always crashing at snpeff step with 404 not found. Indeed the link does not work when manually following the link and looking for same file in sourceforge.

[naumenko-sa]

yes, snpeff moved the databases to the Microsoft cloud in snpeff5.0 https://github.com/bcbio/bcbio-nextgen/issues/3570

To fetch them upgrade bcbio code please:

(updating the existing installation)

bcbio_nextgen.py upgrade -u development --genomes hg38

(new installation)

python3 bcbio_nextgen_install.py --nodata .....where to install
# here you may just softlink the data folder from the old installation
bcbio_nextgen.py upgrade -u development --genomes hg38

[lento2002]

ok thanks. i'm running into weirdness still. after the upgrade with --nodata i see the below; but the version doesnt check out to 1.2.9 ... Any thoughts?

beginning part of run looks like this [root@ip-10-0-0-58 bcbio]# bcbio_nextgen.py upgrade -u development Upgrading bcbio Detected 1.2.8 as latest version of bcbio-nextgen on bioconda. bcbio version 1.2.8 is newer than the conda version 1.2.8, skipping upgrade from conda Upgrading bcbio-nextgen to latest development version Collecting bcbio-nextgen Cloning https://github.com/bcbio/bcbio-nextgen.git to /tmp/pip-install-qq2pe3to/bcbio-nextgen_138091867f4b46c2b820a3a30ca8ebfd Running command git clone -q https://github.com/bcbio/bcbio-nextgen.git /tmp/pip-install-qq2pe3to/bcbio-nextgen_138091867f4b46c2b820a3a30ca8ebfd WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv Upgrade of bcbio-nextgen development code complete. Upgrading third party tools to latest versions Reading packages from /mnt/fsx/bcbio/tmpbcbio-install/cloudbiolinux/contrib/flavor/ngs_pipeline_minimal/packages-conda.yaml Checking for problematic or migrated packages in default environment

end looks like this Creating manifest of installed packages in /mnt/fsx/share/bcbio/manifest Third party tools upgrade complete. Upgrade completed successfully. [root@ip-10-0-0-58 bcbio]# bcbio_nextgen.py -v 1.2.8 [root@ip-10-0-0-58 bcbio]#

[lento2002]

ok i figured it out; many thanks. was able to reference this...from a related ticket #3571

delete all starting with bcbio in

/exdisk/sda1/bcbio/anaconda/lib/python3.7/site-packages/bcbio*

clone bcbio:

cd git clone https://github.com/bcbio/bcbio-nextgen.git cd bcbio-nextgen

make sure your current python is bcbio python

which python

inject bcbio (run from the dir created from the github pull)

python setup.py install

run the data upgrade

bcbio_nextgen.py upgrade -u skip --genomes hg38

[naumenko-sa]

Thanks, good to know! The update process was improved a bit: https://github.com/bcbio/bcbio-nextgen/pull/3583