search

ablab / quast

Genome assembly evaluation tool
http://quast.sf.net
Other
395 stars 76 forks source link

Quast error about internet connection #8

Closed shaman-narayanasamy closed 8 years ago

shaman-narayanasamy commented 9 years ago

Dear authors,

I am attempting to run quast on some metagenomic assemblies. It seems that quast is able to download some of the references, however it terminates after a certain point, even if I repeat the command.

Logging to /output/Analysis/results/quast/metaquast.log

Contigs:

No references are provided, starting to search for reference genomes in SILVA rRNA database and to download them from NCBI...

2015-09-01 13:28:36
  Using existing BLAST alignments for MGMT.assembly.merged... 

Trying to use previously downloaded references...

2015-09-01 13:28:36
Trying to download found references from NCBI. Totally 46 organisms to try.
  Candidatus_Microthrix_parvicella_RN1               | was downloaded previously (total 1, 45 more to go)
  Dechloromonas_sp._SIUL                             | not found in the NCBI database
  Trachelomonas_volvocinopsis_var._spiralis          | not found in the NCBI database

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0".

I am unable to find any other logs files to further troubleshoot this problem... I would be happy to send over my contig fasta file so that you may be able to test it yourself.

Best, Shaman

shaman-narayanasamy commented 8 years ago

Looks like this was actually a network problem.

kubu4 commented 5 years ago

I realize this issue is marked closed, but I've just encountered this error, and thought it would better to post here, rather than open a new issue. Anyway, I definitely have internet access, so I don't have any idea why that error's being thrown. Any suggestions on where to look or how to proceed would be greatly appreciated.

Here're the contents of the metaquast.log:

/home/sam/programs/quast-5.0.2/metaquast.py /home/sam/data/metagenomics/P_generosa/final.contigs.fa --threads 22

Version: 5.0.2

System information:
  OS: Linux-4.4.0-139-generic-x86_64-with-debian-stretch-sid (linux_64)
  Python version: 3.7.2
  CPUs number: 24

Started: 2019-01-10 11:10:58

Logging to /home/sam/analyses/20190110_geo_metagnomics_metaquast/quast_results/results_2019_01_10_11_10_58/metaquast.log

Contigs:
  Pre-processing...
  /home/sam/data/metagenomics/P_generosa/final.contigs.fa ==> final.contigs

No references are provided, starting to search for reference genomes in SILVA 16S rRNA database and to download them from NCBI...

2019-01-10 11:12:29
Running BlastN..
  processing final.contigs
/home/sam/programs/quast-5.0.2/quast_libs/blast/blastn -query /home/sam/data/metagenomics/P_generosa/final.contigs.fa \
-db /home/sam/programs/quast-5.0.2/quast_libs/silva/silva.123.db -outfmt 7 -num_threads \
22 > quast_results/results_2019_01_10_11_10_58/quast_downloaded_references/blast.res_final-contigs \
2>> quast_results/results_2019_01_10_11_10_58/quast_downloaded_references/blast.err
  BLAST results for final.contigs are saved to /home/sam/analyses/20190110_geo_metagnomics_metaquast/quast_results/results_2019_01_10_11_10_58/quast_downloaded_references/blast.res_final-contigs...

2019-01-10 11:56:47
Trying to download found references from NCBI. Totally 50 organisms to try.
  unidentified_proteobacterium                                          | not found in the NCBI database
  unidentified_proteobacterium was not found in NCBI database, trying to download the next best match

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0".

In case you have troubles running QUAST, you can write to quast.support@cab.spbu.ru
or report an issue on our GitHub repository https://github.com/ablab/quast/issues
Please provide us with quast.log file from the output directory.
kdiverson commented 5 years ago

I'm having the same issue. I'm running metaquast on an AWS EC2 instance which has internet access. I have been running it successfully until about last week. Could there be a change to the NCBI API?

kubu4 commented 5 years ago

Was also wondering if government shutdown was possibly impacting access?

ChunfangZheng commented 5 years ago

I fixed the problem by adding "time.sleep(1)" to the source code quast_lib/search_references_meta.py method "def try_send_request(url)".
NCBI recommends that users post no more than three URL requests per second; the time.sleep(1) in the method try_send_request(url) slows the speed of sending requests.

kubu4 commented 5 years ago

I fixed the problem by adding "time.sleep(1)" to the source code quast_lib/search_references_meta.py method "def try_send_request(url)". NCBI recommends that users post no more than three URL requests per second; the time.sleep(1) in the method try_send_request(url) slows the speed of sending requests.

Thanks for this! This alleviated the issue for me.

kubu4 commented 5 years ago

Thanks for this! This alleviated the issue for me.

Never mind. Got too excited, too quickly! Although I didn't get an NCBI connection error, I did get a NameError: name 'time' is not defined. Is there another line I need to add to define time()?

ChunfangZheng commented 5 years ago

import time at the beginning of the file

from __future__ import with_statement
import os
import shlex
import shutil
import re
from collections import defaultdict
import time
from os.path import isdir, isfile, join
kubu4 commented 5 years ago

In order to alleviate the NameError: name 'time' is not defined error, I think you also need to import the time module (e.g.):

import time

Insert that line somewhere at the top of the script with the other import commands should fix it.

holtjma commented 5 years ago

I randomly happened to run into this issue at the same time as you guys; the time import and insertion of time.sleep(1) worked for me as well.

alexeigurevich commented 5 years ago

Thanks @ChunfangZheng, @kubu4 and @holtjma -- we added import time and time.sleep(1) solution to the master branch (will be available in the next release).

By the way, if you find a problem related to a closed issue feel free to reopen it!

microbiomeparty commented 4 years ago

Hi, sorry if I'm missing the resolution to this elsewhere but I'm running quest 5.0.2 in metaquast mode and still getting this issue downloading references from NCBI. My network isn't having any issues that I can find, is there another workaround that you might suggest? Thanks!

Version: 5.0.2

System information: OS: Linux-3.10.0-693.21.1.el7.x86_64-x86_64-with-centos-7.4.1708-Core (linux_64) Python version: 3.6.7 CPUs number: 32

Started: 2019-11-20 08:35:15

Logging to /home/mwoodwo/prj/PREMIX/mg/03.quast_output/metaquast.log NOTICE: Output directory already exists and looks like a QUAST output dir. Existing results can be reused (e.g. previously generated alignments)!

Contigs: Pre-processing... 1 ./PM1-S1/contigs.fasta ==> PM1-S1_contigs 2 ./PM1-S2/contigs.fasta ==> PM1-S2_contigs 3 ./PM1-S3/contigs.fasta ==> PM1-S3_contigs 4 ./PM1-C1D01/contigs.fasta ==> PM1-C1D01_contigs 5 ./PM1-C1D02/contigs.fasta ==> PM1-C1D02_contigs 6 ./PM1-C1D15/contigs.fasta ==> PM1-C1D15_contigs 7 ./PM1-C1D36/contigs.fasta ==> PM1-C1D36_contigs 8 ./PM3-S1/contigs.fasta ==> PM3-S1_contigs 9 ./PM3-C0D15/contigs.fasta ==> PM3-C0D15_contigs 10 ./PM3-C0D1/contigs.fasta ==> PM3-C0D1_contigs 11 ./PM3-C0D2/contigs.fasta ==> PM3-C0D2_contigs 12 ./PM3-C0D36/contigs.fasta ==> PM3-C0D36_contigs 13 ./SD-01/contigs.fasta ==> SD-01_contigs

No references are provided, starting to search for reference genomes in SILVA 16S rRNA database and to download them from NCBI...

2019-11-20 08:35:28 Using existing BLAST alignments for PM1-S1_contigs... Using existing BLAST alignments for PM1-S2_contigs... Using existing BLAST alignments for PM1-S3_contigs... Using existing BLAST alignments for PM1-C1D01_contigs... Using existing BLAST alignments for PM1-C1D02_contigs... Using existing BLAST alignments for PM1-C1D15_contigs... Using existing BLAST alignments for PM1-C1D36_contigs... Using existing BLAST alignments for PM3-S1_contigs... Using existing BLAST alignments for PM3-C0D15_contigs... Using existing BLAST alignments for PM3-C0D1_contigs... Using existing BLAST alignments for PM3-C0D2_contigs... Using existing BLAST alignments for PM3-C0D36_contigs... Using existing BLAST alignments for SD-01_contigs...

2019-11-20 08:35:36 Trying to download found references from NCBI. Totally 164 organisms to try.

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0".

alexeigurevich commented 4 years ago

Hi, Unfortunately, we didn't release a new version of Quast for a while, so the current latest release (version 5.0.2) is not yet affected by the fix mentioned here (I mean using time.sleep(1)). So, could you please try to apply the fix locally to your copy of 5.0.2 and check whether it works for you? To do so, please download the fixed copy of search_references_meta.py (use Right-Click and "Save as" menu), and put it in <your_quast_installation_dir>/quast_libs/ dir to replace the old version. After that, try to restart Quast.

microbiomeparty commented 4 years ago

thanks so much, this is working!

m

On Nov 20, 2019, at 10:03 AM, alexeigurevich notifications@github.com wrote:

Hi, Unfortunately, we didn't release a new version of Quast for a while, so the current latest release (version 5.0.2) is not yet affected by the fix mentioned here (I mean using time.sleep(1)). So, could you please try to apply the fix locally to your copy of 5.0.2 and check whether it works for you? To do so, please download the fixed copy of search_references_meta.py https://raw.githubusercontent.com/ablab/quast/32a76ecd7d0b6419840f0d9d4351109dc669f070/quast_libs/search_references_meta.py (use Right-Click and "Save as" menu), and put it in /quast_libs/ dir to replace the old version. After that, try to restart Quast.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ablab/quast/issues/8?email_source=notifications&email_token=AI7M44RI3VFW72BDYL6ZWFTQUVGVVA5CNFSM4BOXBBBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEESIQFA#issuecomment-556042260, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI7M44W65DZSJHE5GD5Z4YLQUVGVVANCNFSM4BOXBBBA.

Gian77 commented 1 year ago

I am having the same issue with a newer version of quast when I try to run it over a series of assemblies using a slurm array.

quast 5.2.0 py39pl5321h2add14b_2 bioconda

This is how I try to run it:

conda activate quast
echo "version of `quast --version`" 

metaquast.py \
    --threads $cores \
    --no-icarus \
    $project_dir/outputs/b02_assembly_spades/${sampleID}/scaffolds_hq.fasta \
    -o $project_dir/outputs/b05_evaluateAssembly_metaquast/${sampleID}/ 

conda deactivate

And this is the error I got


No references are provided, starting to search for reference genomes in SILVA 16S rRNA database and to download them from NCBI...

2023-04-05 16:48:16
Running BlastN..
  processing scaffolds_hq
  BLAST results for scaffolds_hq are saved to /mnt/research/EvansLab/Benucci/metaGAAP/outputs/b05_evaluateAssembly_metaquast/09_REX_JJ_S5_L003/quast_downloaded_references/blast.res_scaffolds_hq...

2023-04-05 16:49:16
Trying to download found references from NCBI. Totally 50 organisms to try.
WARNING: Empty/No response from NCBI. Could be an Internet connection issue!
  delta_proteobacterium_WY32                      | not found in the NCBI database
  delta_proteobacterium_WY32 was not found in NCBI database, trying to download the next best match
WARNING: Empty/No response from NCBI. Could be an Internet connection issue!
  delta_proteobacterium_LX33                      | not found in the NCBI database
  delta_proteobacterium_LX33 was not found in NCBI database, trying to download the next best match

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0" to disable reference search in the NCBI database.

In case you have troubles running QUAST, you can write to quast.support@cab.spbu.ru
or report an issue on our GitHub repository https://github.com/ablab/quast/issues
Please provide us with quast.log file from the output directory.

Thank you, Gian

MaryoHg commented 1 year ago

Dear @Gian77

How is it going? Were you able to fix the problem? I'm facing the same error while using metaquast.py in a server with full access to the internet.

The option time.sleep(1) is already added to quast_libs/search_references_meta.py script, so I'm not sure how to proceed. I'll run the code in a personal computer to double check.

Dear @alexeigurevich, any advice will be appreciated.

Saludos.

Gian77 commented 1 year ago

Hello @MaryoHg,

no, I still have that issue. @alexeigurevich any solve?

Thanks,

G.

Nitin123-4 commented 1 year ago

I am also facing the same issue with the latest version (5.2.0)

Trying to download found references from NCBI. Totally 35 organisms to try. human gut metagenome has too fragmented reference genome! It will not be downloaded. human_gut_metagenome | not found in the NCBI database human_gut_metagenome was not found in NCBI database, trying to download the next best match Trichuristrichiura(human_whipworm) | not found in the NCBI database Trichuristrichiura(human_whipworm) was not found in NCBI database, trying to download the next best match Prevotella_copri_DSM_18205 | successfully downloaded (total 1, 34 more to go) Veillonella_parvula | successfully downloaded (total 2, 33 more to go) Tyzzerella_sp._Marseille-P3062 | successfully downloaded (total 3, 32 more to go) Sutterellaceae_bacterium_Marseille-P2968 | successfully downloaded (total 4, 31 more to go) Streptococcus_thermophilus_ND03 | successfully downloaded (total 5, 30 more to go) Streptococcus_salivarius_subsp._thermophilus | successfully downloaded (total 6, 29 more to go) Streptococcus_pneumoniae | successfully downloaded (total 7, 28 more to go) Streptococcus_phage_2096.2 | not found in the NCBI database Streptococcus_phage_2096.2 was not found in NCBI database, trying to download the next best match Streptococcus_pyogenes_phage_6180.4 | not found in the NCBI database Streptococcus_pyogenes_phage_6180.4 was not found in NCBI database, trying to download the next best match Streptococcus_phage_10750.4 | not found in the NCBI database Streptococcus_infantarius | successfully downloaded (total 8, 26 more to go) Streptococcus_agalactiae_SS1219 | successfully downloaded (total 9, 25 more to go) Serratia_marcescens | successfully downloaded (total 10, 24 more to go) Neisseria_zalophi | successfully downloaded (total 11, 23 more to go) Myroides_odoratimimus_CCUG_12700 | successfully downloaded (total 12, 22 more to go) WARNING: Empty/No response from NCBI. Could be an Internet connection issue! Lactobacillus_sp._C33LV5 | not found in the NCBI database Lactobacillus_sp._C33LV5 was not found in NCBI database, trying to download the next best match WARNING: Empty/No response from NCBI. Could be an Internet connection issue! Lactobacillus_ruminis | not found in the NCBI database Lactobacillus_ruminis was not found in NCBI database, trying to download the next best match

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0" to disable reference search in the NCBI database.

MCeciC commented 1 week ago

Hello @MaryoHg,

no, I still have that issue. @alexeigurevich any solve?

Thanks,

G.

HI @Gian77 @MaryoHg could you solve the problem? 'm gettin the same error ERROR! Cannot established internet connection to download reference genomes! Check internet connection

Thanks!