A method to solve kraken2-build download error

[aStudyingTurtle]

./kraken2/rsync_from_ncbi.pl

1. line 46 : 48 add # in front of these lines

   #  if (! ($full_path =~ s#^ftp://${qm_server}${qm_server_path}/##)) {
   # die "$PROG: unexpected FTP path (new server?) for $ftp_path\n";
   #}

2. add $ftp_path = substr($ftp_path,37); between line 41 and line 42

Notice : It is needed to add --use-ftp in your kraken2-build command

[ryan-preble]

I have tried this as well as the solution in #292 and neither worked. Whenever my database (sometimes) completes building, I always get the error message kraken2: database does not contain necessary file taxo.k2d

[ryan-preble]

I have tried this as well as the solution in #292 and neither worked. Whenever my database (sometimes) completes building, I always get the error message kraken2: database does not contain necessary file taxo.k2d

it because you haven't enough ram. i have the same issue before, finally i found that when the database is larger than your ram, the database will not be built successfully. so , i finally use --max-db-size 51 to solve the issue(i just have 64G ram), my code is: kraken2-build --build --threads 14 --db ./ --max-db-size 51

My machine has 500 GB RAM. Is that not enough to build a kraken database? If so, that deserves a separate issue of its own. A machine with 500 GB RAM and 64 CPUs should have well enough processing power to create the database.

[nashanghenzan]

I have tried this as well as the solution in #292 and neither worked. Whenever my database (sometimes) completes building, I always get the error message kraken2: database does not contain necessary file taxo.k2d

it because you haven't enough ram. i have the same issue before, finally i found that when the database is larger than your ram, the database will not be built successfully. so , i finally use --max-db-size 51 to solve the issue(i just have 64G ram), my code is: kraken2-build --build --threads 14 --db ./ --max-db-size 51

My machine has 500 GB RAM. Is that not enough to build a kraken database? If so, that deserves a separate issue of its own. A machine with 500 GB RAM and 64 CPUs should have well enough processing power to create the database.

the issue has solved?

[ryan-preble]

I have tried this as well as the solution in #292 and neither worked. Whenever my database (sometimes) completes building, I always get the error message kraken2: database does not contain necessary file taxo.k2d

it because you haven't enough ram. i have the same issue before, finally i found that when the database is larger than your ram, the database will not be built successfully. so , i finally use --max-db-size 51 to solve the issue(i just have 64G ram), my code is: kraken2-build --build --threads 14 --db ./ --max-db-size 51

My machine has 500 GB RAM. Is that not enough to build a kraken database? If so, that deserves a separate issue of its own. A machine with 500 GB RAM and 64 CPUs should have well enough processing power to create the database.

the issue has solved?

No, the database still does not build. RAM is NOT the issue. This is a development issue.

[nashanghenzan]

yes, now i also fail to build custom database, but i build a database successfuly that don't contain bacteria refseq. So i finally use the database from https://benlangmead.github.io/aws-indexes/k2

[sgivan]

./kraken2/rsync_from_ncbi.pl

1. line 46 : 48 add # in front of these lines

  #  if (! ($full_path =~ s#^ftp://${qm_server}${qm_server_path}/##)) {
  # die "$PROG: unexpected FTP path (new server?) for $ftp_path\n";
  #}

2. add `  $ftp_path = substr($ftp_path,37);` between line 41 and line 42

Notice : It is needed to add --use-ftp in your kraken2-build command

I also fixed the "unexpected FTP path (new server?)" problem by changing line 46 of the rsync_from_ncbi.pl script to:

if (! ($full_path =~ s#^https://${qm_server}${qm_server_path}/##)) {

[phelelani]

./kraken2/rsync_from_ncbi.pl

1. line 46 : 48 add # in front of these lines

  #  if (! ($full_path =~ s#^ftp://${qm_server}${qm_server_path}/##)) {
  # die "$PROG: unexpected FTP path (new server?) for $ftp_path\n";
  #}

2. add $ftp_path = substr($ftp_path,37); between line 41 and line 42

Notice : It is needed to add --use-ftp in your kraken2-build command

@aStudyingTurtle - This worked like a charm. Thank you! Also, a suggestion for slow/failing downloads using wget on FTP server - edit the download_genomic_library.sh as follows:

1. Comment out line 32: 
    # wget -q ${FTP_SERVER}${file}

2. Add this below line 32:
    lftp -e "pget -n10 ${FTP_SERVER}${file}; exit"

This method works well with --use-ftp, given you have lftp installed. You can reduce/increase number of connections to a file using the -n option.