search

nf-core / mag

Assembly and binning of metagenomes
https://nf-co.re/mag
MIT License
208 stars 104 forks source link

CAT_DB_GENERATE issues #462

Closed prototaxites closed 4 months ago

prototaxites commented 1 year ago

Description of the bug

CAT_DB_GENERATE fails on my system due to what seem to be network connection issues, despite nodes having internet access:

WARNING: Skipping mount /var/apptainer/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
# CAT v4.6.

CAT prepare is running, constructing a fresh database.
Rawr!

WARNING: preparing the database files may take a couple of hours.

Supplied command: /usr/local/bin/CAT prepare --fresh

Taxonomy folder: 2023-06-23_taxonomy/
Database folder: 2023-06-23_CAT_database/
Log file: 2023-06-23.CAT_prepare.fresh.log

-----------------

[2023-06-23 10:05:36.644592] DIAMOND found: diamond version 2.0.6.
[2023-06-23 10:05:36.705498] 2023-06-23_taxonomy is created.
[2023-06-23 10:05:36.753575] 2023-06-23_CAT_database is created.
[2023-06-23 10:05:36.760679] Downloading and extracting taxonomy files from ftp://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz to 2023-06-23_taxonomy.
[2023-06-23 10:05:36.800651] ERROR: donwload of taxonomy files failed.

Digging into the error a bit, this appears to be a recurring problem with some older biocontainers: https://github.com/bioconda/bioconda-recipes/issues/11583 - the CAT mulled container was built on 15/03/2021, before this issue was fixed. CAT image might want to be updated to use a newer base image, which hopefully avoids this problem.

In debugging this, I also noticed that CAT_DB_GENERATE only gets 1 cpu by default, but there's a --nproc option which specifies the number of cores available when building the Diamond database. Might be good to bump that up to speed up datebase building.

Command used and terminal output

nextflow run nf-core/mag -profile test,singularity --cat_db_generate --outdir results

Relevant files

No response

System information

HPC, Slurm, Singularity, Linux, dev

jfy133 commented 10 months ago

@prototaxites would you be able to investigate this as a bug fix? I guess it could be quite small fix; or slightly more (but more robust) if it was converted to a proper nf-core module.

prototaxites commented 10 months ago

Can take a look but might take me some time to get to it, as my plate is quite full at the moment! I did build an updated version of the container back when I first made this issue, but I am not convinced I remember it fixing the issue.

jfy133 commented 7 months ago

Ultimately I think the best thing is to just replace the custom module with official nf-core ones... so hopefully it gets solved then (as will be using latest verison etc)

jfy133 commented 4 months ago

There are a bunch of issues which I think means this would be fixed by teh whole-sale update, closing in favour of the most recent issue as it's all related https://github.com/nf-core/mag/issues/611