Database installation problems

[agudeloromero]

Hi there,

Good job for this awesome pipeline! I'd really like to try it, but I'm stuck on installing the databases.

I'm working on a cluster and tried:

hecatomb install --profile slurm hecatomb install

Please see the log files enclosed.

Thanks in advance, Patricia

2022-02-03T082640.845628.snakemake.log 2022-02-03T082606.247106.snakemake.log

[beardymcjohnface]

Hi Patricia, Thanks for your interest in the pipeline! There's not much to go on here, but it looks like you have successfully downloaded some of the files in a previous run? It looks like Snakemake had issues submitting the slurm jobs with the profile.

Do you have any slurm logs? (if you're using the example profile then they will be in logs/, otherwise they'll just be slurm-12345...log etc if using the cookiecutter profile). You might need to adjust the profile for your cluster. You should be able to access most of the information you need with sinfo -Nl and of course you clusters user guide.

[agudeloromero]

Hi Michael,

Thanks for your quick reply.

I fixed this issue changing the database path out of conda in my config file.

Cheers,

[beardymcjohnface]

Excellent news!

[lucyintheskyzzz]

@beardymcjohnface I have an issue downloading the databases for HPC:

Looks like my install worked but I can't download the databases for HPC:

(/lustre/project/taw/share/conda-envs/hecatomb) [kvigil@cypress2 conda-envs]$ hecatomb --help

██╗ ██╗███████╗ ██████╗ █████╗ ████████╗ ██████╗ ███╗ ███╗██████╗ ██║ ██║██╔════╝██╔════╝██╔══██╗╚══██╔══╝██╔═══██╗████╗ ████║██╔══██╗ ███████║█████╗ ██║ ███████║ ██║ ██║ ██║██╔████╔██║██████╔╝ ██╔══██║██╔══╝ ██║ ██╔══██║ ██║ ██║ ██║██║╚██╔╝██║██╔══██╗ ██║ ██║███████╗╚██████╗██║ ██║ ██║ ╚██████╔╝██║ ╚═╝ ██║██████╔╝ ╚═╝ ╚═╝╚══════╝ ╚═════╝╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚═════╝

Hecatomb version v1.1.0

Usage: hecatomb [OPTIONS] COMMAND [ARGS]...

Subcommand usage: hecatomb subcommand --help

Options: --help Show this message and exit.

Commands: run Run Hecatomb test Run the Hecatomb test dataset config Copy the system default config file install Install the Hecatomb databases combine Combine multiple Hecatomb runs add-host Add a new host genome to use with Hecatomb list-hosts List the available host genomes (/lustre/project/taw/share/conda-envs/hecatomb) [kvigil@cypress2 conda-envs]$

(/lustre/project/taw/share/conda-envs/hecatomb) [kvigil@cypress2 conda-envs]$ hecatomb install --profile slurm

██╗ ██╗███████╗ ██████╗ █████╗ ████████╗ ██████╗ ███╗ ███╗██████╗ ██║ ██║██╔════╝██╔════╝██╔══██╗╚══██╔══╝██╔═══██╗████╗ ████║██╔══██╗ ███████║█████╗ ██║ ███████║ ██║ ██║ ██║██╔████╔██║██████╔╝ ██╔══██║██╔══╝ ██║ ██╔══██║ ██║ ██║ ██║██║╚██╔╝██║██╔══██╗ ██║ ██║███████╗╚██████╗██║ ██║ ██║ ╚██████╔╝██║ ╚═╝ ██║██████╔╝ ╚═╝ ╚═╝╚══════╝ ╚═════╝╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚═════╝

Hecatomb version v1.1.0

[2023:01:25 08:33:58] Config file hecatomb.config.yaml already exists. Using existing config file. [2023:01:25 08:33:58] Writing runtime config file to hecatomb.config.yaml [2023:01:25 08:33:58] ------------------ [2023:01:25 08:33:58] | Runtime config | [2023:01:25 08:33:58] ------------------

BigJobCpu: 24 BigJobMem: 64000 BigJobTimeMin: 1440 COMPRESSION: 1 CONTIG_MINLENGTH: 1000 CUTTAIL_WINDOW: 25 DEDUP_ACCURACY: 4 Databases: null ENTROPY: 0.5 ENTROPYWINDOW: 25 Host: human MediumJobCpu: 16 MediumJobMem: 32000 MoreRamCpu: 2 MoreRamMem: 16000 Output: hecatomb.out Preprocess: paired QSCORE: 15 READ_MINLENGTH: 90 Reads: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data Search: sensitive SmallJobCpu: 8 SmallJobMem: 16000 canuSettings: correctedErrorRate=0.16 maxInputCoverage=10000 minInputCoverage=0 corOutCoverage=10000 corMhapSensitivity=high corMinCoverage=0 useGrid=False stopOnLowCoverage=False genomeSize=10M -nanopore filtAAprimary: --min-length 30 -e 1e-3 filtAAsecondary: --min-length 30 -e 1e-5 filtNTprimary: --min-length 90 -e 1e-10 filtNTsecondary: --min-length 90 -e 1e-20 linclustParams: --kmer-per-seq-scale 0.3 -c 0.8 --cov-mode 1 --min-seq-id 0.97 --alignment-mode 3 perfAA: --start-sens 1 --sens-steps 3 -s 7 --lca-mode 2 --shuffle 0 perfAAfast: -s 4.0 --lca-mode 2 --shuffle 0 perfNT: --start-sens 2 -s 7 --sens-steps 3 perfNTfast: -s 4.0 taxIdIgnore: 0 1 2 10239 131567 12429 2759

[2023:01:25 08:33:58] --------------------- [2023:01:25 08:33:58] | Snakemake command | [2023:01:25 08:33:58] ---------------------

snakemake -s /lustre/project/taw/share/conda-envs/hecatomb/bin/../snakemake/workflow/DownloadDB.smk --configfile hecatomb.config.yaml --rerun-incomplete --printshellcmds --nolock --show-failed-logs --profile slurm Error: profile given but no config.yaml found. Profile has to be given as either absolute path, relative path or name of a directory available in either /etc/xdg/snakemake or /home/kvigil/.config/snakemake. [2023:01:25 08:34:08] Error: Snakemake failed

2023-01-24T152543.473897.snakemake.log

2023-01-25T071632.241265.snakemake.log