ModuleNotFoundError: No module named 'vbgmm'

[francesco-ricci]

Hey Vini,

I keep having this issue related to concoct:

Traceback (most recent call last): File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/db931a7715ae01092c19e030ea366c0b_/$ import vbgmm ModuleNotFoundError: No module named 'vbgmm'

I try to troubleshoot it in various ways installing mkl, and reinstalling concoct but nothing.

Can you please help me?

Thanks Francesco

[francesco-ricci]

Tried reinstalling concoct, metaphor, and deleting the troubled directory. Nothing works.

This is metaphor log error file:

Config file /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/default-config.yaml is extended by additional config specified via the command line. Assuming unrestricted shared filesystem usage. Building DAG of jobs... Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channelpriority strict'. Creating conda environment /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/concoct.yaml... Downloading and installing remote packages. Environment for /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/concoct.yaml created (location: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/db931a7715ae01092c19e030ea366c0b) Using shell: /bin/bash Provided cores: 128 Rules claiming more threads will be scaled down. Job stats: job count

---

DAS_tool 1 all 1 assembly_report 1 bins_report 1 cog_functional_parser 1 concatenate_benchmarks 1 concatenate_taxonomies 16 concoct 1 diamond 1 diamond_makedb 1 download_COG_database 1 download_taxonomy_database 1 generate_COG_taxonmap 1 jgi_summarize_bam_contig_depths 1 lineage_parser 2 metabat2 1 plot_benchmarks 1 plot_cog_functional 1 taxonomy_parser 1 vamb 1 total 36

Select jobs to execute... Execute 1 jobs...

[Sun May 12 20:38:39 2024] localrule concoct: input: output/mapping/cobinning/cobinning_contigs_catalogue.fna, output/mapping/bam/cobinning/Blank-3L7-to-contigs.sorted.bam, output/mapping/bam/cobinning/FR2A1L7-to-contigs.sorted.bam, output/mapping/bam/cobinning/FR2A2L7-to-contigs.sorted.bam, output/mapping/bam/cobinning/FR2A3L7-to-contigs.sorted.bam, output/mapping/bam/cobinning/FR2B1L7-to-contigs.sorted.bam, output/mapping/bam/cobinning/FR2B2L7-to-contigs.sorted.bam, output/mapping/bam/cobinning/FR2B3$ output: output/binning/concoct/cobinning, output/binning/concoct/cobinning/concoct_scaffolds2bin.tsv log: output/logs/binning/concoct/cobinning.log jobid: 97 benchmark: output/benchmarks/binning/concoct/cobinning.txt reason: Missing output files: output/binning/concoct/cobinning/concoct_scaffolds2bin.tsv, output/binning/concoct/cobinning wildcards: binning_group=cobinning threads: 128 resources: tmpdir=/tmp, mem_mb=99968, mem_mib=95337

    rm -rf output/binning/concoct/cobinning
    mkdir output/binning/concoct/cobinning

    { cut_up_fasta.py output/mapping/cobinning/cobinning_contigs_catalogue.fna                                                   -c 10000                                             -o 0                                                                -b output/binning/concoct/cobinning/contigs.bed                                                     --merge_last                                                        > output/binning/concoct/cobinning/contigs.fa  ; } 2>> $

    { concoct_coverage_table.py output/binning/concoct/cobinning/contigs.bed                                                        output/mapping/bam/cobinning/Blank-3_L7_-to-contigs.sorted.bam output/mapping/bam/cobinning/FR2A1_L7_-to-contigs.sorted.bam output/mapping/bam/cobinning/FR2A2_L7_-to-contigs.sorted.bam output/mapping/bam/cobinning/FR2A3_L7_-to-contigs.sorted.bam output/mapping/bam/cobinning/FR2B1_L7_-to-contigs.sorted.bam output/mapping/bam/cobinning/FR2B2$

    { concoct --composition_file output/binning/concoct/cobinning/contigs.fa                                 --coverage_file output/binning/concoct/cobinning/coverage_table.tsv                             -b output/binning/concoct/cobinning                                                  -t 128  ; } 2>> output/logs/binning/concoct/cobinning.log

    { merge_cutup_clustering.py output/binning/concoct/cobinning/clustering_gt1000.csv                                              > output/binning/concoct/cobinning/clustering_merged.csv  ; } 2>> output/logs/binning/concoct/cobinning.log

    mkdir output/binning/concoct/cobinning/fasta_bins

    { extract_fasta_bins.py output/mapping/cobinning/cobinning_contigs_catalogue.fna                                                   output/binning/concoct/cobinning/clustering_merged.csv                                          --output_path output/binning/concoct/cobinning/fasta_bins ; } 2>> output/logs/binning/concoct/cobinning.log

    sed "s/,/$(echo '   ')concoct./g" output/binning/concoct/cobinning/clustering_merged.csv | tail -n +2 > output/binning/concoct/cobinning/concoct_scaffolds2bin.tsv

Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/db931a7715ae01092c19e030ea366c0b_ [Sun May 12 21:16:18 2024] Error in rule concoct: jobid: 97 input: output/mapping/cobinning/cobinning_contigs_catalogue.fna, output/mapping/bam/cobinning/Blank-3L7-to-contigs.sorted.bam, output/mapping/bam/cobinning/FR2A1L7-to-contigs.sorted.bam, output/mapping/bam/cobinning/FR2A2L7-to-contigs.sorted.bam, output/mapping/bam/cobinning/FR2A3L7-to-contigs.sorted.bam, output/mapping/bam/cobinning/FR2B1L7-to-contigs.sorted.bam, output/mapping/bam/cobinning/FR2B2L7-to-contigs.sorted.bam, output/mapping/bam/cobinning/FR2B3$ output: output/binning/concoct/cobinning, output/binning/concoct/cobinning/concoctscaffolds2bin.tsv log: output/logs/binning/concoct/cobinning.log (check log file(s) for error details) conda-env: /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/db931a7715ae01092c19e030ea366c0b shell:

rm -rf output/binning/concoct/cobinning
    mkdir output/binning/concoct/cobinning

    { cut_up_fasta.py output/mapping/cobinning/cobinning_contigs_catalogue.fna                                                   -c 10000                                             -o 0                                                                -b output/binning/concoct/cobinning/contigs.bed                                                     --merge_last                                                        > output/binning/concoct/cobinning/contigs.fa  ; } 2>> $

    { concoct_coverage_table.py output/binning/concoct/cobinning/contigs.bed                                                        output/mapping/bam/cobinning/Blank-3_L7_-to-contigs.sorted.bam output/mapping/bam/cobinning/FR2A1_L7_-to-contigs.sorted.bam output/mapping/bam/cobinning/FR2A2_L7_-to-contigs.sorted.bam output/mapping/bam/cobinning/FR2A3_L7_-to-contigs.sorted.bam output/mapping/bam/cobinning/FR2B1_L7_-to-contigs.sorted.bam output/mapping/bam/cobinning/FR2B2$

    { concoct --composition_file output/binning/concoct/cobinning/contigs.fa                                 --coverage_file output/binning/concoct/cobinning/coverage_table.tsv                             -b output/binning/concoct/cobinning                                                  -t 128  ; } 2>> output/logs/binning/concoct/cobinning.log

    { merge_cutup_clustering.py output/binning/concoct/cobinning/clustering_gt1000.csv                                              > output/binning/concoct/cobinning/clustering_merged.csv  ; } 2>> output/logs/binning/concoct/cobinning.log

    mkdir output/binning/concoct/cobinning/fasta_bins

    { extract_fasta_bins.py output/mapping/cobinning/cobinning_contigs_catalogue.fna                                                   output/binning/concoct/cobinning/clustering_merged.csv                                          --output_path output/binning/concoct/cobinning/fasta_bins ; } 2>> output/logs/binning/concoct/cobinning.log

    sed "s/,/$(echo '   ')concoct./g" output/binning/concoct/cobinning/clustering_merged.csv | tail -n +2 > output/binning/concoct/cobinning/concoct_scaffolds2bin.tsv

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job concoct since they might be corrupted: output/binning/concoct/cobinning Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-05-12T203350.723229.snakemake.log WorkflowError: At least one job did not complete successfully. Traceback (most recent call last): File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/metaphor", line 10, in sys.exit(main()) ^^^^^^ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/cli.py", line 278, in main args.func(args) File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/execute.py", line 109, in main retcode = run_cmd(cmd) ^^^^^^^^^^^^ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/utils.py", line 92, in run_cmd retcode = check_call(cmd.split()) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['snakemake', '--snakefile', '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/Snakefile', '--configfile', 'metaphor_settings.yaml', '--cores', '128', '--printshellcmds', '--use-conda', '--wrapper-prefix', 'https://github.com/snakemake/snakemake-wrappers/raw/', '--conda-prefix', '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/c$

[vinisalazar]

hey @francesco-ricci,

Thank you for reporting this. This appears to be a CONCOCT problem. Have a look at this issue: https://github.com/BinPro/CONCOCT/issues/181

Could you try activating the concoct environment (should be conda activate /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/db931a7715ae01092c19e030ea366c0b_/ and then install MKL manually with conda install conda-forge::mkl?

It is weird to me because mkl is indeed defined as a explicit reference in the concoct env.

This issue: https://github.com/BinPro/CONCOCT/issues/294 also mentions the problem and this comment suggests using the pip executable in the concoct environment to install mkl, in your case should be something like:

/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/db931a7715ae01092c19e030ea366c0b_/bin/pip install mkl

Let me know how you go.

Vini

[vinisalazar]

By the way, what OS are you running Metaphor?

I'm asking because if I try to create an environment using the concoct environment file from Metaphor in Linux, the import vbgmm line works without issue. On my Mac, if try to install CONCOCT manually, I get an error similar to yours.

Could you please check the CONCOCT version on the environment causing the error? (activate the environment, then conda list | grep concoct)

Thanks

[francesco-ricci]

Hey Vini, thanks and sorry for the late reply I was on leave.

Anyway, I tried activating the environment you recommended and installing mkl with conda install conda-forge::mkl. I'll let you know how it goes.

I am running metaphor on Linux, concoct 1.1.0 py311h245ed52_4 bioconda.

Thanks Fra

[francesco-ricci]

When I try to install mkl with pip in the suggested folder I get this:

Requirement already satisfied: mkl in /fs04/rp24/fra/software/miniconda3/envs/checkm/lib/python3.10/site-packages (2024.1.0) Requirement already satisfied: intel-openmp==2024. in /fs04/rp24/fra/software/miniconda3/envs/checkm/lib/python3.10/site-packages (from mkl) (2024.1.0) Requirement already satisfied: tbb==2021. in /fs04/rp24/fra/software/miniconda3/envs/checkm/lib/python3.10/site-packages (from mkl) (2021.12.0)

However, after running metaphor I get again the vbgmm error.

[vinisalazar]

Requirement already satisfied: mkl in /fs04/rp24/fra/software/miniconda3/envs/checkm/lib/python3.10/site-packages (2024.1.0) Requirement already satisfied: intel-openmp==2024. in /fs04/rp24/fra/software/miniconda3/envs/checkm/lib/python3.10/site-packages (from mkl) (2024.1.0) Requirement already satisfied: tbb==2021. in /fs04/rp24/fra/software/miniconda3/envs/checkm/lib/python3.10/site-packages (from mkl) (2021.12.0)

I'm wondering why does this error message points to miniconda3/envs/checkm/ rather than the miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/db931a7715ae01092c19e030ea366c0b_/ environment?

[francesco-ricci]

Hey Vini,

thanks for your help with this. I fixed this issue by deactivating conda base environment and activating the db931a7715ae01092c19e030ea366c0b_/ environment and installed the mkl there.

Hope this gonna help somebody Francesco

[vinisalazar]

Thank you for reporting the fix @francesco-ricci. I will close this issue for now.