search

gbouras13 / hybracter

Automated long-read first bacterial genome assembly tool implemented in Snakemake using Snaketool.
MIT License
103 stars 8 forks source link

[BUG] Possible to have hybracter install its own copy of mamba? #59

Closed rzelle-lallemand closed 7 months ago

rzelle-lallemand commented 7 months ago

Describe the bug We currently have a non-working version of mamba in our base environment (https://github.com/mamba-org/mamba/issues/1583), which so far I've been unable to fix. I was able to (quite easily) use conda to install hybracter, but the "hybracter install" command fails because it relies on mamba.

I was wondering whether hybracter could install its own copy of mamba since it seems to be a dependency, and not strictly required for installation of the hybracter conda package itself.

To Reproduce Steps to reproduce the behaviour, including the

  1. Start with a broken mamba 0.1.2 in your base environment (or no mamba installed presumably)
  2. Use conda to install hybracter (conda create -n hybracter -c bioconda -c conda-forge hybracter)
  3. Run hybracter install in the new environment

Output:

$ hybracter install

hybracter version 0.7.0

 _           _                    _            
| |__  _   _| |__  _ __ __ _  ___| |_ ___ _ __ 
| '_ \| | | | '_ \| '__/ _` |/ __| __/ _ \ '__|
| | | | |_| | |_) | | | (_| | (__| ||  __/ |   
|_| |_|\__, |_.__/|_|  \__,_|\___|\__\___|_|   
       |___/

[2024:03:04 23:39:22] Copying system default config to hybracter_install_intermediate_files/config.yaml
[2024:03:04 23:39:22] Updating config file with new values
[2024:03:04 23:39:22] Writing config file to hybracter_install_intermediate_files/config.yaml
[2024:03:04 23:39:22] ------------------
[2024:03:04 23:39:22] | Runtime config |
[2024:03:04 23:39:22] ------------------

args:
  databases: null
  log: hybracter_install_intermediate_files/hybracter.log
  medaka: false
  output: hybracter_install_intermediate_files
qc:
  compression: 5
  hostRemoveFlagstat: -f 4 -F 3584
  minimapIndex: -I 8G
  minimapModel: map-ont
resources:
  big:
    cpu: 16
    mem: 32000
    time: '23:59:00'
  med:
    cpu: 8
    mem: 16000
    time: 08:00:00
  sml:
    cpu: 1
    mem: 4000
    time: 00:00:05

[2024:03:04 23:39:22] ---------------------
[2024:03:04 23:39:22] | Snakemake command |
[2024:03:04 23:39:22] ---------------------

snakemake -s /software/anaconda3/envs/hybracter/lib/python3.12/site-packages/hybracter/workflow/install.smk --configfile hybracter_install_intermediate_files/config.yaml --jobs 1 --use-conda --rerun-incomplete --printshellcmds --nolock --show-failed-logs --conda-frontend mamba
Config file /software/anaconda3/envs/hybracter/lib/python3.12/site-packages/hybracter/workflow/../config/config.yaml is extended by additional config specified via the command line.
Assuming unrestricted shared filesystem usage for local execution.
Building DAG of jobs...
Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'.
Creating conda environment /software/anaconda3/envs/hybracter/lib/python3.12/site-packages/hybracter/workflow/envs/plassembler.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /software/anaconda3/envs/hybracter/lib/python3.12/site-packages/hybracter/workflow/envs/plassembler.yaml:
Command:
mamba env create --quiet --file "/sandbox/.snakemake/conda/4ffb3cad61de93db0faf748167c355f0_.yaml" --prefix "/sandbox/.snakemake/conda/4ffb3cad61de93db0faf748167c355f0_"
Output:
Traceback (most recent call last):
  File "/software/anaconda3/condabin/mamba", line 7, in <module>
    from mamba.mamba import main
  File "/software/anaconda3/lib/python3.7/site-packages/mamba/mamba.py", line 53, in <module>
    from mamba.utils import get_index, to_package_record_from_subjson, _supplement_index_with_system
  File "/software/anaconda3/lib/python3.7/site-packages/mamba/utils.py", line 7, in <module>
    from .FastSubdirData import FastSubdirData
  File "/software/anaconda3/lib/python3.7/site-packages/mamba/FastSubdirData.py", line 23, in <module>
    from conda._vendor.auxlib.ish import dals
ModuleNotFoundError: No module named 'conda._vendor.auxlib'

Expected behavior

I think my problem would disappear if hybracter would use mamba from its own environment.

(hybracter) $ which mamba
/software/anaconda3/condabin/mamba
(hybracter) $ which hybracter
/software/anaconda3/envs/hybracter/bin/hybracter

Screenshots NA

Desktop (please complete the following information):

Additional context NA

rzelle-lallemand commented 7 months ago

After sleeping on it, I just realized that I can solve my own problem by manually adding "mamba" to my hybracter conda environment, which did the trick as far as mamba was concerned, but now I ran into an error with the unicycler installation.

(hybracter) $ conda install mamba

(hybracter) $ hybracter install

hybracter version 0.7.0

 _           _                    _            
| |__  _   _| |__  _ __ __ _  ___| |_ ___ _ __ 
| '_ \| | | | '_ \| '__/ _` |/ __| __/ _ \ '__|
| | | | |_| | |_) | | | (_| | (__| ||  __/ |   
|_| |_|\__, |_.__/|_|  \__,_|\___|\__\___|_|   
       |___/

[2024:03:05 10:30:58] Copying system default config to hybracter_install_intermediate_files/config.yaml
[2024:03:05 10:30:58] Updating config file with new values
[2024:03:05 10:30:58] Writing config file to hybracter_install_intermediate_files/config.yaml
[2024:03:05 10:30:58] ------------------
[2024:03:05 10:30:58] | Runtime config |
[2024:03:05 10:30:58] ------------------

args:
  databases: null
  log: hybracter_install_intermediate_files/hybracter.log
  medaka: false
  output: hybracter_install_intermediate_files
qc:
  compression: 5
  hostRemoveFlagstat: -f 4 -F 3584
  minimapIndex: -I 8G
  minimapModel: map-ont
resources:
  big:
    cpu: 16
    mem: 32000
    time: '23:59:00'
  med:
    cpu: 8
    mem: 16000
    time: 08:00:00
  sml:
    cpu: 1
    mem: 4000
    time: 00:00:05

[2024:03:05 10:30:58] ---------------------
[2024:03:05 10:30:58] | Snakemake command |
[2024:03:05 10:30:58] ---------------------

snakemake -s /software/anaconda3/envs/hybracter/lib/python3.12/site-packages/hybracter/workflow/install.smk --configfile hybracter_install_intermediate_files/config.yaml --jobs 1 --use-conda --rerun-incomplete --printshellcmds --nolock --show-failed-logs --conda-frontend mamba
Config file /software/anaconda3/envs/hybracter/lib/python3.12/site-packages/hybracter/workflow/../config/config.yaml is extended by additional config specified via the command line.
Assuming unrestricted shared filesystem usage for local execution.
Building DAG of jobs...
Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'.
Creating conda environment /software/anaconda3/envs/hybracter/lib/python3.12/site-packages/hybracter/workflow/envs/plassembler.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /software/anaconda3/envs/hybracter/lib/python3.12/site-packages/hybracter/workflow/envs/plassembler.yaml:
Command:
mamba env create --quiet --file "/sandbox/.snakemake/conda/4ffb3cad61de93db0faf748167c355f0_.yaml" --prefix "/sandbox/.snakemake/conda/4ffb3cad61de93db0faf748167c355f0_"
Output:
Channels:
 - conda-forge
 - bioconda
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... 
done
Installing pip dependencies: ...working... Pip subprocess error:
  Running command git clone --filter=blob:none --quiet https://github.com/rrwick/Unicycler.git /tmp/pip-req-build-8dwqml_b
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  \u2502 exit code: 1
  \u2570\u2500> [57 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib
      creating build/lib/unicycler
      copying unicycler/assembly_graph.py -> build/lib/unicycler
      copying unicycler/assembly_graph_copy_depth.py -> build/lib/unicycler
      copying unicycler/bridge_loop_unroll.py -> build/lib/unicycler
      copying unicycler/spades_func.py -> build/lib/unicycler
      copying unicycler/misc.py -> build/lib/unicycler
      copying unicycler/bridge_miniasm.py -> build/lib/unicycler
      copying unicycler/unicycler_align.py -> build/lib/unicycler
      copying unicycler/read_ref.py -> build/lib/unicycler
      copying unicycler/miniasm_assembly.py -> build/lib/unicycler
      copying unicycler/alignment.py -> build/lib/unicycler
      copying unicycler/minimap_alignment.py -> build/lib/unicycler
      copying unicycler/path_finding.py -> build/lib/unicycler
      copying unicycler/settings.py -> build/lib/unicycler
      copying unicycler/bridge_long_read_simple.py -> build/lib/unicycler
      copying unicycler/string_graph.py -> build/lib/unicycler
      copying unicycler/unicycler.py -> build/lib/unicycler
      copying unicycler/bridge_long_read.py -> build/lib/unicycler
      copying unicycler/log.py -> build/lib/unicycler
      copying unicycler/blast_func.py -> build/lib/unicycler
      copying unicycler/__init__.py -> build/lib/unicycler
      copying unicycler/version.py -> build/lib/unicycler
      copying unicycler/bridge_common.py -> build/lib/unicycler
      copying unicycler/bridge_spades_contig.py -> build/lib/unicycler
      copying unicycler/cpp_wrappers.py -> build/lib/unicycler
      copying unicycler/assembly_graph_segment.py -> build/lib/unicycler
      /tmp/pip-req-build-8dwqml_b/setup.py:86: SetuptoolsDeprecationWarning: setup.py install is deprecated.
      !!

              ********************************************************************************
              Please avoid running ``setup.py`` directly.
              Instead, use pypa/build, pypa/installer or other
              standards-based tools.

              See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
              ********************************************************************************

      !!
        install.initialize_options(self)
      installing to build/bdist.linux-x86_64/wheel
      running install
      Cleaning previous compilation: make distclean
      Platform: Linux
      Falling back to -dumpversion as compiler did not support -dumpfullversion
      Compiler: g++ 4.8.5
      Makefile:60: *** Unicycler requires GCC version 4.9.1 or greater.  Stop.
      Compiling Unicycler: make -j 8
      Platform: Linux
      Falling back to -dumpversion as compiler did not support -dumpfullversion
      Compiler: g++ 4.8.5
      Makefile:60: *** Unicycler requires GCC version 4.9.1 or greater.  Stop.
      Error: compilation of Unicycler's C++ component failed
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for unicycler
  error: subprocess-exited-with-error

  × python setup.py clean did not run successfully.
  \u2502 exit code: 1
  \u2570\u2500> [6 lines of output]
      usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
         or: setup.py --help [cmd1 cmd2 ...]
         or: setup.py --help-commands
         or: setup.py cmd --help

      error: option --all not recognized
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed cleaning build dir for unicycler
ERROR: Could not build wheels for unicycler, which is required to install pyproject.toml-based projects

Ran pip subprocess with arguments:
['/sandbox/.snakemake/conda/4ffb3cad61de93db0faf748167c355f0_/bin/python', '-m', 'pip', 'install', '-U', '-r', '/sandbox/.snakemake/conda/condaenv.be_4_5wp.requirements.txt', '--exists-action=b']
Pip subprocess output:
Collecting git+https://github.com/rrwick/Unicycler.git (from -r /sandbox/.snakemake/conda/condaenv.be_4_5wp.requirements.txt (line 1))
  Cloning https://github.com/rrwick/Unicycler.git to /tmp/pip-req-build-8dwqml_b
  Resolved https://github.com/rrwick/Unicycler.git to commit 69e712eb95c4b9f8a46aade467260260a9ce7a91
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: unicycler
  Building wheel for unicycler (setup.py): started
  Building wheel for unicycler (setup.py): finished with status 'error'
  Running setup.py clean for unicycler
Failed to build unicycler

failed

CondaEnvException: Pip failed

[2024:03:05 10:32:49] ERROR: Snakemake failed
rzelle-lallemand commented 7 months ago

I now managed to run "hybracter install" successfully after installing a newer system copy of gcc by following the instructions at https://superuser.com/a/1758987. Installing a newer version of gcc in our "hybracter" conda environment (via "conda install -c conda-forge gxx" did not help, as it looks like the system version is used.

gbouras13 commented 7 months ago

Hi @rzelle-lallemand ,

I am glad you managed to fix the issue.

If you have mamba issues in future (in this scenario I personally would've considered a brand new mini forge install as mamba is so key!), you can tell hybracter to use conda instead by using --conda-frontend conda at the end of any hybracter command. You in fact can pass any snakemake arguments you want that way.

George

rzelle-lallemand commented 7 months ago

Ah, good to know. Thanks!