search

mamba-org / mamba

The Fast Cross-Platform Package Manager
https://mamba.readthedocs.io
BSD 3-Clause "New" or "Revised" License
6.95k stars 359 forks source link

micromamba 2.0.0: environment creation process killed when using several channels #3482

Open MartinJepsen opened 1 month ago

MartinJepsen commented 1 month ago

Troubleshooting docs

Anaconda default channels

How did you install Mamba?

Micromamba

Search tried in issue tracker

"cuda", "cuda killed"

Latest version of Mamba

Tried in Conda?

I didn't try

Describe your issue

When running

micromamba env create -f environment.yml -c conda-forge -c renopexconda --override-channels

I get the logs pasted below (only last few lines posted), where it says "Killed" after trying to load CUDA virtual package for a minute or so.

mamba info / micromamba info

libmamba version : 2.0.0
     micromamba version : 2.0.0
           curl version : libcurl/8.9.1 OpenSSL/3.3.2 zlib/1.3.1 zstd/1.5.6 libssh2/1.11.0 nghttp2/1.58.0
     libarchive version : libarchive 3.7.4 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.6
       envs directories : /home/majp/micromamba/envs
          package cache : /home/majp/micromamba/pkgs
                          /home/majp/.mamba/pkgs
            environment : base
           env location : /home/majp/micromamba
      user config files : /home/majp/.mambarc
 populated config files : /home/majp/.mambarc
                          /home/majp/.condarc
       virtual packages : __unix=0=0
                          __linux=5.15.153=0
                          __glibc=2.31=0
                          __archspec=1=x86_64_v3
                          __cuda=12.2=0
               channels : https://artifactory.ramboll.com/artifactory/api/conda/renopexconda-pytools/linux-64
                          https://artifactory.ramboll.com/artifactory/api/conda/renopexconda-pytools/noarch
                          https://artifactory.ramboll.com/artifactory/api/conda/renopexconda-feedstock/linux-64
                          https://artifactory.ramboll.com/artifactory/api/conda/renopexconda-feedstock/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
       base environment : /home/majp/micromamba
               platform : linux-64

Logs

debug    libmamba Locking file '/home/majp/micromamba/pkgs/cache/88ec62ec.solv'
debug    libmamba Successfully locked
info     libmamba Loaded solv metadata : {"etag":"\"3eb60e7ebf1790253c8d290c1aad18b4\"","mod":"Mon, 30 Sep 2024 05:52:00 GMT","tool_version":"2.0_0.7.30","url":"https://conda.anaconda.org/conda-forge/linux-64"}
info     libmamba Metadata from solv are valid, loading successful
debug    libmamba Unlocking '/home/majp/micromamba/pkgs/cache/88ec62ec.solv'
info     libmamba Attempting to read libsolv solv file "/home/majp/micromamba/pkgs/cache/eb045dd1.solv" for repo https://conda.anaconda.org/conda-forge/noarch
info     libmamba Expecting solv metadata : {"etag":"\"3ae893e0abc01808a7d308503d0b4090\"","mod":"Mon, 30 Sep 2024 05:52:21 GMT","tool_version":"2.0_0.7.30","url":"https://conda.anaconda.org/conda-forge/noarch"}
debug    libmamba Locking file '/home/majp/micromamba/pkgs/cache/eb045dd1.solv'
debug    libmamba Successfully locked
info     libmamba Loaded solv metadata : {"etag":"\"3ae893e0abc01808a7d308503d0b4090\"","mod":"Mon, 30 Sep 2024 05:52:21 GMT","tool_version":"2.0_0.7.30","url":"https://conda.anaconda.org/conda-forge/noarch"}
info     libmamba Metadata from solv are valid, loading successful
debug    libmamba Unlocking '/home/majp/micromamba/pkgs/cache/eb045dd1.solv'
debug    libmamba Loading virtual packages
debug    libmamba Loading distribution virtual packages
debug    libmamba Loading CUDA virtual package
debug    libmamba CUDA driver version found: 12.2

Killed

environment.yml

name: pytoolsexecution

channels:
- renopexconda

dependencies:
- python>=3.11
- renopexconda::pyropacs>=1.0.1
- renopexconda::logutils>=1.4
- renopexconda::taskrunner>=1.0.1
# Development
- black>=24 # Assists with automatic formatting to conform to PEP8.
- conda-forge::autoflake # Removes unused imports
- conda-forge::docformatter # Formats docstrings
- conda-forge::pyupgrade # Automatically upgrade syntax for newer versions
- conda-build # Conda tool for building packages
- conda-verify # Performs some packaging checks.
- flake8>=6 # Linting
- conda-forge::flake8-bugbear # Detection of likely bugs and anti-patterns
- conda-forge::flake8-docstrings # Linting of docstrings
- isort # Formatting of import statements for PEP8 compliance
- pep8-naming # PEP8 naming convention checker
- pre-commit # Tool to prevent committing unsuccessfully
- pylint # Advanced linting
- pytest>=7.0.0 # Testing module
- pytest-cov>=4.0.0 # Produces coverage reports for tests.
- pytest-mock # Mocking library
- renopexconda::pyutil>=1.3.0 # A collection of utilities and helper scripts for general developements.
- sphinx>=7 # Documentation build
- sphinx_rtd_theme>=1.3 # Documentation theme
- conda-forge::sphinx-autodoc-typehints # Type hint support
- conda-forge::sphinxcontrib-apidoc # Automatic documentation from docstrings
- conda-forge::myst-parser # Markdown support for Sphinx

~/.condarc

channels:
  - renopexconda
  - defaults
  - conda-forge

custom_multichannels:
  renopexconda:
    - https://****:****@artifactory.ramboll.com/artifactory/api/conda/renopexconda-pytools
    - https://****:****@artifactory.ramboll.com/artifactory/api/conda/renopexconda-feedstock
  renopexconda-dev:
    - https://****:****@artifactory.ramboll.com/artifactory/api/conda/renopexconda-pytooldev
update_dependencies: true
show_channel_urls: true
use_pip: true
variables:
  JUPYTER_PLATFORM_DIRS: 1
ssl_verify: false
jjerphan commented 1 month ago

Thank you for opening this issue, I cannot reproduce the problem locally with a modified version of your environment which does not use renopexconda.

How have you installed micromamba on your system?

MartinJepsen commented 1 month ago

How have you installed micromamba on your system?

I got it via the shell install script from the user guide:

"${SHELL}" <(curl -L micro.mamba.pm/install.sh)

I have now installed miniforge3, and it works fine with conda/mamba

jjerphan commented 1 month ago

I currently cannot reproduce the error with:

this modified environment ```yaml name: pytoolsexecution channels: - conda-forge dependencies: - python>=3.11 # Development - black>=24 # Assists with automatic formatting to conform to PEP8. - autoflake # Removes unused imports - docformatter # Formats docstrings - pyupgrade # Automatically upgrade syntax for newer versions - conda-build # Conda tool for building packages - conda-verify # Performs some packaging checks. - flake8>=6 # Linting - flake8-bugbear # Detection of likely bugs and anti-patterns - flake8-docstrings # Linting of docstrings - isort # Formatting of import statements for PEP8 compliance - pep8-naming # PEP8 naming convention checker - pre-commit # Tool to prevent committing unsuccessfully - pylint # Advanced linting - pytest>=7.0.0 # Testing module - pytest-cov>=4.0.0 # Produces coverage reports for tests. - pytest-mock # Mocking library - sphinx>=7 # Documentation build - sphinx_rtd_theme>=1.3 # Documentation theme - sphinx-autodoc-typehints # Type hint support - sphinxcontrib-apidoc # Automatic documentation from docstrings - myst-parser # Markdown support for Sphinx ```
this modified configuration ``` channels: - defaults - conda-forge update_dependencies: true show_channel_urls: true use_pip: true variables: JUPYTER_PLATFORM_DIRS: 1 ssl_verify: false ```

using:

       libmamba version : 2.0.1
     micromamba version : 2.0.1
           curl version : libcurl/8.9.1 OpenSSL/3.3.2 zlib/1.3.1 zstd/1.5.6 libssh2/1.11.0 nghttp2/1.58.0
     libarchive version : libarchive 3.7.4 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.6
       envs directories : /home/jjerphan/.local/miniforge3/envs
          package cache : /home/jjerphan/.cache/conda/pkgs
            environment : base (active)
           env location : /tmp
      user config files : /home/jjerphan/.mambarc
 populated config files : /home/jjerphan/.condarc
       virtual packages : __unix=0=0
                          __linux=6.10.10=0
                          __glibc=2.39=0
                          __archspec=1=x86_64_v3
                          __cuda=12.6=0
               channels : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/nodefaults/linux-64
                          https://conda.anaconda.org/nodefaults/noarch
       base environment : /tmp
               platform : linux-64

I think the problem might rather be due to the resolution of the environment with packages from other channels rather to CUDA since you can run micromamba info successfully.

Can you try reproducing the error with the specification given above? If it does work, could you try coming up with the smallest possible environment specification reproducing the problem (e.g. omitting renopexconda and using a simpler .condarc)?

mverce commented 1 month ago

Just adding that I have a similar problem. When trying to install a package in an existing environment using the command

micromamba install -vvv -n NanoPack -c bioconda nanoplot

the program hangs on an empty line, while the last line of text in the very verbose (-vvv) output relates to the CUDA driver version ("CUDA driver version found: 12.2"), as in the example of @MartinJepsen.

jjerphan commented 1 month ago

Thank you, @mverce.

I can reproduce the problem.

mverce commented 1 month ago

I solved my problem in the meantime: I tried the same thing as before with the minimal miniforge3 install, but had the same problem. The --debug output made me realise that in my case, it was just an enormous, possibly looping, solving task. Installing/pinning a python version made it possible for the tool to provide a solution, but there initially, there were conflicts, which were resolved after setting the channel_priority to "flexible". I've made another test just now and it seems that the flexible setting solves the original problem on its own, but installing/pinning a python version makes the solution appear faster.

Perhaps the --debug option would also be helpful to the OP, as it provides more information than -vvv.

jjerphan commented 1 month ago

I think this issue is due to difference of channel priority (I am always using strict with conda-forge).

When one uses different channels, one should preferably opt for flexible channel resolution, this resolves the problem in this case.

@MartinJepsen: can you try using --no-channel-priority for installing your environment?

jjerphan commented 1 month ago

This might be related to #3393.