search

conda / conda-lock

Lightweight lockfile for conda environments
https://conda.github.io/conda-lock/
Other
457 stars 102 forks source link

Cannot generate lock for project depending on Dask #539

Closed robertodr closed 8 months ago

robertodr commented 8 months ago

Checklist

What happened?

I'm using the latest version of conda-lock (2.4.1) and trying to generate a lockfile using an environment.yml file:

dependencies:
  - blas=*=*openblas*
  - c-compiler
  - cmake
  - cxx-compiler
  - libopenblas=*=*openmp*
  - pip
  - python >=3.10,<3.11

and a pyproject.toml file:

[project]
name = "example"
description = ""
requires-python = ">=3.10,<3.12"
dependencies = [
  "numpy>=1.23",
  "scipy~=1.11",
  "numba>=0.58",
  "dask[array,diagnostics,distributed]>=2023.1.0",
]

[build-system]
build-backend = "setuptools.build_meta"
requires = [
  "setuptools>=64",
]

[tool.conda-lock]
channels = [
  "conda-forge",
]
default-non-conda-source = "pip"

I use micromamba and run the following command:

micromamba run -n lock conda-lock --micromamba --kind lock --no-dev-dependencies --filter-extras --without-cuda -f env.yml -f pyproject.toml -p linux-64

The error I get is:

INFO:conda_lock.conda_solver:linux-64 using specs ['blas * *openblas*', 'c-compiler', 'cmake', 'cxx-compiler', 'libopenblas * *openmp*', 'pip', 'python >=3.10,<3.11']
Traceback (most recent call last):
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/conda_lock/_vendor/poetry/puzzle/solver.py", line 233, in _solve
    result = resolve_version(
             ^^^^^^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/conda_lock/_vendor/poetry/mixology/__init__.py", line 7, in resolve_version
    return solver.solve()
           ^^^^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 83, in solve
    self._propagate(next)
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 123, in _propagate
    root_cause = self._resolve_conflict(incompatibility)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 321, in _resolve_conflict
    raise SolveFailure(incompatibility)
conda_lock._vendor.poetry.mixology.failure.SolveFailure: Because -dummy-package- depends on dask-core (>=2023.1.0) which doesn't match any versions, version solving failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/shared/home/robertoeikas/micromamba/envs/lock/bin/conda-lock", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 1384, in lock
    lock_func(
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 1092, in run_lock
    make_lock_files(
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 394, in make_lock_files
    fresh_lock_content = create_lockfile_from_spec(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 821, in create_lockfile_from_spec
    deps = _solve_for_arch(
           ^^^^^^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 752, in _solve_for_arch
    pip_deps = solve_pypi(
               ^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/conda_lock/pypi_solver.py", line 351, in solve_pypi
    result = s.solve(use_latest=to_update)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/conda_lock/_vendor/poetry/puzzle/solver.py", line 65, in solve
    packages, depths = self._solve(use_latest=use_latest)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/home/robertoeikas/micromamba/envs/lock/lib/python3.11/site-packages/conda_lock/_vendor/poetry/puzzle/solver.py", line 241, in _solve
    raise SolverProblemError(e)
conda_lock._vendor.poetry.puzzle.exceptions.SolverProblemError: Because -dummy-package- depends on dask-core (>=2023.1.0) which doesn't match any versions, version solving failed.

It seems this has been observed before (#221 and #253) and that #290 should have fixed it, so there must be something wrong that I'm doing.

Conda Info

libmamba version : 1.5.1
     micromamba version : 1.5.1
           curl version : libcurl/7.88.1 OpenSSL/3.1.2 zlib/1.2.13 zstd/1.5.5 libssh2/1.11.0 nghttp2/1.52.0
     libarchive version : libarchive 3.6.2 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.2
       envs directories : /shared/home/robertoeikas/micromamba/envs
          package cache : /shared/home/robertoeikas/micromamba/pkgs
                          /shared/home/robertoeikas/.mamba/pkgs
            environment : None (not found)
           env location : -
      user config files : /shared/home/robertoeikas/.mambarc
 populated config files : /shared/home/robertoeikas/.condarc
       virtual packages : __unix=0=0
                          __linux=4.18.0=0
                          __glibc=2.28=0
                          __archspec=1=x86_64
               channels : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/nodefaults/linux-64
                          https://conda.anaconda.org/nodefaults/noarch
       base environment : /shared/home/robertoeikas/micromamba
               platform : linux-64

Conda Config

Configuration files (by precedence order):
~/.condarc

Conda list

List of packages in environment: "/shared/home/robertoeikas/micromamba/envs/lock"

  Name                         Version    Build               Channel    
───────────────────────────────────────────────────────────────────────────
  _libgcc_mutex                0.1        conda_forge         conda-forge
  _openmp_mutex                4.5        2_gnu               conda-forge
  annotated-types              0.6.0      pyhd8ed1ab_0        conda-forge
  appdirs                      1.4.4      pyh9f0ad1d_0        conda-forge
  brotli-python                1.1.0      py311hb755f60_1     conda-forge
  bzip2                        1.0.8      h7f98852_4          conda-forge
  ca-certificates              2023.7.22  hbcca054_0          conda-forge
  cachecontrol                 0.13.1     pyhd8ed1ab_0        conda-forge
  cachecontrol-with-filecache  0.13.1     pyhd8ed1ab_0        conda-forge
  cachy                        0.3.0      pyhd8ed1ab_1        conda-forge
  certifi                      2023.7.22  pyhd8ed1ab_0        conda-forge
  cffi                         1.16.0     py311hb3a22ac_0     conda-forge
  charset-normalizer           3.3.1      pyhd8ed1ab_0        conda-forge
  click                        8.1.7      unix_pyh707e725_0   conda-forge
  click-default-group          1.2.4      pyhd8ed1ab_0        conda-forge
  clikit                       0.6.2      pyhd8ed1ab_2        conda-forge
  conda-lock                   2.4.1      pyhd8ed1ab_0        conda-forge
  crashtest                    0.4.1      pyhd8ed1ab_0        conda-forge
  cryptography                 41.0.5     py311h63ff55d_0     conda-forge
  dbus                         1.13.6     h5008d03_3          conda-forge
  distlib                      0.3.7      pyhd8ed1ab_0        conda-forge
  ensureconda                  1.4.3      pyhd8ed1ab_0        conda-forge
  expat                        2.5.0      hcb278e6_1          conda-forge
  filelock                     3.12.4     pyhd8ed1ab_0        conda-forge
  gettext                      0.21.1     h27087fc_0          conda-forge
  gitdb                        4.0.11     pyhd8ed1ab_0        conda-forge
  gitpython                    3.1.40     pyhd8ed1ab_0        conda-forge
  html5lib                     1.1        pyh9f0ad1d_0        conda-forge
  idna                         3.4        pyhd8ed1ab_0        conda-forge
  importlib-metadata           6.8.0      pyha770c72_0        conda-forge
  importlib_metadata           6.8.0      hd8ed1ab_0          conda-forge
  jaraco.classes               3.3.0      pyhd8ed1ab_0        conda-forge
  jeepney                      0.8.0      pyhd8ed1ab_0        conda-forge
  jinja2                       3.1.2      pyhd8ed1ab_1        conda-forge
  keyring                      24.2.0     py311h38be061_1     conda-forge
  ld_impl_linux-64             2.40       h41732ed_0          conda-forge
  libexpat                     2.5.0      hcb278e6_1          conda-forge
  libffi                       3.4.2      h7f98852_5          conda-forge
  libgcc-ng                    13.2.0     h807b86a_2          conda-forge
  libglib                      2.78.0     hebfc3b9_0          conda-forge
  libgomp                      13.2.0     h807b86a_2          conda-forge
  libiconv                     1.17       h166bdaf_0          conda-forge
  libnsl                       2.0.1      hd590300_0          conda-forge
  libsqlite                    3.43.2     h2797004_0          conda-forge
  libstdcxx-ng                 13.2.0     h7e041cc_2          conda-forge
  libuuid                      2.38.1     h0b41bf4_0          conda-forge
  libzlib                      1.2.13     hd590300_5          conda-forge
  markupsafe                   2.1.3      py311h459d7ec_1     conda-forge
  more-itertools               10.1.0     pyhd8ed1ab_0        conda-forge
  msgpack-python               1.0.6      py311h9547e67_0     conda-forge
  ncurses                      6.4        hcb278e6_0          conda-forge
  openssl                      3.1.4      hd590300_0          conda-forge
  packaging                    23.2       pyhd8ed1ab_0        conda-forge
  pastel                       0.2.1      pyhd8ed1ab_0        conda-forge
  pcre2                        10.40      hc3806b6_0          conda-forge
  pip                          23.3.1     pyhd8ed1ab_0        conda-forge
  pkginfo                      1.9.6      pyhd8ed1ab_0        conda-forge
  platformdirs                 3.11.0     pyhd8ed1ab_0        conda-forge
  pycparser                    2.21       pyhd8ed1ab_0        conda-forge
  pydantic                     2.4.2      pyhd8ed1ab_1        conda-forge
  pydantic-core                2.10.1     py311h46250e7_0     conda-forge
  pylev                        1.4.0      pyhd8ed1ab_0        conda-forge
  pysocks                      1.7.1      pyha2e5f31_6        conda-forge
  python                       3.11.6     hab00c5b_0_cpython  conda-forge
  python_abi                   3.11       4_cp311             conda-forge
  pyyaml                       6.0.1      py311h459d7ec_1     conda-forge
  readline                     8.2        h8228510_1          conda-forge
  requests                     2.31.0     pyhd8ed1ab_0        conda-forge
  ruamel.yaml                  0.18.2     py311h459d7ec_0     conda-forge
  ruamel.yaml.clib             0.2.7      py311h459d7ec_2     conda-forge
  secretstorage                3.3.3      py311h38be061_2     conda-forge
  setuptools                   68.2.2     pyhd8ed1ab_0        conda-forge
  six                          1.16.0     pyh6c4a22f_0        conda-forge
  smmap                        5.0.0      pyhd8ed1ab_0        conda-forge
  tk                           8.6.13     h2797004_0          conda-forge
  tomli                        2.0.1      pyhd8ed1ab_0        conda-forge
  tomlkit                      0.12.1     pyha770c72_0        conda-forge
  toolz                        0.12.0     pyhd8ed1ab_0        conda-forge
  typing-extensions            4.8.0      hd8ed1ab_0          conda-forge
  typing_extensions            4.8.0      pyha770c72_0        conda-forge
  tzdata                       2023c      h71feb2d_0          conda-forge
  urllib3                      1.26.18    pyhd8ed1ab_0        conda-forge
  virtualenv                   20.24.6    pyhd8ed1ab_0        conda-forge
  webencodings                 0.5.1      pyhd8ed1ab_2        conda-forge
  wheel                        0.41.2     pyhd8ed1ab_0        conda-forge
  xz                           5.2.6      h166bdaf_0          conda-forge
  yaml                         0.2.5      h7f98852_2          conda-forge
  zipp                         3.17.0     pyhd8ed1ab_0        conda-forge

Additional Context

No response

maresb commented 8 months ago

Thanks for the report. Unfortunately this sort of issue is not so easily solved.

What's your motivation for using default-non-conda-source = "pip"? Everything except numba >=0.58 should be on conda-forge.

If you want to manage packages with pip then you may have better luck with the recently-added skip-non-conda-lock = true feature and running pip install separately.

robertodr commented 8 months ago

Thanks for your reply! Indeed I wanted to do something like what skip-non-conda-lock = true would do, but on second thought, it's better to explicitly list the dependencies that aren't on conda-forge.

maresb commented 8 months ago

As long as the dependencies are open-source licensed, it's usually not difficult to add them to conda-forge. (Generate a recipe with grayskull and submit a PR to staged-recipes.)