Issue with mixed conda/pip dependencies when a name does not mean the same package in conda and pip

lesteve commented 1 year ago

Checklist

[X] I added a descriptive title
[X] I searched open reports and couldn't find a duplicate

What happened?

This is another conda/pip mixed dependencies issue, that happens when the same name in conda and pip does not mean the same package.

In our case as noted in https://github.com/scikit-learn/scikit-learn/pull/26184#issuecomment-1510882005, the same name tzdata does not mean the same thing in conda and pip:

conda has tzdata (latest version 2023c) and python-tzdata (latest version 2023.3)
pip has tzdata (latest version 2023.3) which is the same as python-tzdata in conda
somehow poetry tries to find tzdata with version 2023c, which is version of conda tzdata not pip tzdata so there is an error, it should be 2023.3 and not 2023c.

This is taken from https://github.com/conda/conda-lock/issues/179#issuecomment-1511002379 since I believe this is a slightly separate issue as the other ones listed in #179 and #253. Also I checked that #290 does not fix this issue. #398 is attempting to solve this issue but I don't believe this is the right approach.

This was seen in a scikit-learn context where we make heavy use of conda-lock for controling the CI environments.

To reproduce (this is indeed a mixed conda/pip dependencies is because python depends on tzdata)

# environment.yml
channels:
  - conda-forge
dependencies:
  - python=3.9
  - pip
  - pip:
    - tzdata

conda-lock lock --log-level DEBUG -f environment.yml -p linux-64

Error:

conda_lock._vendor.poetry.repositories.exceptions.PackageNotFound: Package tzdata (2023c) not found.

Full traceback

```python-tb Traceback (most recent call last): File "/home/lesteve/miniconda3/envs/conda-lock-dev/bin/conda-lock", line 8, in sys.exit(main()) File "/home/lesteve/miniconda3/envs/conda-lock-dev/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/home/lesteve/miniconda3/envs/conda-lock-dev/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/home/lesteve/miniconda3/envs/conda-lock-dev/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/lesteve/miniconda3/envs/conda-lock-dev/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/lesteve/miniconda3/envs/conda-lock-dev/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/home/lesteve/miniconda3/envs/conda-lock-dev/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 1267, in lock lock_func( File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 997, in run_lock make_lock_files( File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 360, in make_lock_files lock_content = lock_content | create_lockfile_from_spec( File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 747, in create_lockfile_from_spec deps = _solve_for_arch( File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 683, in _solve_for_arch pip_deps = solve_pypi( File "/home/lesteve/dev/conda-lock/conda_lock/pypi_solver.py", line 267, in solve_pypi result = s.solve(use_latest=to_update) File "/home/lesteve/dev/conda-lock/conda_lock/_vendor/poetry/puzzle/solver.py", line 65, in solve packages, depths = self._solve(use_latest=use_latest) File "/home/lesteve/dev/conda-lock/conda_lock/_vendor/poetry/puzzle/solver.py", line 233, in _solve result = resolve_version( File "/home/lesteve/dev/conda-lock/conda_lock/_vendor/poetry/mixology/__init__.py", line 7, in resolve_version return solver.solve() File "/home/lesteve/dev/conda-lock/conda_lock/_vendor/poetry/mixology/version_solver.py", line 84, in solve next = self._choose_package_version() File "/home/lesteve/dev/conda-lock/conda_lock/_vendor/poetry/mixology/version_solver.py", line 400, in _choose_package_version version = self._provider.complete_package(version) File "/home/lesteve/dev/conda-lock/conda_lock/_vendor/poetry/puzzle/provider.py", line 433, in complete_package self._pool.package( File "/home/lesteve/dev/conda-lock/conda_lock/_vendor/poetry/repositories/pool.py", line 149, in package raise PackageNotFound("Package {} ({}) not found.".format(name, version)) conda_lock._vendor.poetry.repositories.exceptions.PackageNotFound: Package tzdata (2023c) not found. ```

Conda Info

active environment : conda-lock-dev
    active env location : /home/lesteve/miniconda3/envs/conda-lock-dev
            shell level : 1
       user config file : /home/lesteve/.condarc
 populated config files : /home/lesteve/miniconda3/.condarc
                          /home/lesteve/.condarc
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.11.3.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.35=0
                          __linux=5.15.0=0
                          __unix=0=0
       base environment : /home/lesteve/miniconda3  (writable)
      conda av data dir : /home/lesteve/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/lesteve/miniconda3/pkgs
                          /home/lesteve/.conda/pkgs
       envs directories : /home/lesteve/miniconda3/envs
                          /home/lesteve/.conda/envs
               platform : linux-64
             user-agent : conda/23.3.1 requests/2.28.2 CPython/3.11.3 Linux/5.15.0-70-generic ubuntu/22.04.2 glibc/2.35
                UID:GID : 202587:26082
             netrc file : None
           offline mode : False

Conda Config

==> /home/lesteve/miniconda3/.condarc <==
channels:
  - conda-forge

==> /home/lesteve/.condarc <==
changeps1: False
channels:
  - conda-forge

Conda list

# packages in environment at /home/lesteve/miniconda3/envs/conda-lock-dev:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
brotlipy                  0.7.0           py310h6acc77f_1003    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
cachecontrol              0.12.10            pyhd8ed1ab_0    conda-forge
cachy                     0.3.0                      py_0    conda-forge
certifi                   2021.10.8       py310hff52083_1    conda-forge
cffi                      1.15.0          py310h0fdd8cc_0    conda-forge
charset-normalizer        2.0.12             pyhd8ed1ab_0    conda-forge
cleo                      0.8.1              pyhd8ed1ab_2    conda-forge
click                     8.1.0           py310hff52083_0    conda-forge
click-default-group       1.2.2              pyhd8ed1ab_1    conda-forge
clikit                    0.6.2              pyh9f0ad1d_0    conda-forge
conda-lock                1.4.1.dev111+g684f00e          pypi_0    pypi
crashtest                 0.3.1              pyhd8ed1ab_0    conda-forge
cryptography              36.0.2          py310h597c629_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
distlib                   0.3.4              pyhd8ed1ab_0    conda-forge
ensureconda               1.4.2              pyhd8ed1ab_0    conda-forge
expat                     2.4.7                h27087fc_0    conda-forge
filelock                  3.10.0                   pypi_0    pypi
gettext                   0.19.8.1          h73d1719_1008    conda-forge
gitdb                     4.0.10                   pypi_0    pypi
gitpython                 3.1.31                   pypi_0    pypi
html5lib                  1.1                pyh9f0ad1d_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.3          py310hff52083_0    conda-forge
importlib_metadata        4.11.3               hd8ed1ab_0    conda-forge
jeepney                   0.7.1              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.1              pyhd8ed1ab_0    conda-forge
keyring                   23.4.0          py310hff52083_2    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 11.2.0              h1d223b6_14    conda-forge
libglib                   2.70.2               h174f98d_4    conda-forge
libgomp                   11.2.0              h1d223b6_14    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_14    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
lockfile                  0.12.2                     py_1    conda-forge
markupsafe                2.1.1           py310h5764c6d_0    conda-forge
msgpack-python            1.0.3           py310h91b1402_0    conda-forge
ncurses                   6.3                  h9c3ff4c_0    conda-forge
openssl                   1.1.1n               h166bdaf_0    conda-forge
packaging                 20.9               pyh44b312d_0    conda-forge
pastel                    0.2.1              pyhd8ed1ab_0    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pip                       22.0.4             pyhd8ed1ab_0    conda-forge
pkginfo                   1.8.2              pyhd8ed1ab_0    conda-forge
platformdirs              2.5.1              pyhd8ed1ab_0    conda-forge
poetry                    1.1.13          py310hff52083_0    conda-forge
poetry-core               1.0.8           py310hff52083_0    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  1.9.0           py310h6acc77f_0    conda-forge
pylev                     1.4.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.7              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1           py310hff52083_4    conda-forge
python                    3.10.4          h9a8a25e_0_cpython    conda-forge
python_abi                3.10                    2_cp310    conda-forge
pyyaml                    6.0             py310h6acc77f_3    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
requests                  2.27.1             pyhd8ed1ab_0    conda-forge
requests-toolbelt         0.9.1                      py_0    conda-forge
ruamel.yaml               0.17.21         py310h6acc77f_0    conda-forge
ruamel.yaml.clib          0.2.6           py310h6acc77f_0    conda-forge
secretstorage             3.3.1           py310hff52083_1    conda-forge
setuptools                61.2.0          py310hff52083_1    conda-forge
shellingham               1.4.0              pyh44b312d_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     5.0.0                    pypi_0    pypi
sqlite                    3.37.1               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1                    pypi_0    pypi
tomlkit                   0.10.1             pyha770c72_0    conda-forge
toolz                     0.12.0                   pypi_0    pypi
typing                    3.10.0.0           pyhd8ed1ab_0    conda-forge
typing-extensions         4.1.1                hd8ed1ab_0    conda-forge
typing_extensions         4.1.1              pyha770c72_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
urllib3                   1.26.9             pyhd8ed1ab_0    conda-forge
virtualenv                20.14.0         py310hff52083_0    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zipp                      3.7.0              pyhd8ed1ab_1    conda-forge
zlib                      1.2.11            h166bdaf_1014    conda-forge

Additional Context

No response

maresb commented 1 year ago

Thank you very much for this!!! I'm really hoping to get a few hours with the debugger this weekend to try and find the root of the issue. I've had the feeling that there were edge cases like this we were missing, so this will be extremely useful!

lesteve commented 1 year ago

Thanks a lot for your answer!

I think #290 already fixes a lot of mixed conda/pip dependencies so I would say merging #290 is higher priority. The reason I opened this issue is that it does not get forgotten once #290 is merged.

But of course, if you can find the time to look at this issue, now that you have wrapped your head around this part of the code, this would be more than welcome :wink:

glemaitre commented 1 year ago

Actually, the recent failure in main looks really similar to this problem: https://github.com/conda/conda-lock/actions/runs/4812192697/jobs/8567116188#step:6:475

lesteve commented 1 year ago

Yep indeed, tzdata ends up being both in the conda and pip section but they do not mean the same thing.

I think the root cause that this test started failing is the same as the one we saw in scikit-learn, namely pandas 2.0 added a tzdata dependency.

For completeness, the environment used by the failing test is https://github.com/conda/conda-lock/blob/main/tests/test-pypi-resolve-namediff/environment.yml

in the conda section: python depends on tzdata
in the pip section: geogif depends on pandas which depends on tzdata (for pandas>=2.0)

lesteve commented 1 year ago

This has been fixed in main by #290, in particular see https://github.com/conda/conda-lock/pull/290#pullrequestreview-1431090574 :tada:.

conda / conda-lock