search

mamba-org / mamba

The Fast Cross-Platform Package Manager
https://mamba.readthedocs.io
BSD 3-Clause "New" or "Revised" License
6.68k stars 346 forks source link

Installing packages in different order gives totally different versions #3230

Closed avouacr closed 5 months ago

avouacr commented 5 months ago

Troubleshooting docs

Anaconda default channels

How did you install Mamba?

Mambaforge or latest Miniforge

Search tried in issue tracker

Potentially linked to #3058

Latest version of Mamba

Tried in Conda?

I do not have this problem with Conda, just with Mamba

Describe your issue

Hi, this is an issue I first mistakenly opened on the conda-libmamba-solver depository. It's possibly related to #3058, but not sure either.

I installed mamba through mambaforge and use it to build Python virtual environments.I encouter a weird inconsistency when installing packages through mamba depending on the order I provide.

If I do the following : 

mamba create -n test s3fs boto3 python=3.11

Mamba ends up installing 48 packages with the following versions : 

  + _libgcc_mutex           0.1  conda_forge         conda-forge     Cached
  + python_abi             3.11  4_cp311             conda-forge     Cached
  + ld_impl_linux-64       2.40  h41732ed_0          conda-forge     Cached
  + ca-certificates    2024.2.2  hbcca054_0          conda-forge     Cached
  + libstdcxx-ng         13.2.0  h7e041cc_5          conda-forge     Cached
  + libgomp              13.2.0  h807b86a_5          conda-forge     Cached
  + _openmp_mutex           4.5  2_gnu               conda-forge     Cached
  + libgcc-ng            13.2.0  h807b86a_5          conda-forge     Cached
  + libxcrypt            4.4.36  hd590300_1          conda-forge     Cached
  + libexpat              2.6.1  h59595ed_0          conda-forge     Cached
  + libzlib              1.2.13  hd590300_5          conda-forge     Cached
  + libffi                3.4.2  h7f98852_5          conda-forge     Cached
  + bzip2                 1.0.8  hd590300_5          conda-forge     Cached
  + ncurses                 6.4  h59595ed_2          conda-forge     Cached
  + openssl               3.2.1  hd590300_0          conda-forge     Cached
  + libuuid              2.38.1  h0b41bf4_0          conda-forge     Cached
  + libnsl                2.0.1  hd590300_0          conda-forge     Cached
  + xz                    5.2.6  h166bdaf_0          conda-forge     Cached
  + tk                   8.6.13  noxft_h4845f30_101  conda-forge     Cached
  + libsqlite            3.45.1  h2797004_0          conda-forge     Cached
  + readline                8.2  h8228510_1          conda-forge     Cached
  + tzdata                2024a  h0c530f3_0          conda-forge     Cached
  + python               3.11.8  hab00c5b_0_cpython  conda-forge     Cached
  + wheel                0.42.0  pyhd8ed1ab_0        conda-forge     Cached
  + setuptools           69.1.1  pyhd8ed1ab_0        conda-forge     Cached
  + pip                    24.0  pyhd8ed1ab_0        conda-forge     Cached
  + six                  1.16.0  pyh6c4a22f_0        conda-forge     Cached
  + idna                    3.6  pyhd8ed1ab_0        conda-forge     Cached
  + pysocks               1.7.1  pyha2e5f31_6        conda-forge     Cached
  + jmespath              1.0.1  pyhd8ed1ab_0        conda-forge     Cached
  + attrs                23.2.0  pyh71513ae_0        conda-forge     Cached
  + typing_extensions    4.10.0  pyha770c72_0        conda-forge     Cached
  + fsspec             2024.2.0  pyhca7485f_0        conda-forge     Cached
  + python-dateutil       2.9.0  pyhd8ed1ab_0        conda-forge     Cached
  + aioitertools         0.11.0  pyhd8ed1ab_0        conda-forge     Cached
  + brotli-python         1.1.0  py311hb755f60_1     conda-forge     Cached
  + multidict             6.0.5  py311h459d7ec_0     conda-forge     Cached
  + frozenlist            1.4.1  py311h459d7ec_0     conda-forge     Cached
  + wrapt                1.16.0  py311h459d7ec_0     conda-forge     Cached
  + yarl                  1.9.4  py311h459d7ec_0     conda-forge     Cached
  + urllib3               2.0.7  pyhd8ed1ab_0        conda-forge     Cached
  + aiosignal             1.3.1  pyhd8ed1ab_0        conda-forge     Cached
  + botocore            1.34.51  pyge310_1234567_0   conda-forge     Cached
  + s3transfer           0.10.0  pyhd8ed1ab_0        conda-forge     Cached
  + boto3               1.34.51  pyhd8ed1ab_0        conda-forge     Cached
  + aiohttp               3.9.3  py311h459d7ec_0     conda-forge     Cached
  + aiobotocore          2.12.1  pyhd8ed1ab_0        conda-forge     Cached
  + s3fs               2024.2.0  pyhd8ed1ab_0        conda-forge     Cached

Now if I just change the order of the packages when installing from :

mamba create -n test s3fs boto3 python=3.11

to :

mamba create -n test boto3 python=3.11 s3fs

mamba installs only 37 packages and I get the following versions specs :

  + _libgcc_mutex          0.1  conda_forge         conda-forge     Cached
  + ld_impl_linux-64      2.40  h41732ed_0          conda-forge     Cached
  + ca-certificates   2024.2.2  hbcca054_0          conda-forge     Cached
  + libstdcxx-ng        13.2.0  h7e041cc_5          conda-forge     Cached
  + python_abi            3.11  4_cp311             conda-forge     Cached
  + libgomp             13.2.0  h807b86a_5          conda-forge     Cached
  + _openmp_mutex          4.5  2_gnu               conda-forge     Cached
  + libgcc-ng           13.2.0  h807b86a_5          conda-forge     Cached
  + libxcrypt           4.4.36  hd590300_1          conda-forge     Cached
  + libexpat             2.6.1  h59595ed_0          conda-forge     Cached
  + libzlib             1.2.13  hd590300_5          conda-forge     Cached
  + libffi               3.4.2  h7f98852_5          conda-forge     Cached
  + bzip2                1.0.8  hd590300_5          conda-forge     Cached
  + ncurses                6.4  h59595ed_2          conda-forge     Cached
  + openssl              3.2.1  hd590300_0          conda-forge     Cached
  + libuuid             2.38.1  h0b41bf4_0          conda-forge     Cached
  + libnsl               2.0.1  hd590300_0          conda-forge     Cached
  + xz                   5.2.6  h166bdaf_0          conda-forge     Cached
  + tk                  8.6.13  noxft_h4845f30_101  conda-forge     Cached
  + libsqlite           3.45.1  h2797004_0          conda-forge     Cached
  + readline               8.2  h8228510_1          conda-forge     Cached
  + tzdata               2024a  h0c530f3_0          conda-forge     Cached
  + python              3.11.8  hab00c5b_0_cpython  conda-forge     Cached
  + wheel               0.42.0  pyhd8ed1ab_0        conda-forge     Cached
  + setuptools          69.1.1  pyhd8ed1ab_0        conda-forge     Cached
  + pip                   24.0  pyhd8ed1ab_0        conda-forge     Cached
  + six                 1.16.0  pyh6c4a22f_0        conda-forge     Cached
  + pysocks              1.7.1  pyha2e5f31_6        conda-forge     Cached
  + fsspec            2024.2.0  pyhca7485f_0        conda-forge     Cached
  + jmespath             1.0.1  pyhd8ed1ab_0        conda-forge     Cached
  + python-dateutil      2.9.0  pyhd8ed1ab_0        conda-forge     Cached
  + brotli-python        1.1.0  py311hb755f60_1     conda-forge     Cached
  + urllib3              2.0.7  pyhd8ed1ab_0        conda-forge     Cached
  + botocore           1.34.60  pyge310_1234567_0   conda-forge     Cached
  + s3transfer          0.10.0  pyhd8ed1ab_0        conda-forge     Cached
  + boto3              1.34.60  pyhd8ed1ab_0        conda-forge     Cached
  + s3fs                 0.4.2  py_0                conda-forge     Cached

Now what I don't understand is how such a small change of order can have such a dramatic impact on what gets installed in the end. In particular, the version of s3fs installed in the second configuration is almost 4 years old, whereas it's the latest one in the first configuration.

There seems to be somewhat a conflict of dependencies between s3fs et boto3 because if I create the environment in the second configuration and then runs mamba install s3fs=2024.2.0 in it, I get the following changes : 

  Upgrade:
────────────────────────────────────────────────────────────────────────────

  - s3fs                  0.4.2  py_0               conda-forge     Cached
  + s3fs               2024.2.0  pyhd8ed1ab_0       conda-forge     Cached

  Downgrade:
────────────────────────────────────────────────────────────────────────────

  - botocore            1.34.60  pyge310_1234567_0  conda-forge     Cached
  + botocore            1.34.51  pyge310_1234567_0  conda-forge     Cached
  - boto3               1.34.60  pyhd8ed1ab_0       conda-forge     Cached
  + boto3               1.34.51  pyhd8ed1ab_0       conda-forge     Cached

But in the optimization process that mamba does, I don't understand how a huge major downgrading of s3fs can be prioritized over downgrading boto3 from a few patches, and especially why the result differs so much even when they are simultaneously installed.

The outcome is different when I use conda to create my environment (e.g. conda create -n test s3fs boto3 python=3.11) : it is consistent across orderings in the sense that the same number of packages and versions are installed. However, in both configurations, it installs the version 0.4.2 of s3fs, prioritizing getting the latest of boto3 while accepting a very outdated version for s3fs.

I wonder if it could be related to the fact that s3fs changed their versioning system at some point on conda-forge (see here), going from usual semantic versioning to date-based one. But still the major of the subsequent versions are (way) higher than the previous ones, so I don't see why this should produce this outcome.

Thank's !

mamba info / micromamba info

mamba version : 1.5.7
     active environment : test
    active env location : /opt/mamba/envs/test
            shell level : 2
       user config file : /home/onyxia/.condarc
 populated config files : /opt/mamba/.condarc
          conda version : 24.1.2
    conda-build version : not installed
         python version : 3.11.6.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake_avx512
                          __conda=24.1.2=0
                          __glibc=2.35=0
                          __linux=6.1.0=0
                          __unix=0=0
       base environment : /opt/mamba  (writable)
      conda av data dir : /opt/mamba/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /opt/mamba/pkgs
                          /home/onyxia/.conda/pkgs
       envs directories : /opt/mamba/envs
                          /home/onyxia/.conda/envs
               platform : linux-64
             user-agent : conda/24.1.2 requests/2.31.0 CPython/3.11.6 Linux/6.1.0-16-amd64 ubuntu/22.04.4 glibc/2.35 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.7
                UID:GID : 1000:100
             netrc file : None
           offline mode : False

Logs

No response

environment.yml

No response

~/.condarc

No response

AntoinePrv commented 5 months ago

This is due to libsolv, the dependency resolver. We added a fix around this in the upcoming 2.0.