Strange behavior from mamba - perhaps update it in the base image?

[fperez]

@consideRatio I'm not sure what's going on here, but I am seeing some odd behavior from mamba. In a custom env where I was testing pymc3 I tried to update its companion plotting package, arviz, whose current version as of 1/25/22 on conda-forge is 0.11.4.

When I built the environment, mamba for some reason grabbed 0.11.2:

(pymc3) jupyter-fperez[~]> mamba list | grep arviz
arviz                     0.11.2             pyhd8ed1ab_0    conda-forge

However, mamba tells me this when I try to update:

(pymc3) jupyter-fperez[~]> mamba update arviz

[TRIM LOGO]

Looking for: ['arviz']

conda-forge/linux-64     Using cache
conda-forge/noarch       Using cache

Pinned packages:
  - python 3.9.*

Transaction

  Prefix: /home/jovyan/shared-readwrite/envs/pymc3

  All requested packages already installed

Conda, on the other hand, recognizes there's an 0.11.4:

(pymc3) jupyter-fperez[~]> conda update arviz
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/jovyan/shared-readwrite/envs/pymc3

  added / updated specs:
    - arviz

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    arviz-0.11.4               |     pyhd8ed1ab_0         1.4 MB  conda-forge
    entrypoints-0.3            |  pyhd8ed1ab_1003           8 KB  conda-forge
    pickleshare-0.7.5          |          py_1003           9 KB  conda-forge
    typing-extensions-3.10.0.2 |       hd8ed1ab_0           8 KB  conda-forge
    typing_extensions-3.10.0.2 |     pyha770c72_0          28 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         1.4 MB

The following packages will be UPDATED:

  arviz                                 0.11.2-pyhd8ed1ab_0 --> 0.11.4-pyhd8ed1ab_0
  entrypoints        conda-forge/linux-64::entrypoints-0.3~ --> conda-forge/noarch::entrypoints-0.3-pyhd8ed1ab_1003
  pickleshare        conda-forge/linux-64::pickleshare-0.7~ --> conda-forge/noarch::pickleshare-0.7.5-py_1003

The following packages will be DOWNGRADED:

  typing-extensions                        4.0.1-hd8ed1ab_0 --> 3.10.0.2-hd8ed1ab_0
  typing_extensions                      4.0.1-pyha770c72_0 --> 3.10.0.2-pyha770c72_0

Proceed ([y]/n)? 

I'm not sure what's happening here... On my home machine (macOS), installing arviz with mamba does the same thing conda just showed: it grabs 0.11.4 and downgrades typing_extensions. The only difference I can see is that I have at home mamba 0.20 and the hub has 0.17. I don't know if that's the issue, but it could be relevant.

This isn't urgent, but it's weird behavior. If you have any ideas I'd love to hear...

[fperez]

Awesome, many thanks!!

[consideRatio]

:) The issue was auto-closed by mistake, I figure we verify it helps or not to use mamba 0.20 as well.

I think these kinds of issues are very hard. I've often struggled a lot to resolve them. For example, the nomkl issue started out with a failure to resolve a version like this:

Encountered problems while solving:
  - package mkl-service-2.3.0-py39h07f9747_1 requires mkl >=2020.2,<2021.0a0, but none of the providers can be installed

I didn't manage to debug that effectively by gaining insight on why the solving step failed. I instead figured it out by careful inspection of the packages dependencies, trial and error of installing packages, and finally searching for mkl in the package list as part of the pangeo/pangeo-notebook image and finding nomkl.

I really hope updating to mamba does the trick, because otherwise I'm not sure at all how to proceed.

[fperez]

Yes, let's see and 🤞, but don't worry about it: these rabbit holes can be endless, and this isn't one we need to get lost on. You've already done more than enough!!

[consideRatio]

Sadly I could still reproduce the difference between mamba and conda, and I'm not sure what makes sense to do.

I know that the mamba projects maintainers has described in the past that it can sometimes solve dependencies differently than conda.

Googling a bit, I also conclude that mamba/conda differs with regards to caring much or little about "the history", as indicated by this comment: https://github.com/mamba-org/mamba/issues/1212#issuecomment-942003352

That same history, was cleared recently as part of #100 attempting to install pyMC3, which required uninstall of nomkl, which required clearing of the mentioned history to avoid a segmentation fault.

https://github.com/pangeo-data/jupyter-earth/blob/3bc0e6d698018273ec3156e498cb14e00b22f4f3/hub.jupytearth.org-image/Dockerfile#L164-L173

We still have the segmentation fault after using mamba when trying to uninstall nomkl even with mamba version 0.20.

I figure I'll report the segmentation fault error or look into that a bit.

[consideRatio]

It seems like uninstalling nomkl has led to this issue, https://github.com/mamba-org/mamba/issues/1347, which reproduce at 0.20.0 as well.

[fperez]

Ah, interesting! Thanks for tracking this down - definitely worth reporting upstream to the mamba team. This is the first time I've run into a situation where I find mamba to not "do the right thing" compared to conda. Otherwise my experience with it has been excellent and I use it as a replacement for conda, but this is worth reporting so it can be fixed upstream.

[consideRatio]

Summary

• This bug was reproduced and resolved in mamba, https://github.com/mamba-org/mamba/issues/1402.
• Mamba 0.21+ doesn't have the issue
• The pangeo/pangeo-notebook base image we rely on still has 0.19 in its latest release, but doesn't have it pinned any more (https://github.com/pangeo-data/pangeo-docker-images/pull/282) so we just need a new release of the images until we can remove the workaround we use to get a modern version of mamba. https://github.com/pangeo-data/jupyter-earth/blob/75a62e02f58c21132c8c6c1a394792a3015c9c93/hub.jupytearth.org-image/Dockerfile#L153-L170

Action points

Remove the workaround as soon as we have a new release of the pange/pangeo-notebook base image.

[fperez]

Thanks @consideRatio! For now your manual mamba update mamba is a good solution.