make update link more reliable

[gdementen]

Unsure we can do anything about this, but here is a dump of what I learned today by diagnosing the windows menu "update larray" link not working for 3 out of 4 of my conda environments.

The symptom was that "conda update larrayenv" returned # All requested packages already installed. when the larray* packages were at version 0.34.2, while the latest version is 0.34.3

It turns out the conditions to trigger the bug is to have 1) at least one of the larrayenv subpackages (larray, larray-editor or larray_eurostat) explicitly (and thus not automatically via larrayenv) installed somewhere in the history of the environment and 2) use a relatively recent version of conda (version 23.10 or later).

Version 23.10 made the "new" libmamba solver the default. This solver has a weird behavior in that case, because if a "package spec" is unversioned, it tries to avoid updating that package at all, unless it is part of the packages the user explicitly asked to update.

So, what happens when we try to update larrayenv from 0.34.2 to its latest (0.34.3), but, for example, larray was explicitly installed too and thus its version is frozen to its currently installed version (0.34.2). Given that larrayenv 0.34.3 depends on exactly larray 0.34.3, it makes larrayenv 0.34.2 the latest installable version.

One workaround is to use the classic solver instead via: conda update larrayenv --solver=classic but the problem is that the update link currently on users computers does not use that.

References:

• https://conda.github.io/conda-libmamba-solver/user-guide/libmamba-vs-classic/

• Using -vv on conda update (conda update -vv larrayenv) can somewhat help diagnose the issue (it is the only way I found to actually see via conda itself those damn package specs). Among a lot of output, you get this interesting bit:

  {
  "ADD_PIN": [
  "python 3.8.*"
  ],
  "INSTALL": [
  "pandas",
  "larray-editor"
  ],
  "UPDATE": [
  "larrayenv",
  "openssl",
  "ca-certificates"
  ],
  "ALLOW_UNINSTALL": [
  "larrayenv",
  "openssl",
  "ca-certificates"
  ],
  "USERINSTALLED": [
  "larray-project/win-64::larrayenv==0.34.2=0",
  "pkgs/main/win-64::python==3.8.18=h1aa4202_0",
  "pkgs/main/win-64::pandas==2.0.3=py38h4ed8f06_0",
  "larray-project/noarch::larray-editor==0.34.2=py_0",
  "pkgs/main/win-64::openssl==3.0.14=h827c3e9_0",
  "pkgs/main/win-64::ca-certificates==2024.7.2=haa95532_0"
  ]
  }

• the current package specs are actually reconstructed every time from the entire environment history found in <miniconda path>\envs\<env name>\conda-meta\history. Look for # update specs: xxx and # remove specs: xxx and # neutered specs: xxx. I find this completely crazy but this is the way it is currently (I successfully solved my update issue by editing this file by hand).

I don't know what is the best way forward. Removing all larray subpackages and larrayenv via conda remove larray larray-editor larray_eurostat larrayenv and then reinstalling just larrayenv fixes the problem for a particular environment but that would have to be done manually on each of our users computers and does not prevent the problem from reappearing later.

Using --solver=classic in the update link would insulate us from such problems for a while (but would not solve already broken environments on our users computers because the update command is the one form the current version they have), but it makes conda much slower and I suspect the option will probably disappear some day.

[gdementen]

Other options which I would need to test is to explicitly update all larray subpackages in addition to larrayenv: conda update larrayenv larray larray-editor larray_eurostat or even use conda update --all.

[alixdamman]

Just for the update link, can't we just do conda update larray larray-editor larray_eurostat ? I wonder if, on the long run, keeping larrayenv worth the time you spend on (trying) to maintain it ?

[gdementen]

Hmm, that's an option I did not think about, but I do not really like it, because larrayenv contains more than just larray, larray-editor and larray_eurostat: it also contains all larray optional dependencies, which is very handy.

Making all those optional dependencies non-optional (at larray level) would make it more realistic to get rid of larrayenv but I wonder if that would not cause us other problems.

We should really solve larray-project/larray#704, which would render larrayenv useless (at least internally)...