search

jupyterhub / the-littlest-jupyterhub

Simple JupyterHub distribution for 1-100 users on a single server
https://tljh.jupyter.org
BSD 3-Clause "New" or "Revised" License
1.03k stars 339 forks source link

Update base user environment to mambaforge 23.1.0-1 (Python 3.10) #858

Closed minrk closed 1 year ago

minrk commented 1 year ago

shifts some duplicated code into utility functions and constants

This significantly updates both conda and mamba. It also updates the default python to 3.10

It still doesn't affect the user environment if it hasn't been updated

closes #864

minrk commented 1 year ago

Still trying to understand the failure when upgrading from too-old conda 4.7. Especially since when I run the test in a Dockerfile, it succeeds!

minrk commented 1 year ago

I'm thinking that probably the right thing to do is to fail explicitly, rather than trying to upgrade.

minrk commented 1 year ago

I'm struggling a bit with this one, particularly what upgrade situations should be handled, and what assumptions should be made.

Past equivalent changes in the initial tljh user env:

And the code suggests that it is attempting to prevent upgrades across those versions, but after writing some tests, I don't believe this is likely to work as intended, with the possible exception of upgrading from the latest version to the latest version.

I believe the changes in #697 prevented upgrades from earlier commits, because they switched ensure_conda to calling mamba, but then use that function to install mamba itself. If that was run on an earlier tljh install, it would fail with mamba: command not found when trying to install mamba itself. I believe that means that no tljh installed before #697 has ever been upgraded past #697 (at least not without manual intervention to install mamba, first).

Further, I've noticed that auto_update_conda is not set to False, so if the user ever ran a conda install command, conda would have been updated, and the pinning logic in the next run to installer.py would not do what was expected. For example, a Python 3.7 env could easily have been upgraded to conda 4.14 or later, which would then get downgraded in the install stage to 4.10.3, not the expected 4.5.8.

So I have these questions about the user environment:

It seems that the goal of the if/else branches we have are to prevent breaking environments by upgrading Python itself too far, but the conda version is used as a proxy for this, when it isn't actually reliable to do so.

So I think what makes the most sense is the following behavior:

  1. specify a minimum supported conda and/or mamba
  2. if it's not present or too old, run a generic conda update conda [mamba] (no version pin, or >=lower bound)
  3. specify a minimum supported Python version and fail if it's too old, refusing to upgrade

This will result in the following different behavior:

  1. conda/mamba will be up-to-date, not pinned to a point in time
  2. if conda has been upgraded by users, it will no longer be downgraded
  3. if an unsupported environment is detected, fail until user intervention (e.g. backup and clear the environment to start fresh), rather than clobber by re-running the latest mambaforge installer

How does that sound?

manics commented 1 year ago

IIRC in #697 I based my changes on the existing code which was also unclear to me!

Regarding upgrades I think your comment about when TLJH should make changes to the user env is key. Any automated upgrade of a user environment package risks "breaking" the user environment, since it may be incompatible with the users' notebooks/code, or it may transitively upgrade another package which is incompatible.

We can't completely avoid updates though, since JupyterHub major releases require a singleuser upgrade.

Now that this repo follows semver could we simplify things?

This means we'd get rid of the multiple version handling, and makes it clear to admins and users when the user environment will change. It also helps avoid some inconsistency e.g. if TLJH needs to update an existing package do we bump them to the minimum required version (minimising the difference from the existing installed versions), or a more recent version as would be installed in a new installation of TLJH?

minrk commented 1 year ago

I think you're points about when we change the user env make sense.

It appears we don't really document the 'upgrading tljh' step and when to do it or what should be expected. Maybe we should start there? I think for the most part, tljh install is only run once at the beginning, and not after. I think that's probably why the upgrade behavior hasn't been dissected too much. It's never really been the point of tljh to serve as a long-running upgraded-in-place distribution, so that use is relatively uncommon.

do we bump them to the minimum required version (minimising the difference from the existing installed versions), or a more recent version as would be installed in a new installation of TLJH?

I think letting the installer pick the latest version is the most likely to be compatible, if we don't pin full environments. The solver will take into account version pins placed there by us or the user (e.g. Python itself), and pick the latest version satisfying existing constraints. This won't always be perfect, as some packages may lack upper bounds, but I think it's more likely to result in a working environment than pinning back only direct dependencies.

I think understanding what cases upgrade is meant to handle will help. For example, it's explicitly supported for users to upgrade Python itself, so we can't use the conda or Python version as an indicator of the previous tljh version.

Maybe we should aim for:

When tljh sees an existing user installation, it will:

  1. check if Python is recent enough, and fail if not (no upper bound until/unless specific incompatibility is found)
  2. ensure conda and mamba are recent enough, and upgrade if not (also no upper bound)
  3. install the required packages, which have major-version pins. If tljh is old, this may involve downgrading some existing packages, as resolved by conda.

This eliminates any multi-version issues because the requirements are clear and relative to the current version requirements (will I work?). I don't think we can take anything about past tljh versions into account, because the user environment appears to be really only designed as a user-upgradeable starting point, not a strictly pinned, static environment.

Another alternative would be to run ensure_conda with a different package spec when an existing env is found, to avoid downgrading of pinned packages, and mostly use lower bounds in that case. It's very likely that any downgrade is undoing an explicit user choice in that case.

I think that only leaves one real case not yet established: what to do if a non-empty user env dir is found with no conda/mamba present - most installers refuse to clobber in this case. In tljh, this directory is generally 'owned' by tljh, so we could make a choice to install over it (this does not delete any files, but it may clobber some in the unlikely event that there are overlapping files).

We also have a choice in the 'fail' case: we could backup the old user env and proceed from scratch instead of halting and waiting for the user. conda directories aren't really relocatable this way, and it can't be guaranteed that the backed up env will fully work without moving it back to the original location. It should at least allow conda list -p BACKUP_PREFIX, to see what packages you had before. This may suit tljh's more automatic design, but it may also be too automatic.

minrk commented 1 year ago

OK, I think this behavior is ready for review. I'd like to update docs, but I think that needs to wait for #863 to avoid conflict.

The new behavior:

I tried running the unbounded upgrade for Python 3.7, and it took a very long time, I think because of conflicts on openssl since the versions are so out of date. This is part of why I set the minimum version to what's in the most recent mambaforge install (3.9).

consideRatio commented 1 year ago

The new behavior:

  • Python version is checked directly. Minimum Python for the user env is 3.9 (the version in 0.2, since 2021). If this requirement is not met, the installer aborts (change from earlier behavior with user envs on old Python, installed prior to 2021)
  • conda/mamba only have lower bounds, not exact pins. They include the previous versions (i.e. will not need updates when upgrading from 0.2). If conda or mamba is out of date or mamba is missing, the packages are upgraded (unbounded). That means these packages will no longer be downgraded during tljh install. But it also means that outdated envs will get updated to 'current' conda/mamba instead of a pinned version. This matches the existing behavior if the user has ever run conda install, since we leave auto_update_conda unset.
  • if the directory exists, is not empty, and does not contain a conda installation, install will abort instead of attempting to clobber the non-environment directory

I appreciate this a lot!!!

Related but off topic, I think we should minimize what is installed in the user environment, and have opened #872 about this.

consideRatio commented 1 year ago

A test failure relates to deprecated use of codecov PyPI package, resolved by #876 that also adds upgrade tests relevant to use to test this PR.

consideRatio commented 1 year ago

Super happy about this work being done @minrk!!! Thank you!!