--force-reinstall old conda to ensure it's working before we try to install conda packages

[minrk]

an old conda bug causes RemoveError: requests is a dependency of conda when installing other packages (closes #918).

Upgrading conda itself first avoids this bug (we may want --force here, which some folks have seen required when conda itself is messed up, but let's avoid it for now). So we know that the version in 0.2.0 (4.10) is not actually new enough to work. It's unclear what the true minimum version is to fix this, but the important thing is that it will be upgraded if it's still the version in 0.2.0 (4.10), so pick the versions from today which we know works, and will be the version upgraded-to if an 0.2 install has yet to update conda.

conda install --force-reinstall conda is the fix for this, because conda install conda will fail to upgrade if we're in this situation. We could make it conditional on old condas, as I don't think this is likely to come up with more recent versions, but it should be safe in general to run this command.

[minrk]

It appears to be a little tricky to upgrade conda itself when the env is considered broken like this. Working on it.

[minrk]

If we don't want to bump versions, it appears the fix is specifically:

conda install --force-reinstall conda

which should ensure that conda itself is in a consistent state (and should not upgrade it). We could do that as a separate, unconditional 'ensure conda is okay' step before installing anything, but I believe using --force-reinstall to ensure that conda/mamba are super consistent ought to be good enough.

We could also try to restrict 'needs force' to a version constraint, but I have no idea what that constraint would be.

[consideRatio]

We could also try to restrict 'needs force' to a version constraint, but I have no idea what that constraint would be.

I think its great if we avoid introducing that precautionary, as it makes the situations that can arise grow and its harder to ensure function in all situations. For the same reasons, I'm happy about seeing more stringent requirements on the conda/mamba versions.

[minrk]

--force-reinstall behaves a bit weird. It doens't upgrade packages, so unconstrained versions result in no upgrade unless the version constraint is in the install command.

I think maybe it makes sense to unconditionally force-reinstall conda/mamba to ensure it's always consistent, followed by the upgrade steps.

[minrk]

Struggling to get the right behavior here to match elaborate test expectations. It seems --force-reinstall does upgrade when run with old versions, unlike more recent ones.

It really seems the 'reinstall if needed' is what we want, but I can't seem to figure out 'if needed'.

[consideRatio]

I searched on reinstall in https://github.com/conda/conda/blob/main/CHANGELOG.md, and found one entry released for 4.4.0 that included at least some kind of change https://github.com/conda/conda/pull/5836.

[minrk]

phew everything's passing and test expectations are unchanged.

One thing that might help these things (or might not, honestly, since it creates an extra env to manage) is to make the root conda install a separate environment from the user env:

• conda root env - /opt/tljh/conda
• user env created with /opt/tljh/conda/bin/conda create --prefix /opt/tljh/user

Then conda/mamba themselves won't actually interact with user packages. This is where the pip install step caused problems by upgrading requests, a dependency of conda itself. There would be no reason to have pip packages in the root env if we did this.

[minrk]

all set from my side, yes

[consideRatio]

I'd love to see tljh 1.0.0 installations - either fresh installs or upgrades, arrive to an even more solid baseline that we can count on going onwards. But that would be a separate discussion!

One thing that might help these things (or might not, honestly, since it creates an extra env to manage) is to make the root conda install a separate environment from the user env:

Yikes, as a maintainer I just feel that I've caught up with the understanding of:

• system env (bootstrap.py runs here, setups hub and user env)
• hub env (tljh installer runs here, setups jupyterhub in the env)
• user env (initialized with misc things, but the only essential part is jupyterhub singleuser logic)

I think the problems solved should be major to justify an additional environment because of the complexity of even just understanding the purpose of each, and documenting that, etc.

[yuvipanda]

btw, I'm still seeing this on https://github.com/jupyterhub/the-littlest-jupyterhub/actions/runs/6357501669/job/17268703149#step:5:157 from https://github.com/jupyterhub/the-littlest-jupyterhub/pull/942 (which is based on latest main)