Install python into a venv

[yuvipanda]

• Continues to use python and venv from Ubuntu LTS repositories, so they are supported as with everything else that is gotten from apt (see https://github.com/rocker-org/rocker-versioned2/issues/670#issuecomment-1619748511)
• Doesn't currently change any permissions, so present behavior is preserved. However, in the future, we should probably change ownership so end users can install packages in there at runtime (see https://github.com/rocker-org/rocker-versioned2/issues/670#issuecomment-1624691981)
• Sets the VIRTUAL_ENV environment variable to path of the venv we create. This is what the source activate script does, and Reticulate also looks for this to discover which python to use (see point 4 of https://rstudio.github.io/reticulate/articles/versions.html#order-of-discovery)
• Sets up PATH appropriately, so python and python3 refer to what is in our venv. This, along with the previous step, ensures same behavior as users typing source ${VIRTUAL_ENV}/bin/activate without actually having to do that, preserving end user behavioral semantics. See https://github.com/rocker-org/rocker-versioned2/issues/670#issuecomment-1601860904
• RStudio is also told about new PATH and VIRTUAL_ENV, using the same pattern as install_texlive.sh
• Remove the explicit symlink of python3 -> python, as venv handles this automatically.
• install_python.sh now needs to be sourced, following same pattern as install_texlive.sh

Decisions to be made:

• Where do we set the appropriate env variables (VIRTUAL_ENV and PATH)? They need to be set for install_python.sh to work correctly. I've set them in the binder image for now, but it should probably be set on a more base image. This is a no-op if install-python.sh is not called anywhere.

TODO:

• [x] Update NEWS (can be done once everything else is finalized)

Ref https://github.com/rocker-org/rocker-versioned2/issues/670

[yuvipanda]

Created this draft mostly to see what CI says!

[yuvipanda]

ok, this is ready for a first round of review :)

[eitsupi]

What is the current status of this?

[yuvipanda]

@cboettig gave me some feedback via slack that I need to dig into. @cboettig can you put them here too?

[cboettig]

Thanks @yuvipanda !

Yup, I think the main thing is considering changing the permissions on /opt/venv from root to something else. (In the R site library we put these in the staff group, allowing any user added to that group to update or add packages to the shared site library. Not sure how nicely pip plays with group-based vs user-based permissions though? In my use/testing of this image I've just been making /opt/venv owned by the default user, https://github.com/boettiger-lab/nasa-topst-env-justice/blob/main/.devcontainer/Dockerfile#L16

A second more minor issue is that under this setup, the default python now points to /opt/venv/bin/python as intended, but if a user calls pip install instead of python -m pip install, they still get /usr/bin/pip instead of /opt/venv/bin/pip, which creates a gotcha that it would be better to avoid if possible??

[yuvipanda]

A second more minor issue is that under this setup, the default python now points to /opt/venv/bin/python as intended, but if a user calls pip install instead of python -m pip install, they still get /usr/bin/pip instead of /opt/venv/bin/pip, which creates a gotcha that it would be better to avoid if possible??

Hmm, since there should be a pip in /opt/venv/bin/pip, if /opt/venv/bin is in $PATH, that should mean pip should call that pip by default, right? Is that not what's happening?

[yuvipanda]

Making /opt/venv be owned by staff seems reasonable to me. If it works for R, my intuition is that it will also work for Python!

[cboettig]

@eitsupi ok just tested and this looks ready to merge to me!

[yuvipanda]

Yay that's awesome, thank you @cboettig!

[eitsupi]

Could you update the NEWS file?

[yuvipanda]

I always seem to struggle with updating NEWS but gave it a shot!

[eitsupi]

Looks good.

The CI failure is notthing to do with changes in this PR, so merging.

[yuvipanda]

Awesome! Thanks a lot, @eitsupi! Will this trigger a rebuild of the images?

[eitsupi]

Sorry, my review was not thorough enough (i.e. this should not have been merged yet), the fix for the stack file is completely incomplete here. We had to update it manually for all versions. See the issue #739.