Possible option: Scl python 3.8 on CentOS 7

[jburel]

This PR installs scl python 3.8 No wheel used in that option

cc @joshmoore @sbesson

[jburel]

general consensus is to go with that option.

[sbesson]

Few comments from today's discussion:

• comparing this workflow with #259, immediate impressions is that the SCL solution is the less disruptive
• it might be worth having a reference issue where we can have a table of pros/cons before making the decision
• strong argument towards deciding on a single solution for our docs/roles rather than attempting to support multiple options
• we can certainly consider recommending e.g. SCL for the sysadmin workflows vs Conda for the developer workflows
• @jburel will look into OMERO.web next
• the preferred solution can be investigated via CI as a next step
• some Ansible role upgrades can be pre-tested on internal servers e.g. ome-training-3 and potentially new IDR deployments
• for existing deployments, the problem of the virtual environment upgrade (from Python 3.6 to Python 3.8) will need to be taken into account

[joshmoore]

• for existing deployments, the problem of the virtual environment upgrade (from Python 3.6 to Python 3.8) will need to be taken into account

Perhaps we start naming the environments explicitly? e.g. /opt/...python38-scl

[sbesson]

Perhaps we start naming the environments explicitly? e.g. /opt/...python38-scl

Agreed, we also discussed options briefly but this is a point of investigation. The main downside of creating a new environment is the potential confusion in existing deployments e.g. we already have deployments with both /opt/omero/server/venv (Python 2) and /opt/omero/server/venv3 (Python 3). On the other side, this probably keeps both the documentation and the deployment infrastructure as simple as possible leaving former virtual environments stale.

[jburel]

I think having a clean env will be the most suitable option. Packages in venv3, will unlikely be compatible with the new python version. This could lead to a big mess. Renaming to python38-scl seems to be safest option.

[jburel]

Changing the name implies introducing a new settings.env file and make the infra a bit more complex.

[sbesson]

From the Ansible perspective, in the absence of built-in method to invalidate an existing virtual environment with a different Python version, a new environment feels like the simplest approach to support both new and existing deployments and meet the idempotence requirement.

One consideration though: will the virtual environment name become OS specific? This would have the downside of creating divergence in files like https://github.com/ome/omero-install/blob/a2d7b36cddd8995355a96f3fa6f7472f4cb72392/linux/omero-server-systemd.service#L20

[jburel]

One consideration though: will the virtual environment name become OS specific? This would have the downside of creating divergence in files like

https://github.com/ome/omero-install/blob/a2d7b36cddd8995355a96f3fa6f7472f4cb72392/linux/omero-server-systemd.service#L20

That one too

[jburel]

According to the documentation, using --upgrade should upgrade the virtual env i.e. python3 -m venv --upgrade $VENV_SERVER

[jburel]

Trying to upgrade an existing env will lead to problem due for example to package like

$VENV_SERVER/bin/pip install https://github.com/ome/zeroc-ice-py-centos7/releases/download/0.2.1/zeroc_ice-3.6.5-cp36-cp36m-linux_x86_64.whl

If other dependencies have been installed from a wheel using a specific Python version, it will be the same problem We will need to add information at the doc level.

[sbesson]

Thanks. That's definitely the downside of wheels which are tied to a specific Python version. Off-hand, I do not see a mechanism by which an upgrade flag could be aware of the new wheel to install.

I suspect if we go down the SCL route, we will have to go for a fresh environment either way. Coming back to the path issue, I am leaning towards not changing the virtual environment path.

One thought: is there a command that would allow to programmatically tell whether a virtual environment was built using a given Python version matching an executable path? If so, I could certainly imagine a Python upgrade worfklow similar to:

1. install SCLPython
2. run the Python version compatibility command
3. if incompatible a. archive the former environment (might be as simple as mv venv3 venv3-old). b. optionally try to create a report of the requirements via pip freeze? c. create a new virtual environment under venv3 d. depending on b), try to reinstall the previous requirements (this might still fail for wheels)
4. install the requirements and start the server

For the Ansible upgrade workflow, I could see implementing 2, 3a and 3c as part of the ome.python3_virtualenv role. The requirements freeze/reinstallation (3b, 3d) should be unnecessary in this case as the Python dependencies should be fully captured using roles & variables and a re-execution of the playbook should re-install everything under the new environment.

[sbesson]

I have performed additional investigation on the Ansible workflow. The current logic for the Python environment management is the following:

• ome.python3_virtualenv installs the base Python as well as a shared wrapper script allowing to create a virtual envrionment
• ome.omero_server and ome.omero_web are consuming this wrapper script via the virtualenv_command parameter of the pip module
• the virtualenv_command is only invoked if the virtual environment does not exist. There is no built-in Ansible parameter that solves the issue of an upgrade where a virtual environment exist but must be recreated/upgrade

I also looked at some options of the venv command (https://docs.python.org/3.6/library/venv.html):

• --upgrade has requirements on the Python upgrade which do not apply to the SCL case and would likely not handle the upgrade of wheels as per the discussion above
• --clear is largely the equivalent of rm -rf VENV_DIR && python -mvenv VENV_DIR. It will create a fresh upgrade environment but does not solve the problem of the one-off upgrade

Based on the above, it feels like there is not built-in native solution to achieve what we want. I would also be very conservative about building new tools. A possible approach would be to treat this type of upgrade similarly to e.g. a PostgreSQL version upgrade in the sense that the process requires more than just updating requirements and remotely executing the playbooks. Minimally, the following worfklow should be sufficient:

• stop the server/web services via service stop
• delete/rename the existing /opt/ome/{server.,web}/venv3 environments
• update the roles to install SCL Python and the associated wheels
• run the Ansible playbooks

[jburel]

Closing this PR A branch was created so it can be used in the current devspace