Closed jburel closed 1 year ago
general consensus is to go with that option.
Few comments from today's discussion:
ome-training-3
and potentially new IDR deployments
- for existing deployments, the problem of the virtual environment upgrade (from Python 3.6 to Python 3.8) will need to be taken into account
Perhaps we start naming the environments explicitly? e.g. /opt/...python38-scl
Perhaps we start naming the environments explicitly? e.g. /opt/...python38-scl
Agreed, we also discussed options briefly but this is a point of investigation. The main downside of creating a new environment is the potential confusion in existing deployments e.g. we already have deployments with both /opt/omero/server/venv
(Python 2) and /opt/omero/server/venv3
(Python 3). On the other side, this probably keeps both the documentation and the deployment infrastructure as simple as possible leaving former virtual environments stale.
I think having a clean env will be the most suitable option. Packages in venv3, will unlikely be compatible with the new python version. This could lead to a big mess. Renaming to python38-scl
seems to be safest option.
Changing the name implies introducing a new settings.env
file and make the infra a bit more complex.
From the Ansible perspective, in the absence of built-in method to invalidate an existing virtual environment with a different Python version, a new environment feels like the simplest approach to support both new and existing deployments and meet the idempotence requirement.
One consideration though: will the virtual environment name become OS specific? This would have the downside of creating divergence in files like https://github.com/ome/omero-install/blob/a2d7b36cddd8995355a96f3fa6f7472f4cb72392/linux/omero-server-systemd.service#L20
One consideration though: will the virtual environment name become OS specific? This would have the downside of creating divergence in files like
That one too
According to the documentation, using --upgrade
should upgrade the virtual env
i.e. python3 -m venv --upgrade $VENV_SERVER
Trying to upgrade an existing env will lead to problem due for example to package like
$VENV_SERVER/bin/pip install https://github.com/ome/zeroc-ice-py-centos7/releases/download/0.2.1/zeroc_ice-3.6.5-cp36-cp36m-linux_x86_64.whl
If other dependencies have been installed from a wheel using a specific Python version, it will be the same problem We will need to add information at the doc level.
Thanks. That's definitely the downside of wheels which are tied to a specific Python version. Off-hand, I do not see a mechanism by which an upgrade flag could be aware of the new wheel to install.
I suspect if we go down the SCL route, we will have to go for a fresh environment either way. Coming back to the path issue, I am leaning towards not changing the virtual environment path.
One thought: is there a command that would allow to programmatically tell whether a virtual environment was built using a given Python version matching an executable path? If so, I could certainly imagine a Python upgrade worfklow similar to:
mv venv3 venv3-old
).
b. optionally try to create a report of the requirements via pip freeze
?
c. create a new virtual environment under venv3
d. depending on b), try to reinstall the previous requirements (this might still fail for wheels)For the Ansible upgrade workflow, I could see implementing 2, 3a and 3c as part of the ome.python3_virtualenv role. The requirements freeze/reinstallation (3b, 3d) should be unnecessary in this case as the Python dependencies should be fully captured using roles & variables and a re-execution of the playbook should re-install everything under the new environment.
I have performed additional investigation on the Ansible workflow. The current logic for the Python environment management is the following:
ome.python3_virtualenv
installs the base Python as well as a shared wrapper script allowing to create a virtual envrionmentome.omero_server
and ome.omero_web
are consuming this wrapper script via the virtualenv_command
parameter of the pip modulevirtualenv_command
is only invoked if the virtual environment does not exist. There is no built-in Ansible parameter that solves the issue of an upgrade where a virtual environment exist but must be recreated/upgradeI also looked at some options of the venv
command (https://docs.python.org/3.6/library/venv.html):
--upgrade
has requirements on the Python upgrade which do not apply to the SCL case and would likely not handle the upgrade of wheels as per the discussion above--clear
is largely the equivalent of rm -rf VENV_DIR && python -mvenv VENV_DIR
. It will create a fresh upgrade environment but does not solve the problem of the one-off upgradeBased on the above, it feels like there is not built-in native solution to achieve what we want. I would also be very conservative about building new tools. A possible approach would be to treat this type of upgrade similarly to e.g. a PostgreSQL version upgrade in the sense that the process requires more than just updating requirements and remotely executing the playbooks. Minimally, the following worfklow should be sufficient:
service stop
/opt/ome/{server.,web}/venv3
environmentsClosing this PR A branch was created so it can be used in the current devspace
This PR installs scl python 3.8 No wheel used in that option
cc @joshmoore @sbesson