Upgrade Python to 3.10 in app VM and psycopg2 to 2.9

Overview

Upgrades the app VM from Python 3.6.9 to 3.10.12 as well as upgrades psycopg2 from 2.8.6 to 2.9 in both app and database VM. Both of these upgrades support upcoming Django and PostgreSQL upgrades in the #1372 task list.

Ideally we would get to Python version 3.12 since that is currently in bugfix maintenance status into 2025, but I ran into a variety of issues trying to upgrade to this version so am leaving it at 3.10.12. This version is still receiving security support through October 2026. (see notes for more detail)

Demo

RUNNING HANDLER [cac-tripplanner.app : Restart nginx] **************************
changed: [app]

PLAY RECAP *********************************************************************
app                        : ok=37   changed=32   unreachable=0    failed=0    skipped=17   rescued=0    ignored=0   

rachelemorino@Racheles-MacBook-Pro cac-tripplanner % vagrant ssh app
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-92-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

  System information as of Fri Mar  1 07:04:49 PM UTC 2024

  System load:  0.39599609375      Processes:             159
  Usage of /:   19.1% of 30.34GB   Users logged in:       0
  Memory usage: 54%                IPv4 address for eth0: 10.0.2.15
  Swap usage:   1%                 IPv4 address for eth1: 192.168.56.24

This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento
Last login: Fri Mar  1 19:03:31 2024 from 192.168.56.1
vagrant@app:~$ python3 --version
Python 3.10.12

vagrant@app:~$ pip show psycopg2-binary
Name: psycopg2-binary
Version: 2.9
Summary: psycopg2 - Python-PostgreSQL Database Adapter
Home-page: https://psycopg.org/
Author: Federico Di Gregorio
Author-email: fog@initd.org
License: LGPL with exceptions
Location: /usr/local/lib/python3.8/dist-packages
Requires:

vagrant@database:~$ pip show psycopg2
Name: psycopg2
Version: 2.9.9
Summary: psycopg2 - Python-PostgreSQL Database Adapter
Home-page: https://psycopg.org/
Author: Federico Di Gregorio
Author-email: fog@initd.org
License: LGPL with exceptions
Location: /usr/local/lib/python3.8/dist-packages
Requires: 
Required-by:

Notes

Context on the nginx role changes:

Upgrading to Python 3.10 failed app provisioning with the following error:

TASK [azavea.nginx : Configure the Nginx PPA] **********************************
fatal: [app]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: E:The repository 'http://ppa.launchpad.net/nginx/stable/ubuntu jammy Release' does not have a Release file."}

After digging it appears upgrading the upgraded Ubuntu version was too modern for the repo defined in the azavea.nginx role. Unfortunately, this is a separate repo so I had to copy over the entire role and update app's role dependencies in order to update this task to use the correct stable PPA.

Notes on issues upgrading to Python 3.12:

The Ubuntu 22.04 base box brings the Python default to 3.10.12, but ideally we would upgrade to Python 3.12 to give us the most runway before contract end. I tried to manually install Python 3.12 from PPA as part of app provisioning and then update the ansible_python_interpreter to locate the correct version to use, but ran into incompatibilities with the ansible version:

TASK [azavea.packer : Download Packer] *****************************************
fatal: [app]: FAILED! => {"changed": false, "dest": "/usr/local/src/packer_1.5.4_linux_amd64.zip", "elapsed": 0, "msg": "An unknown error occurred: 'CustomHTTPSConnection' object has no attribute 'cert_file'", "url": "https://releases.hashicorp.com/packer/1.5.4/packer_1.5.4_linux_amd64.zip"}

The upgraded Python version got rid of some variables, like cert_file, that are used by ansible in the azavea.packer role and we would need to upgrade local ansible-core to version >=2.15 for these to be compatible, see this ansible issue for more detail. This project depends on local installation of ansible, but we have historically had issues using upgraded ansible versions beyond 6.0 with this project. The latest version of ansible v 5.0 only includes up to ansible-core v 2.12.7. I could target an update for ansible-core to get these fixes, but I suspect this could open a can of worms and given our remaining budget and that we're able to at least get to Python 3.10 I think we should leave this for down the road. Curious if you think it's worth it!

Testing Instructions

vagrant destroy (Ensure you have Graph.obj in the otp_data dir before destroying the otp VM so it doesn't rebuild the graph on up)
vagrant up (you may need to run CAC_APP_SHARED_FOLDER_TYPE=virtualbox vagrant up)
Confirm all VMs are successfully brought up and app is working as expected in the browser
ssh into the app VM and confirm python3 version and psycopg2-binary version
ssh into the database VM and confirm psycopg2 version

Checklist

[ ] No gulp lint warnings
[ ] No python lint warnings
[ ] Python tests pass
[ ] All TODOs have an accompanying issue link

Connects #1356

azavea / cac-tripplanner