[Production ssh access]: recreating missing database tables in production

[tadhg-ohiggins]

Start (UTC)

2024-06-06 20:27

End (UTC)

2024-06-06 20:49

Explanation

Deployment to production was failing due to Django migration errors.

Similar errors previously occurred in other environments, including staging. The fix in all cases is to delete some entries from the database table Django uses to manage migrations and then re-create the tables.

This requires ssh access both for shell access to production and to enable ssh tunnel access to the production database.

cf allow-space-ssh production
echo $(date -u +"%Y-%m-%d %H:%M") UTC
# 2024-06-06 20:27 UTC
cf connect-to-service -no-client gsa-fac fac-db
# three migrations deleted via DBeaver from django_migrations table

Via this ssh tunnel I deleted three entries from the django_migrations table, with help from @sambodeme and @danswick.

The next step was to run the migrations:

cf ssh gsa-fac
/tmp/lifecycle/shell
source tools/setup_env.sh
setup_env
set +e
python manage.py migrate census_historical_migration 0002_reportmigrationstatus_migrationerrordetail
python manage.py migrate census_historical_migration 0003_remove_migrationerrordetail_error_stack_and_more
python manage.py migrate census_historical_migration 0004_elecaudits_multipleueis_elecaudits_uei
# django.db.utils.ProgrammingError: column "MULTIPLEUEIS" of relation "census_historical_migration_elecaudits" already exists

0004_elecaudits_multipleueis_elecaudits_uei failed, so we needed to create that in the database directly. I exited the ssh session and created an ssh tunnel again:

cf connect-to-service -no-client gsa-fac fac-db
# Manually added 0004_elecaudits_multipleueis_elecaudits_uei as an entry in django_migrations

Following that, we re-ran the deployment and it succeeded.

cf disallow-space-ssh production
echo $(date -u +"%Y-%m-%d %H:%M") UTC
# 2024-06-06 20:49 UTC