Verify upgrade procedure and document the exact steps for the upgrade to python 3 for all environments

[datadavev]

Need to prepare detailed steps for the process of upgrading an environment to use the python 3 implementation

[datadavev]

Procedure for upgrading the production python 2 version of EZID to python 3.

See this spreadsheet for the production upgrade list: https://docs.google.com/spreadsheets/d/1HDVxR_0aaqDGi_oNstUzelNIsXS_qJw1GJRRgZkchJY/edit?usp=sharing

The upgrade procedure involves significant database migration that is not reversible, except to restore the database as it existed prior to migration. Hence a complete copy of the application and database must be made before migration to ensure a viable recovery path.

Pre-Upgrade (prepare for upgrade)

• [ ] Ensure application is operating as expected.
• [ ] Notify stakeholders of pending upgrade
• [ ] Review upgrade procedure, clarifying who is responsible for each step
• [ ] Create a new EC2 instance to host the python3 version of the application
• [ ] Install the new code base on the new EC2 instance

Upgrade (upgrade database and application)

Herein: Old host = production 01, new host = production 02

• Name of existing (old) production server instance: uc3-ezidui01x2-prd
• Name of existing (old) database instance: rds-ias-ezid-search4-prd
• Name of new production server instance: uc3-ezidui02x2-prd
• Name of new production database instance rds-uc3-ezid5-prd

Zoom Call start

• [ ] (Ashley) Disable nagios alerts for old production uc3-ezidui01x2-prd
• [ ] (Ashley) Disable puppet agent for both new and old production instances
• [ ] (Ashley) Shutdown link checker. Disable in puppet to ensure it does not restart.
• [ ] (Ashley) Shutdown EZID application (apache) and any additional tasks (e.g. cron)
• [ ] (Ashley) Display banner page: run python3 downserver.py localhost 18880 "EZID is intentionally offline for maintenance, check status at <a href=''>EZID Status page</a>."
• [ ] (IAS - Ashley will arrange) Duplicate the production database. Upgrade actions will be applied to the new copy of the database. (approx. 30minutes)
• [ ] (Roger) Copy minter databases from old uc3-ezidui01x2-prd to new uc3-ezidui02x2-prd instance
• [ ] (Ashley) Verify configuration setting for database access, admin account Zoom call ends arounds here
• [ ] (Roger) Perform database migration using the new server uc3-ezidui02x2-prd (total time ~15hrs)
  • [ ] Install additional packages needed only for the migration

    $ pip install mysql.connector

  • [ ] Temporarily set DB user to eziddba with appropriate pw for login in settings.py
  • [ ] Run migration through Tmux shell on stg-py3.
  • [ ] Set environment variables for database access. Manual step to set env vars directly (Roger)
  • [ ] Run the script migrate_db_to_py3/run-migration.py | tee migrate_log.txt This process runs several tasks against the database and will take several hours in total.
  • [ ] (Roger) Set DB user to ezidrw with appropriate pw in settings.py
  • [ ] (Roger) Recreate the admin account manage.py diag-update-admin

Start Zoom call

• [ ] (Ashley) Validate ansible and puppet (verify ezidrw account for database)
• [ ] (Ashley) Startup EZID
• [ ] Verify basic operations by tunneling to the new host on 18880:
  • [ ] (Roger, Rushi, Dave) Verify anonymous ARK mint
  • [ ] (Roger, Rushi, Dave) Verify admin login
  • [ ] (Roger, Rushi, Dave) Verify admin ARK mint
  • [ ] (Roger, Rushi, Dave) Verify admin view groups, correct number of entries
• [ ] (Ashley) Remove existing prod host uc3-ezidui01x2-prd from load balancer configuration
• [ ] (Ashley) Ensure services on uc3-ezidui01x2-prd are disabled.
• [ ] (Ashley) Add the new prod host instance uc3-ezidui02x2-prd to the load balancer
• [ ] (Ashley) Verify load balancer configuration is pointing to the correct host
• [ ] (Ashley) Start background jobs (including link_checker)
  • [ ] add list of background procs here
• [ ] (Roger) Verify DataCite DOI minting, search
• [ ] (Rushi) Verify Crossref DOI minting, search
• [ ] (Rushi) Verify ARK minting, search on EZID, test resolve on N2T
• [ ] (Everyone) Comprehensive automated and manual tests (note that some searches may be slow at this point)

Post-Upgrade part 2 (verify operation)

• [ ] (Ashley) Request Nagios monitoring of background tasks
• [ ] (Maria) Notify general availability at end of window (17:00 PT)
• [ ] Continue testing basic search, login, minting with the application to verify expected functionality

Post-verification

• [ ] Disable and archive or remove the old instance of EZID. Ensure any history (logs, notes, etc) is preserved if necessary.

Rollback Procedure

If migration fails during downtime (no changes to production data):

• [ ] Re-enable ezid service in puppet for host uc3-eziui01x2-prd
• [ ] Start ezid service on uc3-eziui01x2-prd
• [ ] Re-enable ezid-linkcheck service in puppet for host uc3-ezidlinkcheck01x2-prd
• [ ] Start ezid-linkcheck service on uc3-ezidlinkcheck01x2-prd
• [ ] Register host uc3-eziui01x2-prd to ALB TargetGroup uc3-ezidui-prd-tg
• [ ] Deregister host uc3-ezidui02x2-prd from ALB TargetGroup uc3-ezidui-prd-tg
• [ ] Shutdown and disable service ezid on host uc3-ezidui02x2-prd
• [ ] Shutdown and disable all ezid-proc-* services on host uc3-ezidui02x2-prd