haiku / infrastructure

Haiku infrastructure as code
https://hub.docker.com/r/haiku
MIT License
17 stars 16 forks source link

[Change Request] Deploy HaikuDepot version 1.0.168 #145

Closed andponlin closed 4 days ago

andponlin commented 1 week ago

Description

Update HaikuDepot to version 1.0.168 (.167 failed)

Main changes this release;

There are no database changes in this release.

How has the change been tested

Steps to implement the change

Note: Please mark changes from the default steps below in bold

  1. Verify that the image is available in the package registry.
  2. Scale down haikudepotserver and start a job to backup the database:
    $ kubectl scale deploy haikudepotserver --replicas=0
    $ kubectl create job --from=cronjob/haikudepotserver-pgbackup haikudepotserver-pgbackup-manual-(VERSION)
  3. Monitor the job to make sure it finishes correctly:
    $ kubectl logs -f jobs/haikudepotserver-pgbackup-manual-(VERSION)
    Backup haikudepotserver...
    gpg: directory '/root/.gnupg' created
    gpg: keybox '/root/.gnupg/pubring.kbx' created
    Added `s3remote` successfully.
    `/tmp/haikudepotserver_2023-08-06.sql.xz.gpg` -> `s3remote/haiku-backups/pg-haikudepotserver/haikudepotserver_2023-08-06.sql.xz.gpg`
    Total: 0 B, Transferred: 245.32 MiB, Speed: 86.05 MiB/s
    Snapshot of haikudepotserver completed successfully! (haiku-backups/pg-haikudepotserver/haikudepotserver_2023-08-06.sql.xz.gpg)
  4. Apply any pre-deployment configuration changes (see section Configuration Changes)
  5. Update the version in the infrastructure repository in deployments/haikudepotserver.yml.
  6. Apply the update to the server:
    $ kubectl apply -f deployments/haikudepotserver.yml
  7. Apply any post-deployment configuration changes (see section Configuration Changes)
  8. Post-deployment checks (is the web service responding, can you refresh the data using the HaikuDepot app)
  9. Commit and push the updated deployment configuration to GitHub.
  10. Announce the update on the haiku-sysadmin and haiku mailing list.

Configuration Changes

Please list any configuration changes, and note whether they need to be done pre-deploy or post-deploy

None

Rollback Plan

If the update is unsuccessful, the rollback can be executed as follows.

If there has been a (failed) database migration as part of the change, the backup created in step 2 must be restored. If the database has not been affected, then skip to the image rollback steps below.

  1. Stop all instances of haikudepotserver
    $ kubectl scale deploy haikudepotserver --replicas=0
  2. Prepare the restore job in deployments/other/restore-pg.yml by making sure the container args point to the haikudepotserver container
            args: ["restore", "haikudepotserver"]
  3. Restore the database by executing the following commands.
    $ kubectl exec -it deployment/postgres -- dropdb -U postgres haikudepotserver
    $ kubectl exec -it deployment/postgres -- createdb -U postgres -O haikudepotserver haikudepotserver
    $ kubectl apply -f deployments/other/restore-pg.yml
    # follow and validate the output of the following command. If the command completes, the restore is done.
    $ kubectl logs --follow jobs/restore
    # cleanup
    $ kubectl delete job/restore
    $ git checkout deployments/other/restore-pg.yml
  4. Roll back the image with the following commands:
    $ git restore deployments/haikudepotserver.yml
    $ kubectl apply -f deployments/haikudepotserver.yml
  5. Do sense checks in the logs and by some exploratory production testing to validate that the restore has been successful.
nielx commented 5 days ago

Scheduled for tomorrow, Sunday 10 November 2024, at 08:00 GMT.