[Change Request] Deploy HaikuDepot version 1.0.156 - Githubissues

haiku / infrastructure

Haiku infrastructure as code

https://hub.docker.com/r/haiku

MIT License

17 stars 16 forks source link

[Change Request] Deploy HaikuDepot version 1.0.156 #115

Closed andponlin closed 8 months ago

andponlin commented 8 months ago

Description

Update HaikuDepot to version 1.0.156

This change has dependency bumps from dependabot and also a significant reworking of the "natural language" infrastructure in the system as part of a multi-step set of changes to supporting integration with Polygot.

Note that HDS 1.0.156 does carry a database migration (1.6) without rollback possibility so it will not be safe to simply rollback the deployment image in the unlikely event of a deployment issue. Please keep a database backup on hand as a contingency.

How has the change been tested

Dev-tested by the HaikuDepot dev.
Automated tests run by the Github action: here

Steps to implement the change

Note: Please mark changes from the default steps below in bold

Verify that the image is available in the package registry.

Start a job to backup the database:

$ kubectl create job --from=cronjob/haikudepotserver-pgbackup haikudepotserver-pgbackup-manual-(VERSION)

Monitor the job to make sure it finishes correctly:

$ kubectl logs -f jobs/haikudepotserver-pgbackup-manual-(VERSION)
Backup haikudepotserver...
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
Added `s3remote` successfully.
`/tmp/haikudepotserver_2023-08-06.sql.xz.gpg` -> `s3remote/haiku-backups/pg-haikudepotserver/haikudepotserver_2023-08-06.sql.xz.gpg`
Total: 0 B, Transferred: 245.32 MiB, Speed: 86.05 MiB/s
Snapshot of haikudepotserver completed successfully! (haiku-backups/pg-haikudepotserver/haikudepotserver_2023-08-06.sql.xz.gpg)

Apply any pre-deployment configuration changes (see section Configuration Changes)
Update the version in the infrastructure repository in deployments/haikudepotserver.yml.

Apply the update to the server:

$ kubectl apply -f deployments/haikudepotserver.yml

Apply any post-deployment configuration changes (see section Configuration Changes)
Post-deployment checks (is the web service responding, can you refresh the data using the HaikuDepot app)
Commit and push the updated deployment configuration to GitHub.
Announce the update on the haiku-sysadmin and haiku mailing list.

Configuration Changes

Please list any configuration changes, and note whether they need to be done pre-deploy or post-deploy

None

Rollback Plan

If the update is unsuccessful, try rolling back the image with the following commands:

NOTE This version will also require a database rollback if there are issues with the service coming up post-deploy.

$ git restore deployments/haikudepotserver.yml
$ kubectl apply -f deployments/haikudepotserver.yml

nielx commented 8 months ago

Starting this now.

nielx commented 8 months ago

There seems to be an issue with the database migration, this has been stuck for 7 minutes now:

10:02:00.620 [main] INFO  o.f.core.internal.command.DbValidate - Successfully validated 7 migrations (execution time 00:00.178s)
10:02:00.761 [main] INFO  o.f.core.internal.command.DbMigrate - Current version of schema "public": 1.4
10:02:00.825 [main] INFO  o.f.core.internal.command.DbMigrate - Migrating schema "public" to version "1.5 - add category fonts"
10:02:01.033 [main] INFO  o.f.core.internal.command.DbMigrate - Migrating schema "public" to version "1.6 - natural languages enhancements"

nielx commented 8 months ago

I have tried to restart the migration by scaling down, and scaling up. The output right now is:


  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.2.3)

10:12:33.121 [main] INFO  o.haiku.haikudepotserver.Application - Starting Application using Java 21.0.2 with PID 8 (/opt/haikudepotserver/app.jar started by root in /)
10:12:33.125 [main] INFO  o.haiku.haikudepotserver.Application - No active profile set, falling back to 1 default profile: "default"
10:12:34.958 [main] INFO  o.s.c.i.s.PropertySourceProcessor - Properties location [classpath:local.properties] not resolvable: class path resource [local.properties] cannot be opened because it does not exist
10:12:39.537 [main] INFO  o.s.d.r.c.RepositoryConfigurationDelegate - Bootstrapping Spring Data JPA repositories in DEFAULT mode.
10:12:39.824 [main] INFO  o.s.d.r.c.RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 217 ms. Found 0 JPA repository interfaces.
10:12:44.204 [main] INFO  o.s.b.w.e.tomcat.TomcatWebServer - Tomcat initialized with port 8080 (http)
10:12:44.228 [main] INFO  o.a.catalina.core.StandardService - Starting service [Tomcat]
10:12:44.229 [main] INFO  o.a.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/10.1.19]
10:12:44.856 [main] INFO  o.a.c.c.C.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
10:12:44.864 [main] INFO  o.s.b.w.s.c.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 11530 ms
10:12:45.316 [main] INFO  o.h.h.s.d.DesktopApplicationMinimumVersionFilter - desktop application min version; 0.0.2
10:12:46.613 [main] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
10:12:47.604 [main] INFO  com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@4d525897
10:12:47.609 [main] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed.
10:12:48.027 [main] INFO  o.f.c.i.license.VersionPrinter - Flyway Community Edition 9.22.3 by Redgate
10:12:48.027 [main] INFO  o.f.c.i.license.VersionPrinter - See release notes here: https://rd.gt/416ObMi
10:12:48.027 [main] INFO  o.f.c.i.license.VersionPrinter -
10:12:48.086 [main] INFO  org.flywaydb.core.FlywayExecutor - Database: jdbc:postgresql://postgres:5432/haikudepotserver (PostgreSQL 14.2)
10:12:48.348 [main] INFO  o.f.core.internal.command.DbValidate - Successfully validated 7 migrations (execution time 00:00.132s)
10:12:48.425 [main] INFO  o.f.core.internal.command.DbMigrate - Current version of schema "public": 1.5
10:12:48.507 [main] INFO  o.f.core.internal.command.DbMigrate - Migrating schema "public" to version "1.6 - natural languages enhancements"

nielx commented 8 months ago

From the postgresql log:

2024-03-10 10:12:33.096 UTC [635631] LOG:  unexpected EOF on client connection with an open transaction
2024-03-10 10:13:04.049 UTC [635845] LOG:  unexpected EOF on client connection with an open transaction

This may be my attempt to manually kill the migration after the first run though

nielx commented 8 months ago

The migration really seems to be stuck, unless there are long running operations, but there is no logging. Going to revert.

nielx commented 8 months ago

Steps to restore database/revert:

Revert the deployments/haikudepotserver.yml to its previous version
Modify the number of replicas to 0 and apply the file (kubectl apply -f haikudepotserver.yml)
Drop the database kubectl exec -it deployment/postgres -- dropdb -U postgres haikudepotse rver
Recreate the database kubectl exec -it deployment/postgres -- createdb -U postgres -O haikudepotserver haikudepotserver
prepare the database job in deployments/other/restore-pg.yml to point to haikudepotserver database.
Run the job kubectl apply -f restore-pg.yml
Follow the job until it is completed kubectl logs --follow jobs/restore
Delete the job after completion kubectl delete job/restore
Restore the original deployments/haikudepotserver.yml.
Apply

nielx commented 8 months ago

Closing this ticket As part of #116 I will provide a masked database, so maybe that helps reconstruct what went wrong.

Please open a new request once we can reapply (with maybe manual steps) or when there is a new version.