timescale / helm-charts

Configuration and Documentation to run TimescaleDB in your Kubernetes cluster
Apache License 2.0
263 stars 223 forks source link

Maximum version of postgres is still 14 even though the version 15 container is now available #594

Open Dahs81 opened 1 year ago

Dahs81 commented 1 year ago

What is missing? Max version in the values.schema.json is version 14.

 {
   "version": {
      "maximum": 14,
      "minimum": 11,
      "type": [
        "integer",
        "null"
      ]
 }

This needs to be updated to 15 in order to run helm against it.

Why do we need it? We've manually updated our postgres to version 15, but can't actually run helm without it reverting to version 14.

Anything else we need to know?: To upgrade to 15, we used pg15.2-ts2.9.3-latest image and then set the version field to 14. We then manually updated from 14 to 15 by exec'ing into the container and moving the data dir and running pg_upgrade. But we need this field fixed before we can run helm with version: 15.

I can potentially do a PR for this, but wanted to check and make sure that it wasn't already in the works.

CoyoteLeo commented 1 year ago

@Dahs81 Hi, can you share the steps for upgrading the Postgres major version from 14 to 15?

theblinkingusb commented 1 year ago

hey @CoyoteLeo - if helpful this is what I did - YMMV - I really struggled to follow the migration path that @Dahs81 took - I tried a few different iterations with docker locally and in a nonprod k8s but couldn't reconcile the pg_upgrade command requiring both the old data dir and the new data dir (using pg15.4-ts2.11.2-all image, which has the other pg versions) - maybe someone else has some advice here - tried different methods of copying/moving data dir around, but ultimately couldn't get it.

I ended up deploying a second helm chart with the pg15.4-ts2.11.2-oss image to perform a side-by-side upgrade. I think this is the approach I want in the long run anyways for upgrades, just to reduce downtime risk and upgrade risk. For the new helm chart I adjusted the s3 backup location and storage locations. Then moved all the dbs over with pg_dumpall. PG Docs, TSDB Docs (see bottom of article) pg_dumpall -f <filepath>.sql --clean --if-exists --username=<superuser>--password Restored with psql -f <filepath>.sql postgres on the new container This part worked really well - throwing the documented errors as mentioned in the docs. Querying on the new side had no issues.

After that I switched the services out and verified the downstream services/apps were good to go. I could have gone into the individual downstream apps and adjusted their conn strings, but since I have a few third party services (with not all of them using a straightforward configmap pattern), I found that more tedious than swapping the service definitions. I also want to maintain the ability to side-by-side in the future in case an upgrade goes sideways. The big downside of changing the services is that the helm chart won't map these - I deploy with Ansible currently so I added an extra kubectl step to ensure the services were pointed in the right place. Possibly another option here would be to adjust a fork of this chart. After all that was situated and everything was stable, I uninstalled the previous chart (and kept the data dir for now).

Other background - the previous version I was migrating from was pg14.6-ts2.9.1-p1 specified in the 0.33.1 chart. Wanted to migrate to 15 for merge and other features. I'm still learning timescaledb/postgres and my footprint is small - it took about 20mins of dump and restore for ~20gb on ~10dbs with nvme storage. As the environment grows and for pg16 I'll likely ship a service/db at a time and/or try some incremental backups/restores to the new instance to minimize downtime. Overall I'm grateful for how seamless it was - I suppose that's a testament to the tsdb/postgres communities and strong docs - thanks to all there.

CoyoteLeo commented 1 year ago

@theblinkingusb Soooooo Great! I really appreciate and admire your sharing!