search

CrunchyData / postgres-operator

Production PostgreSQL for Kubernetes, from high availability Postgres clusters to full-scale database-as-a-service.
https://access.crunchydata.com/documentation/postgres-operator/v5/
Apache License 2.0
3.92k stars 592 forks source link

How to do a pg-major-upgrade with operator v5 #3144

Closed EugenMayer closed 2 years ago

EugenMayer commented 2 years ago

We are aware of

but this did not help up understand how it is expected to be actually done. What we could grasp is

"Somewhere (some deployment or pod) needs to get an ENV variable called CCP_IMAGE_TAG to the image of the next major, e.g when we are on registry.developers.crunchydata.com/crunchydata/crunchy-postgres:centos8-12.9-0 right now, CCP_IMAGE_TAG must point to registry.developers.crunchydata.com/crunchydata/crunchy-postgres:centos8-13.5-0"

Yet we are not sure how to do that. We are using the kustomize way of deploying the operator via https://github.com/CrunchyData/postgres-operator-examples/tree/main/kustomize/install and using. We are currently using v5.0.4 and pg12.

I understand the question has been asked several times, but either it is us or maybe the answer was yet not sufficient. Thank you a lot for a more detailed answer.

Alternatively, we have seen that v5.1 will have an auto-upgrade mode with https://github.com/CrunchyData/postgres-operator/pull/2853 - is it expected to wait for that? If yes, es there a broad timeline when 5.1 will be available?

EugenMayer commented 2 years ago

Any chance to get a hint or glimps on the procedure here (or the plans with those in 5.1)? Thank you!

EugenMayer commented 2 years ago

Looking at https://github.com/CrunchyData/postgres-operator/blob/master/docs/content/releases/5.1.0.md it seems like the upgrades are not included in 5.1 as maybe planed before.

Leaves us with probably the questions

  1. how to utilize crunchy-upgrade with a k8s operator v5 deployment for an upgrade?
  2. Should we rather use a backup / restore way of upgrading. So create a usual backup with pgbackuprest and then restore on a new cluster running the new versions? Does this even work with a wal based backup?

For all of you looking for some possible context for https://access.crunchydata.com/documentation/crunchy-postgres-containers/latest/examples/administration/upgrade/ - if found https://github.com/CrunchyData/crunchy-containers/tree/master/examples/kube/upgrade - especially https://github.com/CrunchyData/crunchy-containers/blob/master/examples/kube/upgrade/upgrade.json at least gives an idea how to utilizeCCP_IMAGE_TAG but well, leaves a lot open for guessing.

I do not understand why we are not getting a short answer on this - i mean everything would be acceptable, pick one of these if you like - just to make it easier / shorter for the crunchy-data operators to pick from

  1. we are no longer planning to support pg major upgrades with pgo - you have to care about that yourself
  2. we no longer plan to offer convinient / automatic upgrading of pg major versions - but you can go manual using this DOCS
  3. we plan to only offer the convinient / automatic upgrading of pg major versions for paying customers - anybody else is either in 1. or 2.
  4. We are going to provide pg major upgrades, but we will not make it with 5.1
  5. We will release it with 5.1, but we are just not yet done with it
  6. You can already do it in 5.0 using DOCS

No matter which answer it is, it's fine. Just let us know :)

arlllk commented 2 years ago

I found this https://www.crunchydata.com/blog/easy-major-postgresql-upgrades-using-pgo-v51 I have tried and it works well Adding this to the docs would be especially useful Magnificent work on the project

timbrd commented 2 years ago

I found this https://www.crunchydata.com/blog/easy-major-postgresql-upgrades-using-pgo-v51 I have tried and it works well Adding this to the docs would be especially useful Magnificent work on the project

I just installed the certified postgres operator (v5.1.0) on my Openshift cluster, but there is not pgupgrade crd:

$ oc get crd | grep crunchy
postgresclusters.postgres-operator.crunchydata.com                2022-05-02T10:53:36Z

$ oc create -f - <<EOF
apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PGUpgrade
metadata:
  name: hippo-upgrade
spec:
  postgresClusterName: hippo
  fromPostgresVersion: 13
  toPostgresVersion: 14
  image: registry.developers.crunchydata.com/crunchydata/crunchy-upgrade:ubi8-5.1.0-0
EOF
error: unable to recognize "STDIN": no matches for kind "PGUpgrade" in version "postgres-operator.crunchydata.com/v1beta1"

How did you install the operator?

timbrd commented 2 years ago

The pgupgrade crd is also missing in the source code: https://github.com/CrunchyData/postgres-operator/tree/master/config/crd/bases

I guess the feature has been postponed to the next release.

andrewlecuyer commented 2 years ago

Major PG Upgrade functionality is now available in PGO v5.1. Details can be found in the following blog post:

https://www.crunchydata.com/blog/easy-major-postgresql-upgrades-using-pgo-v51

wombat commented 1 year ago

I found this https://www.crunchydata.com/blog/easy-major-postgresql-upgrades-using-pgo-v51 I have tried and it works well Adding this to the docs would be especially useful Magnificent work on the project

I just installed the certified postgres operator (v5.1.0) on my Openshift cluster, but there is not pgupgrade crd:

$ oc get crd | grep crunchy
postgresclusters.postgres-operator.crunchydata.com                2022-05-02T10:53:36Z

$ oc create -f - <<EOF
apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PGUpgrade
metadata:
  name: hippo-upgrade
spec:
  postgresClusterName: hippo
  fromPostgresVersion: 13
  toPostgresVersion: 14
  image: registry.developers.crunchydata.com/crunchydata/crunchy-upgrade:ubi8-5.1.0-0
EOF
error: unable to recognize "STDIN": no matches for kind "PGUpgrade" in version "postgres-operator.crunchydata.com/v1beta1"

How did you install the operator?

I found this https://www.crunchydata.com/blog/easy-major-postgresql-upgrades-using-pgo-v51 I have tried and it works well Adding this to the docs would be especially useful Magnificent work on the project

I just installed the certified postgres operator (v5.1.0) on my Openshift cluster, but there is not pgupgrade crd:

$ oc get crd | grep crunchy
postgresclusters.postgres-operator.crunchydata.com                2022-05-02T10:53:36Z

$ oc create -f - <<EOF
apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PGUpgrade
metadata:
  name: hippo-upgrade
spec:
  postgresClusterName: hippo
  fromPostgresVersion: 13
  toPostgresVersion: 14
  image: registry.developers.crunchydata.com/crunchydata/crunchy-upgrade:ubi8-5.1.0-0
EOF
error: unable to recognize "STDIN": no matches for kind "PGUpgrade" in version "postgres-operator.crunchydata.com/v1beta1"

How did you install the operator?

I also installed the Certified PGO 5.3.0 on OpenShift 4.10 and I cannot find a PGUpgrade CRD - @andrewlecuyer I also couldn´t find the CRD listed in the OperatorHub

zerkms commented 1 year ago

@wombat https://github.com/CrunchyData/postgres-operator-examples/blob/main/helm/install/crds/postgres-operator.crunchydata.com_pgupgrades.yaml

EugenMayer commented 1 year ago

@zerkms i deployed the install/default 5.3.0 variant and installed it based on my old pg12 image version. I updaed pgbackuprest and bouncer to the latest versions and everything seemed to work with.

I verified that i had the pgo-operator and pgo-operator-upgrade on 5.3.0.

When i now switch the following

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: pg-cluster
spec:
  image: kwcr.io/infra/crunchydata/crunchy-postgres:centos8-12.9-0
  postgresVersion: 12

to 

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: pg-cluster
spec:
  image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.6-2
  postgresVersion: 14

and deploy the kustomization, i see that one replica is removed and a new one, based on 14.6.2, is started. This one never compleats because it waits for the cluster leader (the master is still on 12 at this point).

I had to revert to 12 for now.

What is the expected behavior in this case. I used https://access.crunchydata.com/documentation/postgres-operator/5.3.0/tutorial/update-cluster/ as the reference docs, but there is nothing about postgresVersion any longer.

I see there is https://www.crunchydata.com/blog/easy-major-postgresql-upgrades-using-pgo-v51 which reads differently and seems to spawn an additional cluster using the old version. Then this one is migrated to the new version using an task.

What is the current / right way to do that?

Any help would be awesome.

EugenMayer commented 1 year ago

Sorry for the confusion above - if useful for anybody (and not obvious as for me)

  1. https://access.crunchydata.com/documentation/postgres-operator/5.3.0/tutorial/update-cluster/ is only for minor updates within a series, so 14.0 to 14.1 or whatever
  2. https://www.crunchydata.com/blog/easy-major-postgresql-upgrades-using-pgo-v51 / https://access.crunchydata.com/documentation/postgres-operator/5.3.0/guides/major-postgres-version-upgrade/

Hint: if you deploy your cluster into a namespace, you will need to add -n <namespace> to all the operations done in the blog. Also when creating the task ,you will need to create the task in the namespace the cluster runs in.