upgrade postgres

[ywkim312]

Postgres is old. It has to be upgraded to newer version

[ywkim312]

After updating the chart, then deployed. The log shows the incompatibility

2024-06-24 21:13:49.167 GMT [1] FATAL:  database files are incompatible with server
2024-06-24 21:13:49.167 GMT [1] DETAIL:  The data directory was initialized by PostgreSQL version 11, which is not compatible with this version 16.3.```

[ywkim312]

Here's the general process

One very important thing is that to make datawolf to work, the database must be backup using the higher version of pg_dump For example, in this case, the current postgres is 11.14.0 and is being updated to 16.3.0. Then, when the database get backed up, the pg_dump must be the version 16.3. So the version 16.3 version of pg_dump dumps that databases in the 11.14. So, the following step 1, the pg_dump must be 16.3 otherwise datawolf will not work after get restored.

Other thing consider is that, after database get updated, when there is no actual data in the database, if the datawolf pod is alive, it will generated the necessary tables in the database. So to avoid this, after do all the following steps, scale datawolf and postgres down to zero. Then remove the PVC, and PV. Then scale up the postgres to necessary number. Then restore the database. Then scale up the datawolf.

1. back up all the databases (currently, keycloak, datawolf, maestro_galveston, maestro_joplin, maestro_slc)
2. when back up, use both pgAdmin and Script, just in case
3. check updated helm chart (values.yaml, values-incore-*.yaml, passwords inside it for postgres, and other users)
4. helm dep up (check if the tar file get upgraded)
5. make zero scale for statefulsets (datawolf, keycloak)
6. delete postgres statefulset
7. delete PVC for postgres (PVC first)
8. delete PV for postgres
9. deploy the upgraded chart
10. check postgres deployment (password, check initdb was successfully performed)
11. restore backed up databases to new postgres
12. scale up for datawolf and keycloak (check the logs if it is okay)
13. check keycloak (keycloak login, incore login, incore token decode)
14. check data wolf
15. go to radiant and remove the volume that was removed in step 7

[ywkim312]

If you need to redo the things or trouble shooting

1. make postgres statefulset's scale to zero
2. delete PVC for postgres
3. delete PV for postgres
4. make postgres satefulset's scale to one
5. go to radiant and remove the volume that was removed on step 2

[ywkim312]

If you need to have a pod that controls the mounted volume (this might needed after postgres pod is one)

apiVersion: v1
kind: Pod
metadata:
  name: temp-pod
spec:
  containers:
  - name: temp-container
    image: busybox
    volumeMounts:
    - mountPath: "/mnt"
      name: postgres-volume
    command: ['sh', '-c', 'echo Container is Running; sleep 3600']
  volumes:
  - name: postgres-volume
    persistentVolumeClaim:
      claimName: postgres-pvc  # Adjust this to match your PersistentVolumeClaim

deploy by

kubectl apply -f temp-pod.yaml

go inside the pod

kubectl exec -it temp-pod -- /bin/sh

to remove the pod

kubectl delete pod <pod-name>

to get the claimName

kubectl get pvc -n incore

the volume is mounted on /mnt/

[ywkim312]

There was two unusual databases in dev hedgedoc (user rhedgedoc) wiki (user wiki)

[ywkim312]

There must be an upgrade of back up process in incore-dev and incore-prod cluster in gitlab chart

incore-postgresql-headless -> incore-postgresql-hl postgresqlPassword -> poastPassword