Debug: patroni issue for documize

[ShellyXueHan]

Background:

• Documize's db was setup with 1Gi pvc, which has been filled up causing application failure.
• Followed https://github.com/bcgov/ocp-sso/wiki/2.4-Increase-storage-for-statefulset to increase volume.
• However, only copying the db folder caused issues when trying to restart the statefulset (should be doing a pg_dump instead). Error msg:

  2019-12-05 19:24:01,141 ERROR: watch
  Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 535, in _update_chunk_length
  self.chunk_left = int(line, 16)
  ValueError: invalid literal for int() with base 16: b''

• Further debug showed that only a brand new statefulset could be start up without error. However, since there is no proper backup of the data, the existing db cannot join the new cluster as it recognize a different cluster ID.

Fix:

Bring up another postgres v10 pod, copy over the data folder and spin up the db there. Then do pg_dump to obtain a proper copy.

Notes for next steps:

1. could use the instance in test namespace to try it out first as it's having the same issue
2. in prod namespace, pvc postgresql-documize-patroni-prod-2 contains the original copy of db, only remove when patroni is fully up and running with 2 replicas.
3. current patroni documize-patroni-prod is brand new without any data, running only one replica (which is the master pod)
4. refer to secret documize-patroni-prod in prod namespace for db connections.
5. after pg_restore to the patroni pod, update statefulset config to have 2 replicas.
6. if db is running properly, scale the app back to 2 pods
7. if all running, scale patroni to 3 pods

[WadeBarnes]

@caggles, @ShellyXueHan , Some general thoughts on resizing or migrating PVCs, and backup (dump)/restore.

We have some automated tools to help with migrating data from one PVC to another of a different size and/or class that leaves the DCs unchanged in the end; BCDevOps/StorageMigration - MigrationSteps. I've used these tools to migrate data to larger PVCs and PVCs of different classes, for PostgreSQL and Jenkins containers. This should cover nearly all of your PVC resizing or migration needs.

For backup/restore of PostgreSQL (if that is really what is needed) I'd recommend using the BCDevOps/backup-container. At it's core (the PostgreSQL version) is a PostgreSQL v10 container, but the backup scripts provide you tested easy to use backup and restore functionality rather than fiddling with the raw pg_dump and psql backup/restore features directly.