search

sixfeetup / scaf

scaf provides developers and DevOps engineers with a complete blueprint for a new project using Kubernetes
BSD 3-Clause "New" or "Revised" License
80 stars 12 forks source link

Postgres pod doesn't complete startup after cluster shutdown #182

Open sjlawson opened 7 months ago

sjlawson commented 7 months ago

Use this issue to track bug notes (if it is a bug):

Problem:

Postgres pod will sometimes not start back up after system shutdown, entire cluster has to be deleted and rebuilt, losing data.

Possible cause: cannot connect to persistent volume. Possible root cause: any changes to k8s/local/secrets.yml, causes the pv name to change [redacted secret secrets-config:POSTGRES_HOST]-0.

Possible solutions:

  1. Instead of persistent volume, use a mounted volume
  2. Document the behaviour: Any changes to the secrets.yml file will result in loss of pv
  3. Write a script or document how to change the claim of a new pvc (which gets created when secrets.yml is changed) so that the orphaned pv can be attached to a new pvc
sjlawson commented 7 months ago

I think this is really just a documentation need: if k8s/local/secrets.yml changes it's possible the postgres container won't be able to reconnect to the pvc/pv