search

cloudfoundry / cf-for-k8s

The open source deployment manifest for Cloud Foundry on Kubernetes
Apache License 2.0
300 stars 115 forks source link

Support rotation of `uaa.database.password` #566

Open ericpromislow opened 4 years ago

ericpromislow commented 4 years ago

Describe the bug

This is in the category of supporting rotation of all credentials (https://www.pivotaltracker.com/story/show/175210100)

To Reproduce

Steps to reproduce the behavior:

  1. Deploy cf-for-k8s
  2. Verify it works by pushing an app or running smoke tests
  3. Change the value of uaa.database.password in a values file.
  4. Redeploy should succeed, but this is because the UAA node is using its original connection to the database
  5. Delete the UAA container (like by running kubectl rollout restart deployment/uaa -n cf-system)
  6. The UAA container will fail to start. The logs will have repeated messages like the following: 
    [CONTAINER] lina.core.ContainerBase.[Catalina].[localhost].[/] SEVERE  Servlet [spring] in web application [] threw load() ex │
│ uaa org.postgresql.util.PSQLException: FATAL: password authentication failed for user "uaa"

Expected behavior

The new UAA instance should be able to connect to the database and start up in a good state.

Notes & Observations:

The UAA container is not getting the new value for uaa.database.password. Specifically, the database.password field in /etc/secrets/database_credentials.yml has the old password after changing the value and rerunning kapp deploy. The database pod in cf-db/cf-db-postgresql-0/cf-db-postgresql has the new value of/docker-entrypoint-initdb.d/secret/uaadb-password`

The password should be successfully rotated during upgrades, and deployment continues successfully.

But the problem is probably on the postgres/db side and not the UAA side if postgres doesn't actually rerun the init script and set the new password.

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175675844

The labels on this github issue will be updated when the story is started.