PostgreSQL with existing secret broken after update 4.0.0

[archer-321]

Describe your Issue

Setting postgresql.global.postgresql.auth.existingSecret and the secret keys in postgresql.global.postgresql.auth.secretKeys doesn't work after update 4.0.0.

Trying to deploy Nextcloud with the configuration below returns PASSWORDS ERROR: The secret "postgres-creds" does not contain the key "password" instead.

Steps to reproduce:

1. Create values.yml and secret.yml with the below content.
2. Create a new namespace: kubectl create namespace nextcloud
3. Create the secret: kubectl apply -f secret.yml
4. Try to deploy any version >=4.0.0: helm install -n nextcloud -f values.yml albacore nextcloud/nextcloud --version=4.0.0

Version 3.5.22 works as expected.

Logs and Errors

> helm install -n nextcloud -f values.yml albacore nextcloud/nextcloud --version=4.0.0

Error: INSTALLATION FAILED: execution error at (nextcloud/charts/postgresql/templates/secrets.yaml:23:16):
PASSWORDS ERROR: The secret "postgres-creds" does not contain the key "password"

Describe your Environment

• Kubernetes distribution:\ Tested with a fresh multi-node minikube cluster and a production cluster maintained with kubeadm

• Helm Version (or App that manages helm): 3.12.0

• Helm Chart Version: 4.0.0 and above

• values.yml:

ingress:
  enabled: true
externalDatabase:
  enabled: true
  type: postgresql
  host: albacore-postgresql-0:5432
  user: nextcloud
  database: nextcloud
  existingSecret:
    # Both, `true` and `false` for this value don't work.
    enabled: false
    secretName: postgres-creds
    passwordKey: user
internalDatabase:
  enabled: false
postgresql:
  enabled: true
  global:
    postgresql:
      auth:
        existingSecret: postgres-creds
        secretKeys:
          adminPasswordKey: admin
          userPasswordKey: user
          replicationPasswordKey: replication

• secret.yml (for testing purposes):

apiVersion: v1
data:
  admin: Zm9v
  replication: YmF6
  user: YmF6
kind: Secret
metadata:
  name: postgres-creds
  namespace: nextcloud
type: Opaque

[jessebot]

Thanks for reporting this! Can you please let me know if you're also setting your externalDatabase settings? This is also required when using anything that isn't the internalDatabase. Example using some of the values you mentioned in your post:

##
## External database configuration
##
externalDatabase:
  enabled: true

  ## Supported database engines: mysql or postgresql
  type: postgresql

  ## Database host
  host: localhost:5432

  ## Database user - if it's not nextcloud, please change this to your preferred database user
  user: nextcloud

  ## Database name
  database: nextcloud

  ## Use a existing secret
  existingSecret:
    enabled: false
    secretName: postgres-creds
    # usernameKey: db-username
    passwordKey: user
    # hostKey: db-hostname-or-ip
    # databaseKey: db-name

[provokateurin]

I'm using Postgres with existing secret and it still works for me :thinking: Here is the relevant part of my values.yaml:

externalDatabase:
  enabled: true
  type: postgresql
  host: "acid-nextcloud-cluster:5432"
  existingSecret:
    enabled: true
    secretName: nextcloud.acid-nextcloud-cluster.credentials.postgresql.acid.zalan.do
    usernameKey: username
    passwordKey: password

[archer-321]

Can you please let me know if you're also setting your externalDatabase settings? This is also required when using anything that isn't the internalDatabase.

Thank you for the response!

I indeed should have included this section in my testing configuration, but it's filled out in my production configuration.

I included your section in my example, and I can still reproduce the problem on a fresh minikube cluster. Here's my new values.yml file (I'll also update the issue description):

ingress:
  enabled: true
externalDatabase:
  enabled: true
  type: postgresql
  host: albacore-postgresql-0:5432
  user: nextcloud
  database: nextcloud
  existingSecret:
    # Both, `true` and `false` for this value don't work.
    enabled: false
    secretName: postgres-creds
    passwordKey: user
internalDatabase:
  enabled: false
postgresql:
  enabled: true
  global:
    postgresql:
      auth:
        existingSecret: postgres-creds
        secretKeys:
          adminPasswordKey: admin
          userPasswordKey: user
          replicationPasswordKey: replication

[archer-321]

I'm using Postgres with existing secret and it still works for me 🤔 Here is the relevant part of my values.yaml:

Does this also work with keys not named password? From the error message I listed in the issue description, it seems like the key name password might be hardcoded somewhere :thinking:

[provokateurin]

https://github.com/nextcloud/helm/blob/0b6bd840f2757e7737566bf388f0e45aa56467d9/charts/nextcloud/templates/_helpers.tpl#L108 looks like it is not hardcoded

[jessebot]

As Kate mentioned, it doesn't look like it's hardcoded, but I went and checked my own values and it looks like I don't use the word password for my key at all here:

# note this is only indented like this because it's deployed via argo, 
# but normally this would not be indented
            internalDatabase:
              enabled: false
            externalDatabase:
              enabled: true
              type: postgresql
              host: localhost:5432
              user: nextcloud
              database: nextcloud
              existingSecret:
                enabled: true
                secretName: nextcloud-pgsql-credentials
                usernameKey: nextcloudUsername
                passwordKey: nextcloudPassword
            postgresql:
              enabled: true
              # we can't use fullnameOverride till https://github.com/nextcloud/helm/pull/442 is merged
              # fullnameOverride: nextcloud-postgresql
              global:
                postgresql:
                  auth:
                    username: nextcloud
                    database: nextcloud
                    existingSecret: nextcloud-pgsql-credentials
                    secretKeys:
                      userPasswordKey: nextcloudPassword
                      adminPasswordKey: postgresPassword

Out of curiosity, since this seems to be an issue with the bitnami postgresql sub chart, can you try and use the bitnami helm chart on it's own and see if you can deploy it? I ask because the error you posted shows:

nextcloud/charts/postgresql/templates/secrets.yaml:23:16

And that points to this file: https://github.com/bitnami/charts/blob/396cec8e6bd754cffd6b4114c33c2d5f5199b717/bitnami/postgresql/templates/secrets.yaml#L23C44-L23C44

Any parameter passed under the postgresql parameter in values.yaml gets passed to that subchart, which in this case is bitnami/postgresql.

[archer-321]

TL;DR - Working Nextcloud values.yaml

```yaml externalDatabase: enabled: true type: postgresql host: albacore-postgresql-0:5432 user: nextcloud database: nextcloud existingSecret: enabled: false secretName: postgres-creds passwordKey: user internalDatabase: enabled: false postgresql: enabled: true # ========== This section is important ========== auth: existingSecret: postgres-creds secretKeys: adminPasswordKey: admin userPasswordKey: user replicationPasswordKey: replication # ========== End of important section ========== global: postgresql: auth: existingSecret: postgres-creds secretKeys: adminPasswordKey: admin userPasswordKey: user replicationPasswordKey: replication ```

---

can you try and use the bitnami helm chart on it's own and see if you can deploy it?

After copying the values set by Nextcloud (most notably, global.postgresql.auth.username), I could indeed reproduce this problem with the upstream Bitnami chart.

As far as I can tell, this problem is caused by this block setting the key to the value of auth.secretKeys.userPasswordKey for every deployment that doesn't use the postgres user.

With the chart's default values, this evaluates to password.

A simple fix is copying postgresql.global.postgresql.auth.secretKeys to postgresql.auth.secretKeys in values.yml for Nextcloud.

---

Bitnami's documentation states the following for global.postgresql.auth.secretKeys.userPasswordKey.

Name of key in existing secret to use for PostgreSQL credentials (overrides auth.secretKeys.userPasswordKey).

At this point, I'm unsure whether the documentation is wrong, the upstream Bitnami chart is implemented incorrectly, or whether Nextcloud's default values should recommend adding a postgresql.auth section if users decide to use custom keys for their credentials secret :confused:

[jessebot]

First of all, great detective work! I know this was frustrating, so thanks for sticking with it :)

After copying the values set by Nextcloud (most notably, global.postgresql.auth.username), I could indeed reproduce this problem with the upstream Bitnami chart.

Then this should actually be resolved in the bitnami chart. Can you please open an issue there? In the meantime, anyone having issues with nextcloud's community helm chart, this chart, can search the issues here, but it should definitely be fixed upstream.

[archer-321]

Can you please open an issue there?

I agree that this is likely unexpected behaviour (and thus a bug) in Bitnami's chart. I created an issue after double-checking that this has yet to be reported. You can find the bug report here: bitnami/charts#19202.

[archer-321]

With bitnami/charts#19281 merged, this seems to be fixed upstream in bitnami/postgresql 12.11.2 :rocket:

[jessebot]

Awesome work, @archer-321 ! :) And thanks for keeping us updated 💙

[jessebot]

Oh, got too excited there 😅

We are happy to accept a PR to get PostgreSQL upgraded in the sub-chart here as well :) You just need to tick up both the subchart version and the nextcloud helm chart version, and then run helm dep update in the helm chart directory and push up all the files.

[mike12806]

So weirdly I wasn't getting this message before, but I am now after trying to upgrade to the latest chart:

Error: INSTALLATION FAILED: execution error at (nextcloud/charts/postgresql/templates/secrets.yaml:23:16):
PASSWORDS ERROR: The secret "nextcloud-nonprod-postgresql" does not contain the key "[insert password here]"

Here's my values.yaml

postgresql:
  enabled: true
  global:
    postgresql:
      auth:
        database: nextcloud-nonprod
        existingSecret: ''
        password: $PASSWORD
        secretKeys:
          adminPasswordKey: $PASSWORD
          replicationPasswordKey: $PASSWORD
          userPasswordKey: $PASSWORD
        username: nextcloud

I tried removing the password variable without success; I'm guessing I need to remove the reference to userPasswordKey, maybe?

[archer-321]

password: $PASSWORD
        secretKeys:
          adminPasswordKey: $PASSWORD
          replicationPasswordKey: $PASSWORD
          userPasswordKey: $PASSWORD

@mike12806, please note that the entries for secretKeys describe the keys (as in entry names) in the secret. They're not the secret itself. With the global secret keys configuration being effectively ignored, that wasn't an error until now.

The upstream documentation for global.postgresql.auth.secretKeys mentions:

Name of key in existing secret to use for PostgreSQL credentials (overrides auth.secretKeys.adminPasswordKey). Only used when global.postgresql.auth.existingSecret is set.

In other words, secretKeys is only necessary if you use an existing secret.

Does removing the entire secretKeys entry fix your deployment?

[mike12806]

password: $PASSWORD

        secretKeys:

          adminPasswordKey: $PASSWORD

          replicationPasswordKey: $PASSWORD

          userPasswordKey: $PASSWORD

@mike12806, please note that the entries for secretKeys describe the keys (as in entry names) in the secret. They're not the secret itself. With the global secret keys configuration being effectively ignored, that wasn't an error until now.

The upstream documentation for global.postgresql.auth.secretKeys mentions:

Name of key in existing secret to use for PostgreSQL credentials (overrides auth.secretKeys.adminPasswordKey). Only used when global.postgresql.auth.existingSecret is set.

In other words, secretKeys is only necessary if you use an existing secret.

Does removing the entire secretKeys entry fix your deployment?

Removing userPasswordKey did the trick actually