Helm install fails with duplicate postgres secrets

[MCDELTAT]

What's Happening

I am installing a fresh instance of Nautobot via the 2.1.3 tagged release. When doing helm install nautobot nautobot/nautobot -f nautobot-2.1.3-values.yaml, I get the following error:

Error: INSTALLATION FAILED: 1 error occurred:
        * secrets "nautobot-postgresql" already exists

Consequently, I get the following error in the events of the postgres pod

Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Warning  FailedScheduling        26s (x2 over 29s)  default-scheduler        0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..
  Normal   Scheduled               23s                default-scheduler        Successfully assigned nautobot/nautobot-postgresql-0 to worker-1.company.com
  Normal   SuccessfulAttachVolume  22s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-3e4c9f83-2cdf-4b17-ae9c-c687cd5875bf"
  Normal   Pulled                  15s                kubelet                  Successfully pulled image "docker.io/bitnami/postgresql:15.4.0-debian-11-r45" in 4.034s (7.129s including waiting)
  Normal   Pulling                 14s (x2 over 22s)  kubelet                  Pulling image "docker.io/bitnami/postgresql:15.4.0-debian-11-r45"
  Warning  Failed                  14s (x2 over 15s)  kubelet                  Error: couldn't find key postgres-password in Secret nautobot/nautobot-postgresql
  Normal   Pulled                  14s                kubelet                  Successfully pulled image "docker.io/bitnami/postgresql:15.4.0-debian-11-r45" in 661ms (661ms including waiting)

I did a dry run to get the full manifest output and confirmed that it is trying to create duplicate secrets. helm install nautobot nautobot/nautobot -f nautobot-2.1.3-values.yaml --dry-run > nautobot-2.1.3_fullmanifest.yml

Within the full manifest output, I get the two duplicates. Line 96:

---
# Source: nautobot/charts/postgresql/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: nautobot-postgresql
  namespace: "nautobot"
  labels:
    app.kubernetes.io/instance: nautobot
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: postgresql
    app.kubernetes.io/version: 15.4.0
    helm.sh/chart: postgresql-12.12.10
type: Opaque
data:
  postgres-password: "adifferentsecret"
  password: "thiswasmypassword"
  # We don't auto-generate LDAP password when it's not provided as we do for other passwords

Line 149:

---
# Source: nautobot/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: nautobot-postgresql
  namespace: "nautobot"
  labels:
    app.kubernetes.io/name: nautobot
    helm.sh/chart: nautobot-2.1.3
    app.kubernetes.io/instance: nautobot
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: "2.2.7"
    app.kubernetes.io/component: nautobot
type: Opaque
data:
  password: "thiswasmypassword"

How to replicate

Use the following values.yml file in a manual installing with a command like helm install nautobot nautobot/nautobot -f nautobot-2.1.3-values.yaml (⚠️NOTE: Might need to change/remove ingress settings).

# Nautobot server
nautobot:
  enabled: true
  image:
    registry: "ghcr.io"
    repository: "nautobot/nautobot"
    tag: "2.2.7-py3.11"
    pullPolicy: "Always"
  replicaCount: 2
  livenessProbe:
    enabled: true
    exec:
      command:
        - "bash"
        - "-c"
        - "nautobot-server health_check"
    initialDelaySeconds: 3
    periodSeconds: 15
    timeoutSeconds: 10
    failureThreshold: 3
    successThreshold: 1
  resources:
    requests:
      cpu: "300m"
      memory: "1280M"
    limits:
      cpu: "1000m"
      memory: "8704M"
  podSecurityContext:
    # -- Enable the Nautobot Pod security context
    enabled: true
    fsGroup: 999
    seccompProfile:
      type: "RuntimeDefault"
  db:
    engine: "django.db.backends.postgresql"
    host: "postgres"
    name: "nautobot"
    user: "nautobot"
    password: "thiswasmypassword"
    port: 5432
    timeout: 300
  superUser:
    enabled: true
    email: "admin@example.com"
    username: "admin"
    password: "thiswasmypassword"
  persistenceMediaFiles:
    enabled: true
  extraVolumes:
    - name: "jobs-data"
      persistentVolumeClaim:
        claimName: "jobs-data"
  extraVolumeMounts:
    - mountPath: "/opt/nautobot/jobs/"
      name: "jobs-data"

ingress:
  enabled: true
  ingressClassName: "nginx"
  hostname: "myhostname.company.com"
  path: /
  tls: true
  secretName: "nautobot-tls"
  backendProtocol: "http"

postgresql:
  enabled: true
  image:
    pullPolicy: "Always"
  auth:
    username: "nautobot"
    database: "nautobot"
    password: "thiswasmypassword"
  primary:
    podSecurityContext:
      seccompProfile:
        type: "RuntimeDefault"
    containerSecurityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: false  # Must be set to false
      capabilities:
        drop:
          - "ALL"

redis:
  enabled: true
  architecture: "standalone"
  auth:
    enabled: true
    password: "thiswasmypassword"
  image:
    pullPolicy: "Always"
  master:
    containerSecurityContext:
      enabled: true
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop:
          - "ALL"
    disableCommands: []
    podSecurityContext:
      seccompProfile:
        type: "RuntimeDefault"
  metrics:
    containerSecurityContext:
      enabled: true
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop:
          - "ALL"
  replica:
    containerSecurityContext:
      enabled: true
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop:
          - "ALL"
      disableCommands: []
    podSecurityContext:
      seccompProfile:
        type: "RuntimeDefault"
  sentinel:
    containerSecurityContext:
      enabled: true
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: false
      capabilities:
        drop:
          - "ALL"
    podSecurityContext:
      seccompProfile:
        type: "RuntimeDefault"
  serviceAccount:
    automountServiceAccountToken: false

celery:
  enabled: true
  concurrency: 20 # If set to 0, then it will use all cores on a machine (96+), causing low memory per worker.
  replicaCount: 2
  revisionHistoryLimit: 3
  resources:
    requests:
      cpu: "400m"
      memory: "1G"
    limits:
      cpu: "3000m"
      memory: "8G"
  extraVolumes:
    - name: "jobs-data"
      persistentVolumeClaim:
        claimName: "jobs-data"
  extraVolumeMounts:
    - mountPath: "/opt/nautobot/jobs/"
      name: "jobs-data"

[MCDELTAT]

I was able to get around this using a manually created secret. It took lots of fighting and looking between this chart and the postgresql chart. You'll want to create an opaque secret with the data fields of postgres-password (admin user password) and password (nautobot user password). The relevant sections to change are:

nautobot:
  db:
    engine: "django.db.backends.postgresql"
    host: "postgres"
    name: "nautobot"
    user: "nautobot"
    existingSecret: "nautobot-postgresql"
    existingSecretPasswordKey: "password"

and

postgresql:
  auth:
      username: "nautobot"
      database: "nautobot"
      existingSecret: "nautobot-postgresql"
      secretKeys:
        adminPasswordKey: postgres-password
        userPasswordKey: password

Hopefully this can still get fixed. I can stare at it a little longer. I guess I just want confirmation of what the expected behavior from the team is so I'm not changing it just to suit my needs.

[gertzakis]

Hello @MCDELTAT! If you put credentials under postgresql.auth then nautobot will use those ones to connect to database, so there is no need to also add them under nautobot.db. You need to add credentials under nautobot.db if you connect to an external database. Regarding the existing secrets, it's a solution to not store credentials inside values.yaml, more info can be found here . Hope this simplifies it a bit :)

[MCDELTAT]

Good point. I swore I had looked at the template for the secrets and it seemed unconditional for the Nautobot one. I see however that's not the case now, as seen here: https://github.com/nautobot/helm-charts/blob/v2.1.3/charts/nautobot/templates/secret.yaml#L23C5-L23C36

You can go ahead and close this, unless you think this could use better documentation in the values.yml. I can try to pick that up for you if you'd like.

[gertzakis]

Sure, I can leave it open for a while if you want to document it better. Help is always welcome 😉