Error: couldn't find key DATABASE_USER in Secret airbyte-airbyte-secrets

hongbo-miao commented 2 weeks ago

Helm Chart Version

0.199.0

What step the error happened?

On deploy

Relevant information

Originally asked at Stack Overflow, here is a copy:

I am trying to deploy Airbyte to Amazon EKS by its helm chart. I am using Airbyte v0.63.1 and Helm chart v0.199.0.

Based on this tutorial, I have created Kubernetes secret production-hm-airbyte-secret in advance with values

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
LOG_S3_BUCKET_NAME
LOG_S3_BUCKET_REGION
POSTGRES_USER_NAME
POSTGRES_PASSWORD

Based on this original values.yaml, here is my-values.yaml:

global:
  state:
    storage:
      type: S3
  database:
    type: external
    host: production-hm-airbyte-postgres.xxx.us-west-2.rds.amazonaws.com
    port: 5432
    database: airbyte_db
    user: airbyte_user
    secretName: production-hm-airbyte-secret
    passwordSecretKey: POSTGRES_PASSWORD
  logs:
    accessKey:
      existingSecret: production-hm-airbyte-secret
      existingSecretKey: AWS_ACCESS_KEY_ID
    secretKey:
      existingSecret: production-hm-airbyte-secret
      existingSecretKey: AWS_SECRET_ACCESS_KEY
  storage:
    type: S3
  minio:
    enabled: false
server:
  extraEnv:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_ACCESS_KEY_ID
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_SECRET_ACCESS_KEY
    - name: STATE_STORAGE_S3_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_ACCESS_KEY_ID
    - name: STATE_STORAGE_S3_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_SECRET_ACCESS_KEY
    - name: STATE_STORAGE_S3_BUCKET_NAME
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: LOG_S3_BUCKET_NAME
    - name: STATE_STORAGE_S3_REGION
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: LOG_S3_BUCKET_REGION
worker:
  extraEnv:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_ACCESS_KEY_ID
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_SECRET_ACCESS_KEY
    - name: STATE_STORAGE_S3_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_ACCESS_KEY_ID
    - name: STATE_STORAGE_S3_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_SECRET_ACCESS_KEY
    - name: STATE_STORAGE_S3_BUCKET_NAME
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: LOG_S3_BUCKET_NAME
    - name: STATE_STORAGE_S3_REGION
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: LOG_S3_BUCKET_REGION
postgresql:
  enabled: false
externalDatabase:
  host: production-hm-airbyte-postgres.xxx.us-west-2.rds.amazonaws.com
  port: 5432
  database: airbyte_db
  user: airbyte_user
  existingSecret: production-hm-airbyte-secret
  existingSecretPasswordKey: POSTGRES_PASSWORD

However, after I deploy Airbyte Helm chart v0.199.0, I got error inside pod "airbyte-airbyte-bootloader":

Error: couldn't find key DATABASE_USER in Secret production-hm-airbyte/airbyte-airbyte-secrets

Also, I noticed Airbyte created a secret airbyte-airbyte-secrets during deployment with values

DEFAULT_MINIO_ACCESS_KEY
DEFAULT_MINIO_SECRET_KEY
WORKLOAD_API_BEARER_TOKEN

And missing DATABASE_USER inside.

I try to append airbyte-bootloader section in my-values.yaml. Both

airbyte-bootloader:
  extraEnv:
    - name: DATABASE_USER
      value: airbyte_user

and

airbyte-bootloader:
  secrets:
    DATABASE_USER: airbyte_user

will give error

failed to create typed patch object (production-hm-airbyte/airbyte-airbyte-bootloader; /v1, Kind=Pod): .spec.containers[name="airbyte-bootloader-container"].env: duplicate entries for key [name="DATABASE_USER"]

I searched online and found

But none of them provides a solution.

Any guide would be appreciate, thanks!

My Current Workaround Solution

I found a workaround solution which is patching Kubernetes secret airbyte-airbyte-secrets to add missing DATABASE_USER immediately after deployment.

kubectl patch secret airbyte-airbyte-secrets \
  --namespace=production-hm-airbyte \
  --patch="{\"stringData\":{\"DATABASE_USER\":\"airbyte_user\"}}"

Then airbyte-airbyte-bootloader can pick up this DATABASE_USER from Kubernetes secret airbyte-airbyte-secrets value.

hongbo-miao commented 2 weeks ago

I found a clean solution. I posted at https://stackoverflow.com/a/78644975/2000548 😃

Here is a copy:

After I change from

global
  database:
    # ...
    secretName: production-hm-airbyte-secret
    user: airbyte_user
    passwordSecretKey: POSTGRES_PASSWORD
  # ...

to

global
  database:
    # ...
    secretName: production-hm-airbyte-secret
    userSecretKey: POSTGRES_USER_NAME
    passwordSecretKey: POSTGRES_PASSWORD
  # ...

It does not expect DATABASE_USER any more and can deploy smoothly!

hongbo-miao commented 2 weeks ago

I think users would expect user: airbyte_user should also work though. I will leave it open. Thanks!

marcosmarxm commented 2 weeks ago

Thanks for reporting the issue @hongbo-miao I added this to the deployment team take a look

airbytehq / airbyte