airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.95k stars 4.09k forks source link

Storage AWS Credentials with custom secret name does not work #46329

Open bgadrian opened 2 weeks ago

bgadrian commented 2 weeks ago

Helm Chart Version

1.0.0

What step the error happened?

On deploy

Relevant information

Airbyte 1.0.0 OSS, Help deployment via abctl install

Given the following values.xml

minio:
  enabled: false

global:
  storage:
    type: "s3" # s3 | gcs | minio - default | (local)
    storageSecretName: "airbyte-config-secrets"
    bucket:
      activityPayload: "XXX"
      log: "XXX"
      state: "XXX"
      workloadOutput: "XXX"
    s3:
      region: "eu-central-1"
      authenticationType: "credentials"
  state:
    storage:
      type: "s3"
  secretsManager:
    type: "awsSecretManager" # awsSecretManager | googleSecretManager | vault
    secretsManagerSecretName: "airbyte-config-secrets"
    awsSecretManager:
      region: "eu-central-1"
      authenticationType: "credentials" # credentials | instanceProfile

Produces an invalid YAML more exactly it has new lines in it


        - name: AWS_ACCESS_KEY_ID 
          valueFrom:
            secretKeyRef:
              name: 
          airbyte-config-secrets
              key: s3-access-key-id
        - name: AWS_SECRET_ACCESS_KEY 
          valueFrom:
            secretKeyRef:
              name: 
          airbyte-config-secrets
              key: s3-secret-access-key

Relevant log output

Error: YAML parse error on airbyte/charts/server/templates/deployment.yaml: error converting YAML to JSON: yaml: line 226: could not find expected ':'
helm.go:86: 2024-10-03 12:09:10.332925 +0300 EEST m=+0.620652751 [debug] error converting YAML to JSON: yaml: line 226: could not find expected ':'
YAML parse error on airbyte/charts/server/templates/deployment.yaml
ayanguas commented 1 day ago

I solved changing storageSecretName and secretsManagerSecretName to secretName