airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.35k stars 4.16k forks source link

[deploy/helm] Issue with Airbyte Server/Worker components with S3 secret keys #48008

Open talha-naeem1 opened 1 month ago

talha-naeem1 commented 1 month ago

Helm Chart Version

0.634.3

What step the error happened?

Upgrading the Platform or Helm Chart

Relevant information

I am using s3 as storage for logs, and I want to configure credentials for accessing it. I have passed the parameters in the values files as below, but when airbyte server and workers pods are created we are getting this issue: Image

Error: couldn't find key s3-access-key-id in Secret dev/airbyte-airbyte-secrets

As In these deployments the env vars are configured from template as:

        - name: STORAGE_BUCKET_WORKLOAD_OUTPUT
          valueFrom:
            configMapKeyRef:
              name: airbyte-airbyte-env
              key: STORAGE_BUCKET_WORKLOAD_OUTPUT
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: airbyte-airbyte-secrets
              key: s3-access-key-id
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: airbyte-airbyte-secrets
              key: s3-secret-access-key

While in the secret we have key values are following:

apiVersion: v1
kind: Secret
metadata:
  name: airbyte-airbyte-secrets
  namespace: dev
data:
  AWS_ACCESS_KEY_ID: <keyid>
  AWS_SECRET_ACCESS_KEY: <key-sec>
  DATABASE_PASSWORD: <db-pass>
  DATABASE_USER: <user>

So in the deployment yaml of aibryte server and worker, it should be like this:

     - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: airbyte-airbyte-secrets
              key: AWS_ACCESS_KEY_ID
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: airbyte-airbyte-secrets
              key: AWS_SECRET_ACCESS_KEY

Please let me know if I am missing anything here? or is it really a bug in the helm that needs to be fixed in its template?

Thanks in advance!

Relevant log output

No response

marcosmarxm commented 4 weeks ago

Did you follow instructions from our docs? https://docs.airbyte.com/deploying-airbyte/integrations/storage