airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.05k stars 4.11k forks source link

[helm, external-minio] Cannot find how to use an external Minio #36826

Open francois-travais opened 7 months ago

francois-travais commented 7 months ago

Helm Chart Version

0.64.0

What step the error happened?

Upgrading the Platform or Helm Chart

Revelant information

I'm using an external minio with Airbyte. I'm using Helm CHart 0.53.313 with success today. But I cannot upgrade to 0.64.0 and keeping the internal minio chart disabled, the property global.logs.minio.enabled is still in the configuration but is useless.

Relevant log output

No response

marcosmarxm commented 6 months ago

Thanks for reporting @francois-travais I added the issue to the deployment team backlog for future invesgation. There are some work being done in this area hope we can get an update soon.

aqeelat commented 2 months ago

I'm highly interested in this as well.

I tried avoiding setting global.storage.type: "minio" because this will install the statefulset.

So far, this is what I have:

global:
  storage:
    type: "" 
    storageSecretName: "airbyte-config-secrets"

    bucket:
      log: "dev-airbyte-bucket"
      state: "dev-airbyte-bucket"
      workloadOutput: "dev-airbyte-bucket"
    s3:
      authenticationType: "credentials"
      accessKeyIdSecretKey: "minio-access-key"
      secretAccessKeySecretKey: "minio-secret-key"
    minio:
      endpoint: "http://minio.minio.svc.cluster.local"

However, I still cannot set S3_PATH_STYLE_ACCESS to true because it specifically checks the if type is equal to minio (link).

I'm considering subcharting this and overriding the helper functions but I don't know if that will even work.

djpirra commented 1 month ago

Any update on this? Struggling with using an external Minio configuration as well.

Plus @aqeelat I also tried your approach but my server fails saying that Region cannot be blank.