search

kestra-io / plugin-aws

https://kestra.io/plugins/plugin-aws/
Apache License 2.0
12 stars 7 forks source link

Create a MinIO plugin #144

Open loicmathieu opened 1 year ago

loicmathieu commented 1 year ago

Create a dedicated plugin for MinIO.

The following task will not work on MinIO S3-compatible object storage.

  - id: play_minio
    type: io.kestra.plugin.aws.s3.List 
    accessKeyId: <accessKeyId>
    bucket: test1
    endpointOverride: https://play.min.io:9000/
    region: us-east-1
    secretKeyId: <secretKeyId>

It generates an UnknownHostException test1.play.min.io.

Therefore, we should create a new plugin that will mimic the existing S3 tasks. For example:

loicmathieu commented 1 year ago

This is because the MinIO installation is not configured for domain style access.

We previously support path style access but had to drop it to support multipart upload and download via the transfer manager.

Pah style access is deprecated on S3 so it may be preferable to have a dedicated plugin for MinIO but it will imply duplicated all tasks.

loicmathieu commented 1 year ago

We decided to go to a new MinIO plugin.

loicmathieu commented 9 months ago

Now that we have a compatibilityMode we can add a property to enable forcePathStyle when using the sync client (only possible in compatibility mode).

jackgray commented 8 months ago

Eagerly awaiting this feature!! Any projections on its availability or how active the endeavor is atm?

No solutions seem to exist for data automations on MinIO that run in container environments. Prefect's S3 plugin requires paid enterprise version, and Airflow does not have a clean Docker integration. The rest out there are lagging on S3 support. Now that I have learned Kestra and have it set up and running, I'd rather not go do it all over again with some other alternative. Really thought S3 support included Minio at the start of this journey :(

anna-geller commented 7 months ago

Hi @jackgray - I saw your request a bit late. Did you manage to solve the issue? if not, we can look into it to add a Minio plugin in the next release in January