search

redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.09k stars 814 forks source link

[Request] Output to S3 compatible #1455

Open reski-rukmantiyo opened 1 year ago

reski-rukmantiyo commented 1 year ago

Hi all,

I just found this wonderful apps. Help us very much. And also very handy to solve our problem. Just wondering, can I make request or do you have plan to support s3 compatible? If not support, please share us, where I can start from to develop output support for s3.

Thanks you very much

mihaitodor commented 1 year ago

Hey @reski-rukmantiyo, thanks for trying out Benthos! Just to clarify, are you looking to have an input and output based on https://github.com/minio/minio-go ? Currently, there is an aws_s3 input and output.

reski-rukmantiyo commented 1 year ago

Yes @mihaitodor . You're correct. Currently I'm looking for input and output to s3 compatible Our S3 based on NetApp StorageGrid which similiar like aws s3. But the difference only using path style. The missing right now is the configuration for url (not reference to s3) and also path style active or not. Do you have any solutions for this?

Thanks

mihaitodor commented 1 year ago

It will require an enhancement (feel free to send a PR if you have time). Otherwise https://github.com/benthosdev/benthos-plugin-example shows you how to create your own input / output in a custom build and you can expand that to use https://github.com/minio/minio-go.

LE: See below.

Jeffail commented 1 year ago

Hey, I'm pretty sure it should "just work" if you set this field: https://www.benthos.dev/docs/components/inputs/aws_s3#force_path_style_urls, all s3 components have it as an option.

reski-rukmantiyo commented 1 year ago

hi @Jeffail thanks for the update. so the missing piece only for s3 compatible url. thanks for this