search

amphi-ai / amphi-etl

Python-based Low-code ETL for data manipulation and transformation. Generates Python code you can deploy anywhere.
https://amphi.ai
Other
789 stars 34 forks source link

Cloud inputs and output #21

Closed DuSegura closed 1 month ago

DuSegura commented 3 months ago

Hi Thibaut.

I think it would be great to have cloud sources and targets, specifically data lake ones from the main cloud providers.

I know this is challenging because you would need to deal with security configurations for each cloud but it would be useful.

tgourdel commented 3 months ago

Hi @DuSegura, S3 will come very shortly. I'm indeed working on the connections management to deal with security before tackling the cloud inputs. Thanks a lot

antonisar commented 1 month ago

Related to #57 smart_open supports many cloud backends in a unified way. The examples in its home page are S3 https://github.com/piskvorky/smart_open

tgourdel commented 1 month ago

Hi everyone, I've added support for S3 for all file inputs (using the native pandas capabilities for now). @antonisar I plan to use smart_open in different components (orchestration components). Thanks for the suggestion