search

amphi-ai / amphi-etl

Python-based Low-code ETL for data manipulation and transformation. Generates Python code you can deploy anywhere.
https://amphi.ai
Other
797 stars 35 forks source link

Multiple backends shortcut #57

Open antonisar opened 3 months ago

antonisar commented 3 months ago

Disclaimer: I have not run this just a coupke of thoughts.

Use ibis-project as a dataframe code output that translates to many known backends incl duckdb, pandas, polars, spark

Use smart_open library in the code output that translates to many storages including cloud ones

Might be not applicable but it will greatly enhance the user base by allowing local testing and large infra deployment with just two configuration changes

tgourdel commented 3 months ago

Hi @antonisar, thanks for the suggestions.

I'm already considering implementing the ibis framework, especially for in-database execution as mentionned in #15.

However, I didn't know about smart_open, very good to know, I'll definitely take a look.