mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
MIT License
2.07k stars 100 forks source link

Adding support for Mara Storage #80

Open leo-schick opened 2 years ago

leo-schick commented 2 years ago

The implementation of the storage module from @ice1e0 has been merged into the master branch. I see this as a breaking change and suggest to publish this therefore in the next major version 4.

In addition I think some additional commands should be added for working with files. Here some samples:

optional additional commands:

ice1e0 commented 2 years ago

Its fine for me. There wasn't much contribution so I decided to merge it to master.

leo-schick commented 2 years ago

I digged a bit into it and I think the mara storage/pipeline module should support path configuration from fsspec. Maybe then supporting storages via the URL without specifically mentioning the storage_alias as other tools like pandas does it.