mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
MIT License
2.07k stars 100 forks source link

Run pipeline in UI stops when closing website #76

Open leo-schick opened 2 years ago

leo-schick commented 2 years ago

This is probably a known issue to many users using mara but I want to mention it here because I think this should not be that way:

When running a pipeline via the UI using run or run with upstreams it runs in a flask UI thread. This creates several issues:

In addition, when you run a pipeline via the server e.g. from a scheduled CI script or a cron job, you are not able to "connect" to the execution to see what is currently running. The only option to see what is going on is to open the node page and refresh it manually.