mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
MIT License
2.07k stars 100 forks source link

Event logging in case of error #33

Closed jankatins closed 4 years ago

jankatins commented 4 years ago

Two problems in this PR:

The first showed up when the mara DB was shut down unexpectedly and we didn't get a slack notification (until our alerting based on last finish times kicked in). the second when you do SQL only fixes which need no restart...

martin-loetzsch commented 4 years ago

It looks good, but why do we need a new Event type? Couldn't we just send Output events? It looks to that the event is only used for printing / displaying stuff.

jankatins commented 4 years ago

Will remove.

When I wrote this I wasn't sure if the code path has a valid pipeline. Seems it does and this exception will probably also not be reused outside of pipelines runs...