mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
MIT License
2.07k stars 100 forks source link

Fix inconsistent state after error #24

Closed jankatins closed 4 years ago

jankatins commented 4 years ago

There are two places where we could end up with inconsistent state between what the DB has and what the incremental copy state/ the file dependency hashes know about:

In both cases, we now simply delete the old state before the action (copy/execute) is attempted. This should trigger a full copy in the next run in case the copy is not successful/ a full run if the deploy is reverted.

Closes: #18

jankatins commented 4 years ago

I tested this by triggering the problems locally and this fixes the problems. I will report back in a few days and a few more regular loads...

jankatins commented 4 years ago

I've tested this now locally for about a week worth of loads (most if not all of them not triggering the new codepaths itself, as the loads were without problems). In artificial runs which simulated the error conditions it worked as intended. So I feel comfortable to merge this.