search

mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
MIT License
2.07k stars 100 forks source link

Fix Problems when frontend and database are in a different timezone #34

Closed jankatins closed 4 years ago

jankatins commented 4 years ago

The problem was that a) the frontend generated timestamps without timezones which postgresql then interpreted as the UTC but which were actually in the timezone which you ran the python code and b) the closing of open runs after an exception which kills the ETL runner itself (not individual tasks) after an error happened to insert a timestamp with timezone of the time on the postgresql server.

-> the error only happens when an exception happens which triggers this code path, e.g. because something shuts down the the runner (e.g. docker instance is shut down or ctrl+c on the command line).

Should fix https://github.com/mara/mara-example-project/issues/12

martin-loetzsch commented 4 years ago

Looks like a no-brainer :)

jankatins commented 4 years ago

Confirmation that this fixes the problem: https://github.com/mara/mara-example-project/issues/12#issuecomment-619790838