search

mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
MIT License
2.08k stars 102 forks source link

Add ability to specify modification value type in CopyIncrementally #53

Closed jankatins closed 3 years ago

jankatins commented 4 years ago

Some databases do not automatically convert from string to the type of the other side in a comparison: e.g. timestamp_column >= '2020-01-01' fails. This would break incremental loading as there we generated this kind of comparison. The new modification_comparison_type argument to CopyIncrementally() will allow to alter this to timestamp_column >= TIMESTAMP '2020-01-01'.

Also fixup the docstring for comparison_value_placeholderargument which should actually be named comparison_placeholderand now at least the docstring reads like that.

jankatins commented 3 years ago

We have run this in Prod (mostly without types, but in one daily pipe with a type) now for quite some time, I consider it read to go in.

martin-loetzsch commented 3 years ago

LGTM