transferwise / pipelinewise-target-redshift

Singer.io Target for Amazon Redshift - PipelineWise compatible
https://transferwise.github.io/pipelinewise/
Other
12 stars 65 forks source link

_sdc_batched_at is per row #28

Open Limess opened 4 years ago

Limess commented 4 years ago

Expected

_sdc_batched_at is set consistently per batched load into redshift.

Actual

_sdc_batched_at is set to the current time when the row was processed, this results n distinct values where n is the number of rows processed. This means that to do any kind of comparison on the batch a datetime operator needs to be used.