transferwise / pipelinewise-target-redshift

Singer.io Target for Amazon Redshift - PipelineWise compatible
https://transferwise.github.io/pipelinewise/
Other
12 stars 65 forks source link

Added option to skip updates in case of immutable records #44

Closed joachimdraeger closed 4 years ago

joachimdraeger commented 4 years ago

Adds _skipupdates configuration option which skips the update step when using a primary key.

We have a use case where we import immutable events into Redshift. We require using an event-id as primary key to avoid duplicates, however, in case of a duplicate event, we don't need to update any existing events. Currently, the join on insert and update is the slowest part of the load and this option allows to skip half of it.

koszti commented 4 years ago

@joachimdraeger this has been merged and added a new test case covering this feature: https://github.com/transferwise/pipelinewise-target-redshift/commit/a4a51bf66a2ea4425839b7d34644bfb85a4c12c2

New version of pipelinewise-target-redshift will be released to pypi soon. Thanks for contributing. 🙇