Closed joachimdraeger closed 4 years ago
@joachimdraeger this has been merged and added a new test case covering this feature: https://github.com/transferwise/pipelinewise-target-redshift/commit/a4a51bf66a2ea4425839b7d34644bfb85a4c12c2
New version of pipelinewise-target-redshift will be released to pypi soon. Thanks for contributing. 🙇
Adds _skipupdates configuration option which skips the update step when using a primary key.
We have a use case where we import immutable events into Redshift. We require using an event-id as primary key to avoid duplicates, however, in case of a duplicate event, we don't need to update any existing events. Currently, the join on insert and update is the slowest part of the load and this option allows to skip half of it.