search

dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
https://dlthub.com/docs
Apache License 2.0
2.38k stars 154 forks source link

incremental `scd2` with `merge_key` #1818

Open jorritsandbrink opened 1 week ago

jorritsandbrink commented 1 week ago

Description

Adds merge_key support for scd2 to enable incremental extracts.

When a merge_key is provided, absent records are only retired if their merge_key value is present in the source extract.

@dlt.resource(
    merge_key="my_natural_key",
    write_disposition={"disposition": "merge", "strategy": "scd2"}
)
def my_incremental_dim_data():
    ...

Related Issues

Closes #1789

netlify[bot] commented 1 week ago

Deploy Preview for dlt-hub-docs ready!

Name Link
Latest commit 84c101bd7049967afca60e6fad703d09b61f62bb
Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/66f7cfb2d32b140008588a65
Deploy Preview https://deploy-preview-1818--dlt-hub-docs.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

jorritsandbrink commented 13 hours ago

@rudolfix I've adapted the code per your suggestions. Can you review?