meltano / sdk

Write 70% less code by using the SDK to build custom extractors and loaders that adhere to the Singer standard: https://sdk.meltano.com
https://sdk.meltano.com
Apache License 2.0
97 stars 69 forks source link

docs: Dealing with sorted streams that have an unsorted replication_key_value (check_sorted vs is_sorted) #1580

Open visch opened 1 year ago

visch commented 1 year ago

Documentation type

Tutorials

Description

This ended up being hard for me to figure out, and my brain completely melded check_sorted and is_sorted together, not the sdk's fault it's my fault but I never read about a distinction anywhere so I thought maybe we could write one up.

I think we should add a section to https://sdk.meltano.com/en/latest/implementation/state.html#dealing-with-unsorted-streams for something along the lines of

"Dealing with sorted streams that have an unsorted replication_key_value common with a cursor type replication key":

There are two values in regards to stream sorting built into the SDK one is check_sorted and the other is is_sorted both a stream properties. In the case that you have something like a cursor based replication key value where the cursor is a GUID we know that the data is sorted for us so is_sorted=True , but we also know that we can't check to be sure the replication_key is sorted because it's a GUID so we tell the sdk not to check the sort by using check_sorted=False .

stale[bot] commented 1 year ago

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

tayloramurphy commented 1 year ago

Still relevant

stale[bot] commented 3 months ago

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

edgarrmondragon commented 3 months ago

We have https://sdk.meltano.com/en/v0.39.1a1/faq.html#i-m-seeing-note-progress-is-not-resumable-if-interrupted-in-my-state-files, but should probably expand with what's suggested here.