Open visch opened 1 year ago
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen
label, or request that it be added.
Still relevant
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen
label, or request that it be added.
We have https://sdk.meltano.com/en/v0.39.1a1/faq.html#i-m-seeing-note-progress-is-not-resumable-if-interrupted-in-my-state-files, but should probably expand with what's suggested here.
Documentation type
Tutorials
Description
This ended up being hard for me to figure out, and my brain completely melded
check_sorted
andis_sorted
together, not the sdk's fault it's my fault but I never read about a distinction anywhere so I thought maybe we could write one up.I think we should add a section to https://sdk.meltano.com/en/latest/implementation/state.html#dealing-with-unsorted-streams for something along the lines of
"Dealing with sorted streams that have an unsorted replication_key_value common with a cursor type replication key":
There are two values in regards to stream sorting built into the SDK one is
check_sorted
and the other isis_sorted
both a stream properties. In the case that you have something like a cursor based replication key value where the cursor is a GUID we know that the data is sorted for us sois_sorted=True
, but we also know that we can't check to be sure the replication_key is sorted because it's a GUID so we tell the sdk not to check the sort by usingcheck_sorted=False
.