Make Tap Resumable By Setting Sorted Flag

[wesseljt]

Currently if I run an incremental tap I get a state like the one below. If the tap doesn't complete the run cannot be resumed.

{"singer_state": {"bookmarks": {"table": {"replication_key": "last_upd_dt", "replication_key_value": "2023-08-02T10:07:59.963000"}, "table2": {"replication_key": "last_upd_dt", "replication_key_value": "2023-07-26T16:04:20"}, "table2": {"replication_key_signpost": "2023-08-25T11:53:52.673748+00:00", "starting_replication_value": "2000-01-01T00:00:00Z", "progress_markers": {"Note": "Progress is not resumable if interrupted.", "replication_key": "last_upd_dt", "replication_key_value": "2020-05-01T14:40:06.790000"}}}}}

While it appears that the data is indeed sorted by the tap It seems the is_sorted = true needs to be set to enable progress to be resumed.

Docs: https://sdk.meltano.com/en/latest/incremental_replication.html#example-code-timestamp-based-incremental-replication

Previous Discussion Thread: https://meltano.slack.com/archives/C01TCRBBJD7/p1661992776436389 Current Discussion: https://meltano.slack.com/archives/C01TCRBBJD7/p1692976998972359

[BuzzCutNorman]

@wesseljt A quick question do you use meltano run or meltano elt ?

[wesseljt]

Meltano run

[BuzzCutNorman]

If you run meltano state list you can see the all the pipelines that have run. Is the pipeline you have been running present in the listing? if it is present, please run meltano state get <pipeline name here>. Does the output of the state get command match your initial example?

[BuzzCutNorman]

I was able to replicate this issue as defined on my test server using meltano run tap-mssql. I was able to make a change to tap-mssql locally that stops the following message from being put in the state.json file.

"Note": "Progress is not resumable if interrupted." 

I still need to work on setting up a scenario in which I can test if resume does or does not work.