search

slingdata-io / sling-cli

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
https://docs.slingdata.io
GNU General Public License v3.0
301 stars 16 forks source link

Retry on Error/Fail #242

Open ataft opened 3 months ago

ataft commented 3 months ago

Is there any option to retry a replication stream if it fails? I'm working in an environment with flaky connectivity issues (Oracle Cloud DB) that somewhat randomly ends the connection. E.g. 5 out of my 80 streams will fail. I'd like to just say retry n times on fail rather than some more sophisticated orchestration. Is this a bad idea (i.e. is there a better approach)?

Or does Sling handle this automatically? I'm wondering if each error during iteration in nextFunc is a retry...

--------------------------- schema.tablename ---------------------------
~ execution failed

~ error during iteration in nextFunc
read tcp 1.2.3.4:41734->5.6.7.8:1521: i/o timeout

context canceled

~ error during iteration

~ error during iteration in nextFunc
read tcp 1.2.3.4:41734->5.6.7.8:1521: i/o timeout

context canceled

context canceled

~ error writing to s3

~ error during iteration in nextFunc
read tcp 1.2.3.4:41734->5.6.7.8:1521: i/o timeout

context canceled

~ error during iteration

~ error during iteration in nextFunc
read tcp 1.2.3.4:41734->5.6.7.8:1521: i/o timeout

context canceled

~ error during iteration in nextFunc
read tcp 1.2.3.4:41734->5.6.7.8:1521: i/o timeout

context canceled

~ error during iteration

~ error during iteration in nextFunc
read tcp 1.2.3.4:41734->5.6.7.8:1521: i/o timeout

context canceled
flarco commented 3 months ago

Hi, error during iteration in nextFunc means error getting next record, so it's not a retry.

Retries are in the works. Will inform you when ready!