Closed henriquemeloo closed 2 weeks ago
Here: https://github.com/airbytehq/airbyte/blob/5ea2b93c38ee8aaf5e4a161a82b4d6ee5513ac1c/airbyte-integrations/connectors/source-google-sheets/source_google_sheets/source.py#L227 we can see that this exception is logged and not re-raised, as it maybe should be, so that the sync would exit as failed.
We're rather actively looking into cases like this, and have a specific new error type for these — it definitely should not be just swallowed. Thank you for raising this!
/cc @maxi297 and @ChristoGrab, and @brianjlai — perhaps google sheets could be a good candidate for a cleanup once we release error handlers in low-code CDK and Python CDK sane defaults.
@natikgadzhi maybe https://github.com/airbytehq/airbyte/pull/35404 could help?
@natikgadzhi maybe #35404 could help?
It could help, but does not guarantee we won't still be swallowing these errors when they happen, which is, in my opinion, a critical problem, since this is leading to (very) incomplete syncs showing up as successful.
Yeah, if we're silently giving up but not raising the transient error to the platform, it's bad. We just introduced transient (rate limit) errors in the protocol, and @maxi297 and @katmarkham's team are looking in implementing them and cleaning up the highest usage connectors, which this is one of.
So — we will get to this rather quickly. @henriquemeloo, if we gave you an example of a fix in another connector, would you be interested in trying to put together a PR?
@henriquemeloo can you try latest version of Google Sheets connector. It has the batch size, maybe you can increase the value to not hit the 429 error
@natikgadzhi yes. I could try to implement it. I'd appreciate if you could send the example.
@strosek, @ChristoGrab, do you happen to have an example of error improvements work that you're doing now to highlight how to exit with non-zero exit code and send an error message for transient errors?
@natikgadzhi @strosek @ChristoGrab do you have an example of how this is being handled so I could try to contribute with this fix?
Connector Name
source-google-sheets
Connector Version
0.4.0
What step the error happened?
During the sync
Relevant information
When the connector reaches rate limits and cannot continue the sync, it exits as success, so you have a successful job and incorrect data in the destination.
Relevant log output
Contribute