anelendata / tap-bigquery

Singer.io tap for extracting data from BigQuery tables
Apache License 2.0
16 stars 33 forks source link

BQ to handle Table Expiration Errors #35

Open Ella6882 opened 1 month ago

Ella6882 commented 1 month ago

When syncing a large volume of data, you may encounter a 'Table Not Found Exception' error. According to Google's documentation, temporary tables expire after 24 hours (discussed here https://www.googlecloudcommunity.com/gc/Data-Analytics/Getting-Table-Not-found-Exception/td-p/710166).

CRITICAL 404 GET [REDACTED] Not found: Table [REDACTED]

google.api_core.exceptions.NotFound: 404 GET [REDACTED] Not found: Table [REDACTED]

This poses a problem because the sync process retries with the same data points instead of progressing to the next timestamp.