search

estuary / connectors

Connectors for capturing data from external data sources
Other
50 stars 16 forks source link

materialize-bigquery: Failure to clean up tables during local development #96

Closed willdonnelly closed 1 year ago

willdonnelly commented 2 years ago

When deploying with --wait-and-cleanup to a local temp-data-plane the BigQuery materialization should delete the tables it created on exit. But, at least in my specific situation, it isn't doing this and the tables just linger. This in turn means that subsequent attempts to run the same catalog will silently fail to materialize any new data.

The deploy exits with the following error message (edited lightly for readability):

applying schema updates: query: wait: googleapi: Error 400: Query error: Transaction is aborted due to
concurrent update against table helpful-kingdom-273219:fenestra_materialization_test.flow_checkpoints_v1.
Transaction ID: 61c2c7ad-0000-2ef4-aadd-14c14eec2a00 at [1:185], invalidQuery

This error message appears to be coming from the code responsible for deleting the materialized tables on exit, which would explain why those tables aren't deleted as expected. However according to Johnny: "this smells like a bigger problem [...] because raced transactions against the table are expected to happen".

I have not dug any further into the issue for now, just filing a tracking bug so it doesn't get forgotten.

williamhbaker commented 1 year ago

Since we don't (to the best of my knowledge) use temp-data-plane in this way anymore, and we now understand that BigQuery concurrent update errors are something we have to deal with and retry (see https://github.com/estuary/connectors/pull/742), I'm going to close this out, since I don't think it's currently an issue.

We also do now have tests for the BigQuery materialization which can be run locally and use flowctl to drive the connector, and they will clean up after themselves: https://github.com/estuary/connectors/blob/main/tests/materialize/materialize-bigquery/cleanup.sh

But LMK if I missed anything and we can re-open.