Mainly introduces a deploy module for table deployment logic. Additionally does the following:
Adds specific exceptions for skipped and failed deploys.
Updates multiprocessing code to concurrent.futures to collect individual exceptions as exception handling instead of using return values.
Moves metadata attaching logic to metadata module.
Update backfill initiate tests to mock deployment. These tests currently pass because skipped deploys currently "fail" silently in the click deploy command. Running a backfill without a schema in production will skip the deploy and fail when running the backfill queries. An example of this is the conversation linked in https://github.com/mozilla/bigquery-etl/pull/5646. (Missing schemas should be fixed by https://github.com/mozilla/bigquery-etl/pull/5643).
quiiver
commented
1 month ago
Mainly introduces a
deploymodule for table deployment logic. Additionally does the following:concurrent.futuresto collect individual exceptions as exception handling instead of using return values.