mozilla / bigquery-etl

Bigquery ETL
https://mozilla.github.io/bigquery-etl
Mozilla Public License 2.0
241 stars 98 forks source link

Union default_table_workgroup_access with table-level workgroup_access in metadata generation logic #5830

Open whd opened 6 days ago

whd commented 6 days ago

When we implemented retention metadata support via default_table_workgroup_access, we introduced a gotcha where unless you specify the value of default_table_workgroup_access explicitly in a table's manually created workgroup_access, access to that table will be restricted to just the manual grants if there are deprecated tables in the same dataset.

https://github.com/mozilla/bigquery-etl/commit/80dc78673f60c2f4d5141d797ff84ec1875298f2 is an example of this: if there were deprecations in mozilla_org_derived, give the way our metadata generation currently works, workgroup:mozilla-confidential would have lost access to the table.

We should for non-deprecated tables automatically apply default_table_workgroup_access to the table-level metadata.yaml workgroup_access, and probably have a CI/lint check to warn against the redundant specification in both places once this is implemented.

┆Issue is synchronized with this Jira Task