search

m-lab / etl-gardener

Gardener provides services for maintaining and reprocessing mlab data.
Apache License 2.0
13 stars 5 forks source link

Update schema on output tables for JOIN'd tables #368

Closed stephen-soltesz closed 2 years ago

stephen-soltesz commented 2 years ago

This change addresses a long standing failure in gardener where new fields in raw tables would introduce failures to write the joined, materialized tables. The failure was due to the joined table not having the new fields and etl/cmd/update-schema not populating them. This change allows BigQuery to do the update automatically. Removing fields is still not supported.

This change supports deployment of:

Though without integration tests, this change

This change is Reviewable

coveralls commented 2 years ago

Coverage Status

Coverage increased (+0.04%) to 61.106% when pulling 75b4e4a6cb0cfb139c9b54d2f65d2f4f32cd0953 on sandbox-soltesz-fix-joins into 34e026aa8dcb59b68c3b37fd5d274b6d6486151f on master.