m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

Add sidestream to BQ update-schema #987

Closed stephen-soltesz closed 3 years ago

stephen-soltesz commented 3 years ago

This change adds support for updating the sidestream BQ tables using update-schema.

This change is in service of automatically expanding the legacy tables to support the new annotation-service fields.

This change also fixes a bug in the etl/schema unit tests by adding the m.Run() which was previously missing.

Testing: verified to WAI on mlab-sandbox.batch.sidestream and mlab-sandbox.base_tables.sidestream tables. This change removes "Description" fields (though they are currently very sparse; more thorough descriptions should be added during standard column migration).


This change is Reviewable

coveralls commented 3 years ago

Pull Request Test Coverage Report for Build 6410


Changes Missing Coverage Covered Lines Changed/Added Lines %
schema/sidestream_schema.go 8 36 22.22%
<!-- Total: 8 36 22.22% -->
Totals Coverage Status
Change from base Build 6404: -4.03%
Covered Lines: 3471
Relevant Lines: 5551

💛 - Coveralls