m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

Add NDTWeb100 struct to mirror current BQ schema #988

Closed stephen-soltesz closed 3 years ago

stephen-soltesz commented 3 years ago

This change adds a new Go struct that mirrors the current ndt web100 schema.

Notable differences:

I accidentally updated part of the mlab-sandbox.base_tables.ndt table during testing of update-schema so the extra columns noted above are already present in the sandbox tables.

To compare this and production schemas:

$ bq show --format=prettyjson mlab-sandbox:base_tables.ndt | jq .schema.fields > new.schema
$ bq show --format=prettyjson mlab-oti:base_tables.ndt | jq .schema.fields > orig.schema
$ diff -Ndur <( sort orig.schema ) <( sort new.schema ) | less

Testing: manual update of sandbox schemas using new update-schema command. Since this is a mirror struct, it does not impact parser behavior.


This change is Reviewable

coveralls commented 3 years ago

Pull Request Test Coverage Report for Build 6435


Changes Missing Coverage Covered Lines Changed/Added Lines %
schema/ndt_web100.go 0 7 0.0%
<!-- Total: 0 7 0.0% -->
Totals Coverage Status
Change from base Build 6411: -0.04%
Covered Lines: 3471
Relevant Lines: 5558

💛 - Coveralls
stephen-soltesz commented 3 years ago

@gfr10598 PTAL?

gfr10598 commented 3 years ago

Better diff bq show --format=prettyjson mlab-sandbox:base_tables.ndt | jq -c .schema.fields | sed "s/},/},\n/g" > new.schema bq show --format=prettyjson mlab-oti:base_tables.ndt | jq -c .schema.fields | sed "s/},/},\n/g" > orig.schema diff -Ndur <( sort orig.schema ) <( sort new.schema ) | less