m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

Support more options for new experiments #1113

Closed stephen-soltesz closed 1 year ago

stephen-soltesz commented 1 year ago

Previously, the update-schema utility only supported legacy datatypes and ndt sidecars. The command assumed that datasets already existed. And the logic was straight forward but very redundant.

This change adds support to update-schema for creating datasets and sidecar tables for alternate experiments. Previous behavior is still available but no longer run by default. Now, the -standard and -legacy options are independent. We typically do not need to update the legacy tables any longer and the template table iteration is time consuming.

Example usage now includes:

# Make standard supported tables.
update-schema -project mlab-sandbox -standard

# Make all datasets and sidecar tables for new foobar experiments.
update-schema -project mlab-sandbox -experiment foobar -sidecars

# Make a single scamper1 table for the wehe experiment.
update-schema -project mlab-sandbox -experiment wehe -datatype scamper1

This change is Reviewable

coveralls commented 1 year ago

Pull Request Test Coverage Report for Build 7460


Files with Coverage Reduction New Missed Lines %
active/active.go 4 88.54%
<!-- Total: 4 -->
Totals Coverage Status
Change from base Build 7444: -0.08%
Covered Lines: 3326
Relevant Lines: 4947

💛 - Coveralls
stephen-soltesz commented 1 year ago

Thank you!

stephen-soltesz commented 1 year ago

Part of: