Automate publishing measurement-lab passthrough views

[stephen-soltesz]

Problem:

• Today, we maintain explicit SQL configurations for pass-through views between mlab-oti and measurement-lab.
• Once created, we separately configure public visibility of these pass-through views to discuss@ users via ACLs.

Idea:

• Rather than having the cumbersome configurations we maintain in etl-schema for pass-through views - what if we had a simple daemon in the data-processing cluster that, say, once an hour looked for some role or label or other attribute we could add to tables in mlab-oti (maybe with the label value equal to the target name) and this daemon would simply create the public measurement-lab dataset & view automatically?

This idea would be an alternative to fully Terraform managed configurations. Arguably, since this is public facing, we might want more fluid changes. But, that would limit who could mange table selection and publication.

[stephen-soltesz]

BigQuery tables do have a key/value label mechanism available.

E.g. mlab-oti.ndt.ndt7 could have a label like: passthrough:measurement-lab.ndt.ndt7 and the daemon could do the rest.