search

mozilla / redash-stmo

Extensions to Redash for Mozilla's STMO
https://sql.telemetry.mozilla.org/
Mozilla Public License 2.0
9 stars 9 forks source link

Pull table and column descriptions from BQ into redash metadata #43

Open jklukas opened 5 years ago

jklukas commented 5 years ago

BigQuery has pretty nice support for adding descriptions to tables and columns. We aren't yet making heavy use of it, but we are starting to add some descriptions into JSON schemas in mozilla-pipeline-schemas and should at some point start copying those descriptions to the BigQuery tables where we store raw pings. If we could complete the loop by having those descriptions copied from BQ into redash, it would unlock some very nice discoverability improvements.

For example, we could have the machinery for running ETL queries from bigquery-etl inject a link to the source code in GitHub that generates a table into the description for that table. All sorts of possibilities.

akkomar commented 5 years ago

I suggest to look at Data Catalog first.

DC allows to attach arbitrary metadata to tables/columns, its tag templates would work nicely with Redash for determining which fields to show there. We were thinking about storing some data quality and retention-related metadata there, link to the source code in GitHub is also a good fit for a DC tag.