ADR - Serve data from BQ

[edugfilho]

Accepting feedback on this ADR, with code attached.

Moving this forward we probably need to coordinate with DSRE to make sure GLAM has proper BQ permissions.

[scholtzan]

I'd recommend that whatever service account is used for accessing BigQuery only has access to the tables that should be exposed. There seem to be a bunch of tables in glam_etl that might contain data we'd not want users to get access to if somehow the account credentials are leaked or the API accidentally allows access to them somehow.

[edugfilho]

I'd recommend that whatever service account is used for accessing BigQuery only has access to the tables that should be exposed. There seem to be a bunch of tables in glam_etl that might contain data we'd not want users to get access to if somehow the account credentials are leaked or the API accidentally allows access to them somehow.

I agree but I'd like to strike some middle ground there in the likely case we'll need to add more tables to the glam_etl dataset for a new product, for example. In that case I wouldn't like to get DSRE involved, so if BQ supports asterisk for access control that may be a good idea. Or even creating a whole new dataset only for prod tables and allowing GLAM to that dataset only.

[edugfilho]

I initially intended to create a feature flag for this and select a number of users to rollout this to, but it proved too complicated since a full rollback is as simple as switching a flag and re-deploying. So I intend to deploy this in its current state, which means the moment it hits dev all requests in that environment will be fetching data from the BigQuery prod tables, and we can test it out. If things look good then we do stage, then prod.

Rollback is as simple as switching the data_source string in glam/api/views.py back to Postgres and re-deploying.

So my plan is to deploy this to dev on Jan 29.

@mikaeld requesting your input for the above deployment/rollback plan, because if we want to rollback prod it will involve a prod re-deployment. But just button pressing as usual.

Also, we want GLAM to have read permissions to the following tables and eventually other tables (e.g. in case of new products getting added), while taking into account Anna's comment below. I was thinking of a dataset-level access, so it doesn't involve DSRE adding access every time for a new table. Would you have a different recommendation?

I'd recommend that whatever service account is used for accessing BigQuery only has access to the tables that should be exposed. There seem to be a bunch of tables in glam_etl that might contain data we'd not want users to get access to if somehow the account credentials are leaked or the API accidentally allows access to them somehow.

Here are the tables to which GLAM currently needs access:

• moz-fx-data-shared-prod.telemetry.buildhub2
• moz-fx-data-shared-prod.telemetry_derived.latest_versions_v2

Desktop (legacy):

• moz-fx-data-glam-prod-fca7.glam_etl.glam_desktop_release_aggregates_v1
• moz-fx-data-glam-prod-fca7.glam_etl.glam_desktop_nightly_aggregates_v1
• moz-fx-data-glam-prod-fca7.glam_etl.glam_desktop_beta_aggregates_v1 Desktop(Glean)
• moz-fx-data-glam-prod-fca7.glam_etl.glam_fog_release_aggregates_v1
• moz-fx-data-glam-prod-fca7.glam_etl.glam_fog_nightly_aggregates_v1
• moz-fx-data-glam-prod-fca7.glam_etl.glam_fog_beta_aggregates_v1

Android:

• moz-fx-data-glam-prod-fca7.glam_etl.glam_fenix_release_aggregates_v1
• moz-fx-data-glam-prod-fca7.glam_etl.glam_fenix_nightly_aggregates_v1
• moz-fx-data-glam-prod-fca7.glam_etl.glam_fenix_beta_aggregates_v1

[mikaeld]

Table-level access is preferable to dataset-level. You can use the existing workgroup:dataops-managed/glam in bqetl metadata for all the tables you listed.