Bug 1594111 - Change data source to BigQuery

[BenWu]

Everything seems to be working except esr which doesn't work in prod either so I assume it's a non-issue.

Need to make some updates in the deployed postgres:

UPDATE 
  measure 
SET 
  enabled=false 
WHERE 
  name='browser_shim_usage_blocked'
  OR name='defective_permissions_sql_removed' 
  OR name='permissions_sql_corrupted'
  OR name='slow_script_notice_count' 
  OR name='slow_script_page_count';

UPDATE platform SET telemetry_name = 'Mac' WHERE name = 'mac';

UPDATE platform SET telemetry_name = 'Windows' WHERE name = 'windows';

[wlach]

Also, there are a few test/flake8 errors which should be addressed before this lands.

[whd]

Looks pretty good to me, minus one thing that looks like an oversight. @whd what do you think? In particular, how should we make the service account credentials available to the container? This PR currently specifies a file here:

https://github.com/mozilla/missioncontrol/pull/376/files#diff-fc1d52d91894465e6fac0a9792c0d4f4R98

I'm going to experiment a bit with this today but the standard variable is GOOGLE_APPLICATION_CREDENTIALS which you should use instead.

I'll update when I've got the docker credential mounting ops logic squared away.

[whd]

The ops pieces of this are prepared in https://github.com/mozilla-services/cloudops-deployment/pull/3525/files and https://github.com/mozilla-services/cloudops-infra/pull/1609. Aside from switching to GOOGLE_APPLICATION_CREDENTIALS (after which the SDK will pick up credentials automatically), this looks fine to me.

[BenWu]

@whd I can't see those repos. Should I be able to? For the credentials nothing needs to be done here except assume GOOGLE_APPLICATION_CREDENTIALS is defined right? Should we put a fake one in the env file?

[whd]

@whd I can't see those repos. Should I be able to?

You should, but it looks like you're missing from the "everyone" group which has read on the repos. You don't need to worry about those PRs though.

For the credentials nothing needs to be done here except assume GOOGLE_APPLICATION_CREDENTIALS is defined right?

Correct.

Should we put a fake one in the env file?

I don't have an opinion on this. It will be overridden in the deployment config.

[wlach]

@Ben-Wu unit tests are failing, I think because we're somehow trying to access GCP directly. Other than that, lgtm

[BenWu]

Trying to figure out why it's failing; GCP access should be patched in the test. It works locally in docker so it's confusing.