Support for BigQuery authentication using environment credentials: GOOGLE_APPLICATION_CREDENTIALS

[danieldeleo]

Avoiding downloaded private keys (if possible) is a security best practice. Running dataform in GCP gives you the option of authenticating without having to download private keys.

[BenBirt]

Hi @danieldeleo , thanks for the report.

However, I don't think I follow. While we do run Dataform in GCP, we run it under our own service account(s), in our own GCP project(s). We wouldn't be able to use the GOOGLE_APPLICATION_CREDENTIALS approach to authenticate as user-owned service accounts.

Unless you mean something like running the CLI in a GCP environment?

[danieldeleo]

Ah yes I should have been more specific. This is in regards to the CLI in a GCP environment, or also on a developer's laptop.

As a developer using the cli on my laptop, I'd rather run "gcloud auth application-default login" and have dataform pick up on those credentials instead of downloading a private key onto my laptop.

[BenBirt]

Makes sense, thanks!

[jonasholtkamp]

Is there any chance of supporting Application Default Credentials? We'd love to see ADC support to be able to let Dataform run "credential-less" e.g. inside Kubernetes Engine.

[andres-lowrie]

So I just installed the latest version with ADC without issues:

❮ jq '.' package.json                                                                                                                                                                                                                                                                             <<<
{
  "dependencies": {
    "@dataform/core": "1.19.0",
    "@dataform/cli": "1.19.0"
  }
}

❯ npx dataform init-creds bigquery

[1] US (default)
[2] EU
[3] other

Enter the location of your datasets [1, 2, 3]: 1

[1] ADC (default)
[2] JSON Key

Do you wish to use Application Default Credentials or JSON Key [1/2]: 1
Enter your billing project ID:

worked like a charm when actually using it too

❯ npx dataform run
Compiling...

Compiled successfully.

Running...

Dataset created:  dataform.example [view]

I think this issue is resolved maybe?

[jonasholtkamp]

Yeah, 1.19 added that feature: https://github.com/dataform-co/dataform/releases/tag/1.19.0

[danieldeleo]

Fixed by https://github.com/dataform-co/dataform/pull/1210 Available starting with v1.19 https://github.com/dataform-co/dataform/releases/tag/1.19.0