anelendata / tap-bigquery

Singer.io tap for extracting data from BigQuery tables
Apache License 2.0
16 stars 33 forks source link

Path to `client_secrets.json` cannot be specified via `config.json` #22

Open kgpayne opened 2 years ago

kgpayne commented 2 years ago

By convention:

The config file contains whatever parameters the Tap needs in order to pull data from the source. Typically this will include the credentials for the API or data source.

Reliance on a required GOOGLE_APPLICATION_CREDENTIALS env var makes this tap an exception, complicating integration with singer-runner tooling.

kgpayne commented 2 years ago

Looks like the way to load google service account credentials from a specified file is the google-auth package:

from google.oauth2 import service_account

credentials = service_account.Credentials.from_service_account_file(
    '/path/to/key.json')

scoped_credentials = credentials.with_scopes(
    ['https://www.googleapis.com/auth/cloud-platform'])

Ideally, to be backwards compatible, we could add a new optional credentials_path config field allowing users to specify a config file location. If none is provided, the tap would fall back on the env var via .default():

import google.auth

credentials, project = google.auth.default()

I don't mind making this change and submitting a PR 👍

kgpayne commented 2 years ago

Credentials can be passed directly to the Client as an argument:

from google.cloud import bigquery
from google.oauth2 import service_account

credentials = service_account.Credentials.from_service_account_file('/path/to/key.json')
client = bigquery.Client(credentials=credentials)
daigotanaka commented 2 years ago

@kgpayne Thanks for bringing it up :) I would appreciate if you can help me by creating a PR. I very much prefer maintaining the backward compatibility. Thx!