search

kestra-io / plugin-gcp

Apache License 2.0
6 stars 10 forks source link

Error reading credential file from environment variable GOOGLE_APPLICATION_CREDENTIALS #376

Closed shrutimantri closed 5 months ago

shrutimantri commented 5 months ago

Expected Behavior

I am trying to replicate the flow in this blueprint: https://kestra.io/blueprints/4-read-a-google-spreadsheet-and-load-it-to-bigquery I am expecting it to run successfully.

Actual Behaviour

I start the kestra instance using docker with the following command:

 % docker run --pull=always --rm -it -p 8080:8080 -p 8081:8081 --user=root --env-file=kestra.env -e GOOGLE_APPLICATION_CREDENTIALS=/Users/shrutimantri/Downloads/<service-account-key-file>.json -v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp kestra/kestra:v0.16.6-full server local

I perform all the steps as mentioned in the blueprint about creating spreadsheet and sharing it with the service account. I run the flow in the blueprint (also copied in the example flow). On flow execution, I get the following error:

2024-05-07 15:40:28.595 Error reading credential file from environment variable GOOGLE_APPLICATION_CREDENTIALS, value '/Users/shrutimantri/Downloads/<service-account-key-file>.json': File does not exist.
2024-05-07 15:40:28.595 File does not exist.
2024-05-07 15:40:28.595 java.io.IOException: Error reading credential file from environment variable GOOGLE_APPLICATION_CREDENTIALS, value '/Users/shrutimantri/Downloads/<service-account-key-file>.json': File does not exist.
    at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentialsUnsynchronized(DefaultCredentialsProvider.java:154)
    at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:120)
    at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:152)
    at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:124)
    at io.kestra.plugin.googleworkspace.AbstractTask.credentials(AbstractTask.java:56)
    at io.kestra.plugin.googleworkspace.sheets.AbstractSheet.connection(AbstractSheet.java:28)
    at io.kestra.plugin.googleworkspace.sheets.Read.run(Read.java:55)
    at io.kestra.plugin.googleworkspace.sheets.Read.run(Read.java:27)
    at io.kestra.core.runners.Worker$WorkerThread.run(Worker.java:769)
Caused by: java.io.IOException: File does not exist.
    at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentialsUnsynchronized(DefaultCredentialsProvider.java:145)
    ... 8 more

The file does exist in my local.

Steps To Reproduce

  1. Perform all the steps mentioned in the blueprint.
  2. Change the spreadsheet Id in the task.
  3. Execute the task.

Environment Information

Example flow

id: gsheet_to_bigquery
namespace: blueprint

tasks:
  - id: read_gsheet
    type: io.kestra.plugin.googleworkspace.sheets.Read
    description: Read data from Google Spreadsheet
    spreadsheetId: 1ybRy9G-sGznXI9GM6FEb0duQyByJbgq4LoYd7oYFr5c
    store: true
    valueRender: FORMATTED_VALUE

  - id: write_csv
    type: io.kestra.plugin.serdes.csv.CsvWriter
    description: Write CSV into Kestra internal storage
    from: "{{ outputs.read_gsheet.uris.Sheet }}"

  - id: load_biqquery
    type: io.kestra.plugin.gcp.bigquery.Load
    description: Load data into BigQuery
    autodetect: true
    csvOptions:
      fieldDelimiter: ","
    destinationTable: kestra-dev.demo.spotify_song_feature
    format: CSV
    from: "{{ outputs.write_csv.uri }}"
shrutimantri commented 5 months ago

I am able to read this if I use the secret that contains the base64 encode value of the service account json file in the serviceAccount field of the io.kestra.plugin.googleworkspace.sheets.Read task. I would like to understand if the GOOGLE_APPLICATION_CREDENTIALS support still exists and I am unable to use it as expected, or if its support is discontinued and the blueprint description requires corresponding change.

shrutimantri commented 5 months ago

This works when I copy the file in the docker container, and use that file path. Thanks @loicmathieu for the help.