search

benthosdev / benthos

Fancy stream processing made operationally mundane
https://www.benthos.dev
MIT License
7.68k stars 752 forks source link

Permit Bigquery output to create jobs in a different project #2532

Open Roviluca opened 4 weeks ago

Roviluca commented 4 weeks ago

Bigquery allows to have 3 different projects involved when querying Bigquery:

  1. The project where the service account is created
  2. The project where the job is created
  3. The project where the data is stored.

The current implementation does not allow this flexibility. if the project is set the job will be created in the Dataset's project, otherwise the client tries to use bigquery.DetectProjectID that returns the service account's project and searches the dataset in there.

With this PR I added the possibility to set JobProjectID so that users can have the flexibility to set the 3 parameters with different values.

Roviluca commented 3 weeks ago

Hello @Jeffail does this PR make sense or do you prefer me do approach this in another way?