apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.81k stars 4.23k forks source link

[Feature Request]: Provide option in ReadFromBigQuery to set project from which queries are executed/billed #26904

Open kisssam opened 1 year ago

kisssam commented 1 year ago

What would you like to happen?

The ReadFromBigQuery method should provide an option to set the query execution project:

Currently, the project is inferred from the project running the pipeline or the project which contains the temporary dataset, if the temp dataset is set.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

TSultanov commented 1 year ago

It seems that we need a similar functionality to support ReadFromBigQuery transform for Python examples in Playground.

Our examples need to read the sample data from a different GCP project (clouddataflow-readonly), but use our GCP project for running the job, as we cannot give any permissions for our service account to create jobs in the clouddataflow-readonly one.

This already works in Java (https://play-dev.beam.apache.org/?path=SDK_JAVA_MinimalBigQueryTornadoes&sdk=java).