search

cloudyr / bigQueryR

R Interface with Google BigQuery
http://code.markedmondson.me/bigQueryR
Other
41 stars 21 forks source link

Google APIs base URL - Private and Restricted Google APIs support #81

Open husseyd opened 1 year ago

husseyd commented 1 year ago

Hi,

Raising an issue here nearly identical to one I just raised for 'bigrquery' at https://github.com/r-dbi/bigrquery/issues/541

I'm working at an org where all traffic to Google needs to use Private or Restricted Google APIs. This involves DNS config with CNAMEs that point services like bigquery.googleapis.com to restricted.googleapis.com. This is for VPC Service Controls, and ensures all traffic stays off of the internet and within client networks/VPCs.

After a lot of headscratching about why this was not working with bigrquery (I received VPC SC errors, and digging into GCP logs saw that requests were coming over the internet from on-prem and thus being blocked Googleside), I dug into the source code on this repo and found the issue.

Generally client libraries and SDKs talk to GCP services at [service].googleapis.com - eg https://bigquery.googleapis.com.

At a number of locations in bigqueryr, the URLs/hosts used to reach BQ are hardcoded to www.googleapis.com EG : R/jobs.R

15 | googleAuthR::gar_api_generator("https://www.googleapis.com/bigquery/v2",
16 | "POST",

and similar at R/listBigQuery.R, R/tables.R, R/datasets.R and others

Due to the generic 'www.googleapis.com', the Google-provided processes for using private/restricted googleapis are blocked when using bigrquery. googleapis.com itself can't really be CNAMED to [restricted || private].googleapis.com as not all services (maps, analytics) are suitable for this.

Anyway this was a long winded way of asking - please consider one of the following options:

  1. Update the BigQuery API URL globally to: 
    https://bigquery.googleapis.com/bigquery/v2/
and
https://bigquery.googleapis.com/upload/bigquery/v2/
  2. Or - provide a mechanism (like a boolean flag) for users to use the 'bigquery.googleapis' url above instead of the current generic 'www.googleapis.com'

Reference:

husseyd commented 1 year ago

Raised PR #82 to address

MarkEdmondson1234 commented 1 year ago

Thank you, this is a scenario that I didn't think to address but makes sense for your use case