gluent / goe

GOE: a simple and flexible way to copy data from an Oracle Database to Google BigQuery.
Apache License 2.0
8 stars 2 forks source link

Dataproc Batches support does not cater for a shared VPC #150

Closed nj1973 closed 5 months ago

nj1973 commented 5 months ago

It is possible for GCP users to have networking in project "A" and components such as Dataproc Batches in project "B".

GOE does not cater for that currently, it builds the subnet as described below:

# If set then this variable is used to form a value for Batches subnet of this form:
# projects/${GOOGLE_DATAPROC_PROJECT}/regions/${GOOGLE_DATAPROC_REGION}/subnetworks/${GOOGLE_DATAPROC_BATCHES_SUBNET}
# export GOOGLE_DATAPROC_BATCHES_SUBNET=

Unfortunately this is forcing the GOOGLE_DATAPROC_PROJECT setting into the subnet.

We should change GOOGLE_DATAPROC_BATCHES_SUBNET to access the full subnet address and not only the name part.