search

gluent / goe

GOE: a simple and flexible way to copy data from an Oracle Database to Google BigQuery.
Apache License 2.0
8 stars 2 forks source link

feat: Switch GOOGLE_DATAPROC_BATCHES_SUBNET to require a full URI #151

Closed nj1973 closed 5 months ago

nj1973 commented 5 months ago

This PR relaxes the format of the subnet we pass to gcloud dataproc batches submit, before this PR the code built a subnet string from three inputs as described below:

# If set then this variable is used to form a value for Batches subnet of this form:
# projects/${GOOGLE_DATAPROC_PROJECT}/regions/${GOOGLE_DATAPROC_REGION}/subnetworks/${GOOGLE_DATAPROC_BATCHES_SUBNET}

This was forcing the GOOGLE_DATAPROC_PROJECT setting into the subnet and there was no way to deviate from the project hosting Dataproc.

This PR removes the fixed formatting and passes the value from offload.env through to gcloud which provides extra flexibility when a shared VPC is used.