snowplow / sql-runner

Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake
http://snowplowanalytics.com
Apache License 2.0
80 stars 14 forks source link

BigQuery: Explicitly set Client.Location from the region playbook variable #173

Closed colmsnowplow closed 3 years ago

colmsnowplow commented 3 years ago

BQ normally does a good job of inferring the dataset location to use, so it seems we never previously explicitly set it.

However, with the implementation of multi-statement queries, I have discovered a fun quirk whereby sometimes it just doesn't do it, and defaults to 'US'. So far I've only seen this behaviour when there's more than one DROP TABLE statement in one script.

It seems BQ Web UI does the same thing.

We can mitigate by setting the client's location via the 'region' variable. Looks like if this is left empty or not set it will still infer the region correctly unless it comes across this specific case.