jamiet-msm
commented
1 year ago This is tricky because its the underlying API that (IMO) is at fault here. Nevertheless, it would be handy if the TF provider could mitigate this.
We first mitigated this by putting a big warning at the top of any TF files that manipulate BigQuery dataset IAM:
#_________ _______ _______ _______ _______ _________ _______ _ _________
#\__ __/( )( ____ )( ___ )( ____ )\__ __/( ___ )( ( /|\__ __/
# ) ( | () () || ( )|| ( ) || ( )| ) ( | ( ) || \ ( | ) (
# | | | || || || (____)|| | | || (____)| | | | (___) || \ | | | |
# | | | |(_)| || _____)| | | || __) | | | ___ || (\ \) | | |
# | | | | | || ( | | | || (\ ( | | | ( ) || | \ | | |
#___) (___| ) ( || ) | (___) || ) \ \__ | | | ) ( || ) \ | | |
#\_______/|/ \||/ (_______)|/ \__/ )_( |/ \||/ )_) )_(
# Please never use
# google_bigquery_dataset_iam_member
# google_bigquery_dataset_iam_policy
# google_bigquery_dataset_iam_binding
# https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/bigquery_dataset_iam
# to grant access to this dataset. We create authorised views on top of that dataset and any use of
# those resources will clobber those authorised views.This obviously isn't failsafe though so we now have a check in our CI pipeline that greps the codebase and throws an error if it finds "google_bigquery_dataset_iam_member". We use GitHub Actions so we wrote our own action to do this:
name: 'prevent use of BigQuery dataset IAM'
description: 'We make extensive use of authorised views and authorised datasets hence we want to avoid the use of the terraform resources that can cause them to be removed.'
runs:
using: "composite"
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Check no google_bigquery_dataset_iam_* resources are being used
shell: bash
run: |
RED_BACKGROUND='\033[41m'
NC='\033[0m' # No colour
re='\"google_bigquery_dataset_iam_member\"|\"google_bigquery_dataset_iam_binding\"|\"google_bigquery_dataset_iam_policy\"'
if [ "$(grep -rnI -E ${re} . | wc -l)" -gt 0 ]; then
# grep returns non-zero exit code if it finds zero matches so the next line would cause CI job failure. Hence I'm running it once its already been decided some matches have been found
INVALID_RESOURCES=$(grep -rnI -E ${re} .)
echo "$INVALID_RESOURCES"
echo -e "${RED_BACKGROUND}"There are "$(echo "$INVALID_RESOURCES" | wc -l)" occurrences of google_bigquery_dataset_iam_member, google_bigquery_dataset_iam_binding or google_bigquery_dataset_iam_policy"${NC}"
echo -e "Please read https://link/to/internal/jira/JIRA-X to understand why this is not allowed"
exit 1
fiIts a bit makeshift, but it works
killthekitten
rileykarson
melinath
walterjking
wj-chen
Hello,
I want to discuss the issue #10156 that has been locked.
I have had exactly the same problem and it has been tedious to find the culprit since the Terraform config is maintained by different people in the Team. I was not aware that google_bigquery_datasetiam would delete all the access created by google_bigquery_dataset_access. Indeed, there is a warning on the official documentation page but maybe it's worth considering to integrate that warning in terraform code, while deploying google_bigquery_datasetiam ?
Because, having a resource (dataset_iam) deleting another resource (dataset_access) without any warning on the CLI is not what people would expect as behaviour ?
b/321924266