Closed shrikant-rajappan closed 3 days ago
ggtisc
commented
3 weeks ago Hi @shrikant-rajappan!
To replicate this issue and detect the bug please share us your terraform configuration code(everything that we need to replicate the issue). You could use examples like this for sensitive data:
project = "my-project" org_id = 1234567890 user = "my-user@my-domain"
shrikant-rajappan
commented
2 weeks ago Here are the two approaches we tried where in the plan succeeds but the apply fails :
1) Here are two resources defined to give different role access to all the queriers in billing_queriers local variable :
resource "google_bigquery_dataset_access" "billing-access" { for_each = local.billing_queriers project = local.cluster_project dataset_id = local.billing_dataset_id role = "roles/bigquery.dataViewer" user_by_email = each.value }
resource "google_project_iam_member" "billing-access-jobuser" { for_each = local.billing_queriers project = local.cluster_project role = "roles/bigquery.jobUser" member = each.value }
2) The other approach we tried is to create a custom role which encompasses all permissions of both roles :
resource "google_project_iam_custom_role" "costing" { role_id = "costing" title = "Dataset permissions for costing data" permissions = [ "bigquery.config.get", "bigquery.datasets.get", "bigquery.datasets.getIamPolicy", "bigquery.jobs.create", "bigquery.models.export", "bigquery.models.getData", "bigquery.models.getMetadata", "bigquery.models.list", "bigquery.routines.get", "bigquery.routines.list", "bigquery.tables.createSnapshot", "bigquery.tables.export", "bigquery.tables.get", "bigquery.tables.getData", "bigquery.tables.getIamPolicy", "bigquery.tables.list", "bigquery.tables.replicateData", "dataform.locations.get", "dataform.locations.list", "dataform.repositories.create", "dataform.repositories.list", "resourcemanager.projects.get" ] }
resource "google_service_account" "costing_sa" { account_id = "costing-sa" display_name = "" }
resource "google_project_iam_member" "sa_custom_role_binding" { for_each = toset([ "roles/costing", ])
project = local.cluster_project role = each.value member = "serviceAccount:${google_service_account.costing_sa.email}" }
resource "google_bigquery_dataset_access" "billing_access" { for_each = local.billing_queriers project = local.cluster_project dataset_id = local.billing_dataset_id role = "roles/costing" user_by_email = each.value }
Please do advise if our approaches are not per provider specification.
Thank you
ggtisc
commented
2 weeks ago The code you are sharing has many variables that we don't have access to, so with a simplified code replacing these variables that we don't have, the code has been tested and no errors were found.
The use of locals, variables and modules is a good practice, but in terms to test I suggest you to replace with static values to see what is happening and then when you are 100% sure that everything works you could replace it with your customized code.
I'm letting you the used code, if you found any other issues please share the complete simplified code with all the necessary to test the functionality of both errors you are facing.
resource "google_bigquery_dataset" "bq_ds_18640" {
dataset_id = "bq_ds_18640"
}
resource "google_bigquery_dataset_access" "bq_ds_access_18640" {
project = "my-project"
dataset_id = google_bigquery_dataset.bq_ds_18640.dataset_id
role = "roles/bigquery.dataViewer"
user_by_email = "my-user@my-domain.com"
}
resource "google_project_iam_member" "project_iam_member_18640" {
project = "my-project"
role = "roles/bigquery.jobUser"
member = "my-user@my-domain.com"
}
resource "google_project_iam_custom_role" "project_iam_cr_18640" {
role_id = "project_iam_cr_18640"
title = "project iam custome role 18640"
permissions = [
"bigquery.config.get",
"bigquery.datasets.get",
"bigquery.datasets.getIamPolicy",
"bigquery.jobs.create",
"bigquery.models.export",
"bigquery.models.getData",
"bigquery.models.getMetadata",
"bigquery.models.list",
"bigquery.routines.get",
"bigquery.routines.list",
"bigquery.tables.createSnapshot",
"bigquery.tables.export",
"bigquery.tables.get",
"bigquery.tables.getData",
"bigquery.tables.getIamPolicy",
"bigquery.tables.list",
"bigquery.tables.replicateData",
"dataform.locations.get",
"dataform.locations.list",
"dataform.repositories.create",
"dataform.repositories.list",
"resourcemanager.projects.get"
]
}
resource "google_service_account" "sa_18640" {
account_id = "sa-18640"
display_name = "service account 18640"
}hi @ggtisc Thank you for testing this out and sharing your code.
I have one callout though - are you using both Custom_role as well as bq.JobUser + Dataviewer resources to provide access to the SA ?
Would it be recommended to add the iam_binding resource as well in either of the two approaches ?
Br, Shrikant R
ggtisc
commented
1 week ago The code I shared with you is based on the code you shared with us and the given permissions was set according to this. On the other hand I can't tell you which is better, because it depends on your requirements and this goes beyond attending bugs, but I can provide you the documentation of terraform registry, API and Google Cloud.
shrikant-rajappan
commented
3 days ago Thank you for your time in looking into the issue. We are now able to make it work with the custom_role option with a subset of permissions.
ggtisc
commented
3 days ago It was a pleasure have a good coding :)
Community Note
Terraform Version & Provider Version(s)
terraform --version Terraform v1.0.11 on linux_amd64
Affected Resource(s)
google_bigquery_dataset_access google_project_iam_member
Terraform Configuration
Debug Output
No response
Expected Behavior
Requirement : Using terraform google provider can the same user account(or service account) be provided with the bigquery.DataViewer and bigquery.JobUser roles. This can easily be done from Google cloud console but we strictly use only IAC for production environments and console access for editing is denied for all.
We tried with two different approaches and in both cases the plan succeeds but apply fails :
1) By using a Custom role with all the role permissions(listed out below) from both the roles
Error received is below :
2) By directly defining two seperate resources each for google_bigquery_dataset_access and google_project_iam_member and assigning the same user account with both roles.
Actually dataViewer was already provided :
This fails too with the error :
Error creating DatasetAccess: googleapi: Error 400: IAM setPolicy failed for Dataset:: Role roles/bigquery.JobUser is not supported for this resource., invalid
IMPORTANT NOTE : WE ARE NOT USING google_bigquery_dataset_access FOR DEFINING THE bigquery.JobUser ROLE INSTEAD WE DEFINE A google_project_iam_member RESOURCE AND GET THE SAME ERROR AS ABOVE
something like below :
Actual Behavior
We really need to understand why we are able to apply the same two roles to a user-account or service-account through the google console however cannot through the IAC which uses terraform google provider. Is this not an aberration ? Arent both the console and provider supposed to act in a similar manner ?
The use case is typical wherein an external service-account might need both these roles to run reporting jobs being run at an external service.
We understand the dichotomy of this requirement that we cannot create the access using the same resource i.e., bigquery_dataset_access for the bq.JobUser and google_project_iam_member for the bq.dataViewer.
Because we believed that creating a custom roles with all these below permissions (this encompasses all permissions in bq.DataViewer and bq.Jobuser) :
What is the approach to follow per the provider specification ?
Especially when we wish to encompass all OR atleast subset of permissions from multiple roles.
Steps to reproduce
terraform applyImportant Factoids
No response
References
No response