Permissions issue with google_logging_billing_account_sink

[choppedpork]

Terraform Version

Terraform v0.11.7 provider.google: version = "~> 1.15"

Affected Resource(s)

• google_logging_billing_account_sink

Terraform Configuration Files

resource "google_logging_billing_account_sink" "billing" {
  name = "${var.project_id}-billing"
  billing_account = "${var.billing_id}"
  destination = "storage.googleapis.com/${google_storage_bucket.billing.name}/${var.billing_prefix}"

  depends_on = [ "google_project_service.billing-api", "google_project_service.storage-api" ]
}

Expected Behavior

🎉

Actual Behavior

* google_logging_billing_account_sink.billing: googleapi: Error 403: The caller does not have permission, forbidden

Steps to Reproduce

1. terraform apply

Hello!

I've just started my GCP adventure with terraform so this might be a case of missing or misunderstood documentation rather than a bug. I'm using terraform with my gcloud credentials here rather than a service account as I am bootstrapping a project from scratch - I'm also assuming I wouldn't be able to create a billing sink with a service account as it belongs to the billing account rather than the project (please correct me if I'm wrong).

My user account has Billing Account Administrator privileges for my billing account. The strangest thing is that the 403 error is very much straightforward but I can create a billing file sink into the same bucket using the UI perfectly fine. I can provide debug output but it's not particularly interesting - it's just a 403 error in response to terraform's POST /v2/billingAccounts/123456-123456-123456/sinks?alt=json&uniqueWriterIdentity=true. The APIs currently enabled for my project are:

cloudbilling.googleapis.com          Cloud Billing API
cloudresourcemanager.googleapis.com  Cloud Resource Manager API
compute.googleapis.com               Compute Engine API
oslogin.googleapis.com               Cloud OS Login API
storage-api.googleapis.com           Google Cloud Storage JSON API

Thanks!

[paddycarver]

I'm also assuming I wouldn't be able to create a billing sink with a service account as it belongs to the billing account rather than the project (please correct me if I'm wrong).

You should be able to! Just give the service account the billing administrator privileges. :)

I can provide debug output but it's not particularly interesting - it's just a 403 error in response to terraform's POST /v2/billingAccounts/123456-123456-123456/sinks?alt=json&uniqueWriterIdentity=true

I'd actually love to see the debug output, because I get a feeling this may be a weird issue related to using the application default credentials (e.g. a user account).

[choppedpork]

Thanks @paddycarver - here's the debug log: https://gist.github.com/choppedpork/236d70ab099ea738b7a728fcbce54f2a I've hopefully done an alright job of removing any PII :)

Thanks for the service account tip - I shall give that a shot too!

[nat-henderson]

Well, it's certainly not the application default credentials issue! I think your user account might not have permission to apply the billing account to the project. Are you successful in doing that in the Cloud Console?

[choppedpork]

Yes I am, though this is me assuming I'm doing the exact same thing :) Here's what I'm trying to achieve (I ended setting this up via the Cloud Console):

I've been able to successfully set up a google_project_usage_export_bucket resource but I understand it uses the storage API rather than the billing API which probably makes all the difference here.

[paddycarver]

This is a very silly question, but have you tried turning on logging.googleapis.com? That's the API endpoint being hit, after all.

[Tony-Proum]

Hello,

I encounter the same issue using a service account with Billing Administrator role. The service account have access to the destination bucket and logging.googleapis.com is enable but still have this error : google_logging_billing_account_sink.billig-account-sink: googleapi: Error 403: The caller does not have permission, forbidden

@choppedpork did you find any workaround ?

[choppedpork]

@paddycarver not a silly question at all - I wasn't aware of this and while I haven't checked if it got implicitly enabled previously (I'm assuming it did when I've done this via the console) I did add it to the module I'm writing. Unfortunately this didn't help :(

@Tony-Proum my shameful workaround was adding a TODO line to my module and setting this up via the console, for the time being (I'm setting this up for Cloudhealth which only requires me to do this once per organisation rather than project so it's not really a blocker for me at this stage).

[Tony-Proum]

Oh, I were able to solve this issue reviewing all roles used by the service account. Using debug mode, I found several call that require different roles : Billing Account Administrator and roles/logging.configWriter are the relevant ones in my case.

[choppedpork]

That's interesting! I guess this might mean that billing account sinks are one of the features that don't work correctly with gcloud auth login credentials - as per https://www.terraform.io/docs/providers/google/index.html

not guaranteed to work for all APIs

Interestingly I've not encountered any other issues using those credentials so far but I'll try to retest with a service account. I'm wondering if it would make sense to refer to this from google_logging_billing_account_sink documentation page if this turns out to be the case?

[chrisst]

FWIW i've found that the generic 403 error you are encountering is usually related to IAM roles not being set up correctly. It's also important to note that there permissions are managed in different places depending on what the permission is attached to.

You should check to make sure you have the correct permissions set on your billing account and not necessarily at a project level. So even if you see 'logging Admin' set in the IAM section of the cloud console, this doesn't affect whether you have permission to the logs for the billing account. Billing account permissions can be managed under the Billing section of the cloud console on the far right [see image].

[choppedpork]

Thank you so much @chrisst!! That was it! Apply successful :)

Looking at the docs, there's already an explicit note about the permissions required - do you think it would make sense to make it explicit that the permission is a billing account permission rather than IAM permission?

I still find it very odd that it worked fine using the console but this has nothing to do with the terraform provider and is just a GCP oddity :)

[chrisst]

I think it's a difference of how you are granting auth to the credentials that Terraform is trying to use. Also to be a bit more precise, billing accounts are still using IAM permissions it's just that the IAM tab in the console refers only to a subset of all the IAM permissions (IIRC you can access project and organization IAM policies there).

[choppedpork]

Thanks @chrisst, that makes sense.

Thanks again for pointing me in the right direction and for the docs update!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!