Unable to add storage bucket to project - billing cannot be assigned

derekschauland commented 5 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment
If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Terraform v0.11.13

Affected Resource(s)

google_project

Terraform Configuration Files

resource "google_project" "Project" { name = "${var.project_name}"

project_id = "${local.project_id}"

folder_id = "${data.google_folder.test.name}"

billing_account = "${var.gcp_billing_account}" auto_create_network = false }

resource "google_project_services" "services" { project = "${google_project.Project.number}"

services = [ "compute.googleapis.com", "cloudresourcemanager.googleapis.com", "oslogin.googleapis.com", "iam.googleapis.com", "cloudbilling.googleapis.com", "storage-component.googleapis.com", "admin.googleapis.com", "appengine.googleapis.com", ]

provisioner "local-exec" { command = "sleep 30" }

depends_on = ["google_project.Project"] }

resource "google_storage_bucket" "default_project_bucket" { name = "${var.project_name}-general-storage" project = "${google_project.Project.name}" storage_class = "MULTI_REGIONAL" location = "US"

logging { log_bucket = "${"${google_project.Project.name}"}-logs" } }

Debug Output

Panic Output

Expected Behavior

The project should be created with billing enabled and a storage account, but it seems permissions or other are getting in the way. The account running terraform is a service account which has access across the project and should be able to complete this task.

Actual Behavior

google_project.Project: 1 error(s) occurred:
google_project.Project: Error setting billing account "removed-for-safety" for project "projects/project-testing-4324573": googleapi: Error 403: The caller does not have permission, forbidden

Steps to Reproduce

terraform apply

Important Factoids

References

0000

emilymye commented 5 years ago

From the GCP docs

To change the billing account for an existing project, you must be an owner on the project and a billing administrator on the destination billing account.

It sounds like your TF service account has project owner, but does it have the Project Billing Manager role on the project?

derekschauland commented 5 years ago

The account used to execute terraform has project billing manager (inherited from org) and owner permission on the project being created. Image of permissions for terraform running acct below

terraform-iam

derekschauland commented 5 years ago

I tried assigning billing.ProjectManager to the service account for the project and it does not seem to like that either

emilymye commented 5 years ago

Can you run with debug logs on (TF_LOG=DEBUG) to confirm what project the billing account is being set for and what API endpoint the call is failing for? Do both the project and the billing account belong to your organization? Based on docs you might also need to add Billing Account User on the billing account. To be more clear (for myself mostly), it looks like:

Billing Account User --> permission set on billing account so you can apply it Project Billing Manager --> permission set on project (or parent org) so you can apply billing accounts to it

derekschauland commented 5 years ago

@emilymye thanks for the guidance - the Billing Account User permission on the billing account appears to have moved me forward. It claims that the billing account is absent (new error):

google_storage_bucket.default_project_bucket: 1 error(s) occurred:
google_storage_bucket.default_project_bucket: googleapi: Error 403: The project to be billed is associated with an absent billing account., accountDisabled
google_project_services.services: Error running command 'sleep 30': exit status 1. Output: 'sleep' is not recognized as an internal or external command, operable program or batch file.

Looking through the debug logs - the billing_account appears to be correct but also says that the project to be billed is associated with an absent billing account.

emilymye commented 5 years ago

Does the Terraform service account have permissions to get the billing account? Having the debug logs would help us.

derekschauland commented 5 years ago

It does have access to the billing account

I will see if I can generate a set of debug logs in the morning

Sent from my iPhone

On Apr 29, 2019, at 1:25 PM, emily notifications@github.com<mailto:notifications@github.com> wrote:

Does the Terraform service account have permissions to get the billing account? Having the debug logs would help us.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/terraform-providers/terraform-provider-google/issues/3429#issuecomment-487690529, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEWWFIOVIYG4OQI6K7M2M3TPS44R3ANCNFSM4HF77V3A.

derekschauland commented 5 years ago

the debug log is attached tf-logs.txt

kristofgiber commented 5 years ago

Any solution to this error message? I also get the same 403 (absent billing account) error after trying to create bucket with gsutil after having run gcloud auth activate-service-account --project tensorflow-gcloud --key-file=$KEYFILE. I too have the mentioned roles already set for the service account and the project. I am on free trial but my billing settings seem to be ok in the console. I am running it inside a docker container.

derekschauland commented 5 years ago

I believe this is still open and Google is looking in to it

Get Outlook for iOShttps://aka.ms/o0ukef

From: Kristof Giber notifications@github.com Sent: Saturday, May 25, 2019 12:32:39 PM To: terraform-providers/terraform-provider-google Cc: Derek Schauland; Author Subject: Re: [terraform-providers/terraform-provider-google] Unable to add storage bucket to project - billing cannot be assigned (#3429)

Any solution to this error message? I also get the same 403 (absent billing account) error after trying to create bucket with gsutil after having run gcloud auth activate-service-account --project tensorflow-gcloud --key-file=$KEYFILE. I too have the mentioned roles already set for the service account and the project. I am on free trial but my billing settings seem to be ok in the console. I am running it inside a docker container.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/terraform-providers/terraform-provider-google/issues/3429?email_source=notifications&email_token=AEWWFIJ4O7T4D6LRR4JESFLPXFZ3PA5CNFSM4HF77V3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWHV62Y#issuecomment-495935339, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEWWFILAXOESWHYUVXK6GELPXFZ3PANCNFSM4HF77V3A.

kristofgiber commented 5 years ago

On my end it is resolved if I re-initialize gcloud with gcloud init and just manually confirm the same account and project names that were already set by the gcloud auth activate-service-account --project tensorflow-gcloud --key-file=$KEYFILE command beforehand. As the documentation says the activation command should be called instead of the gcloud init command and we are not supposed to use the manual initialization for service accounts, I assume the need for re-initializing may be a bug indeed. It is a workaround to me for now though.

derekschauland commented 5 years ago

Wouldn't the service account being used to create the resources (using Terraform) already be active against a project that it just created? Since Account A created the resource, I would expect it is active in the project if further resources will be created.... vs needing to activate again, or activate a different account after creation... I am hoping to use one account to stand up any and all resources needed

kristofgiber commented 5 years ago

My issue was with google cloud running inside a Docker container I am not using terraform and I've created all resources from GCE console not terraform. But otherwise yes I agree, calling glcoud init shouldn't make a difference if the service account authorization was bug free, to me it shows the account is set in the gcloud configurations to the service account and the active project is set to the right project. By calling gcloud init I just have to go through the same config questions again and confirm everything is unchanged yet once this seemingly unnecessary step is done, the 'The project to be billed is associated with an absent billing account' error goes away in my case. I guess it's a Gcloud bug.

derekschauland commented 5 years ago

Just wanted to check in and see if there was any movement on this issue and/or if anything was needed from this end

derekschauland commented 5 years ago

Just wanted to check in and see if there was any movement on this issue and/or if anything was needed from this end

kristofgiber commented 5 years ago

On my issue Im pretty sure it's related to gcloud but not terraform and Ive worked around it with the aforementioned gcloud init so it's ok

DanielJonesEB commented 5 years ago

I'm getting this repeatably, using code that worked a few days ago.

This CI script executes some Terraform that creates a project and a bucket.

I get the following error:

2019/08/11 08:22:40 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error occurred:
    * google_storage_bucket.ci: googleapi: Error 403: The project to be billed is associated with an absent billing account., accountDisabled

The billing account is definitely active. The same code worked last week (I know this, as it's been running in CI).

Here's the user-agent header from debug logs:

google-api-go-client/0.5 Terraform/0.12.4 (+https://www.terraform.io) terraform-provider-google/2.12.0

Running the script interactively and sticking a gcloud init in didn't help.

DanielJonesEB commented 5 years ago

I think this is a problem at Google's end.

My service account originally had:

Billing Account User
Billing Account Viewer
Folder Viewer
Project Creator

So, in desperation I added Billing Account Administrator. Run the CI job, and it worked.

I then removed Billing Account Administrator, and added Project Billing Manager. This did not work.

I removed Project Billing Manager, added Billing Account Administrator back again, and it did not work. I tried this several times.

I then added Project Billing Manager back in (so now the service account had both this and Billing Account Administrator) and it just worked.

Run the same thing again, and it did not work.

So the problem is either:

intermittent, and my changes were coincidental
somehow affected by changing service account permission state

If I had to hazard a guess, I'd suggest that maybe there's some internal eventual consistency thing going on at Google, and that maybe the provider should allow some amount of retrying in the event of a 403 on this particular call.

DanielJonesEB commented 5 years ago

I added a sleep into the project create step, and now it works every time:

https://github.com/EngineerBetter/concourse-gcp-tf-bootstrap/blob/master/tf/pipeline-reqs.tf#L30-L32

I'm pretty sure that adding sleeps to Terraform means that I'll be going to hell.

derekschauland commented 5 years ago

Hmmm the sleep code is interesting. I still think there are issues with the gcp terraform code for this, but will take a run at a sleep as well

Get Outlook for iOShttps://aka.ms/o0ukef

From: Daniel Jones notifications@github.com Sent: Sunday, August 11, 2019 4:45:12 AM To: terraform-providers/terraform-provider-google terraform-provider-google@noreply.github.com Cc: Derek Schauland derek@derekschauland.com; Author author@noreply.github.com Subject: Re: [terraform-providers/terraform-provider-google] Unable to add storage bucket to project - billing cannot be assigned (#3429)

I added a sleep into the project create step, and now it works every time:

https://github.com/EngineerBetter/concourse-gcp-tf-bootstrap/blob/master/tf/pipeline-reqs.tf#L30-L32

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/terraform-providers/terraform-provider-google/issues/3429?email_source=notifications&email_token=AEWWFIMEWMER23Z25W4LK33QD7NSRA5CNFSM4HF77V3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4A5QIQ#issuecomment-520214562, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEWWFIMAPVP5HNT4PKHD6U3QD7NSRANCNFSM4HF77V3A.

derekschauland commented 5 years ago

I tried a sleep in the project creation file and it produced the error below:

google_project_services.services (local-exec): 'sleep' is not recognized as an internal or external command, google_project_services.services (local-exec): operable program or batch file.

Error: Error running command 'sleep 30': exit status 1. Output: 'sleep' is not recognized as an internal or external command, operable program or batch file.

Error: googleapi: Error 403: The project to be billed is associated with an absent billing account., accountDisabled

on gcp-project.tf line 39, in resource "google_storage_bucket" "default_project_bucket": 39: resource "google_storage_bucket" "default_project_bucket" {

u2g-tg commented 5 years ago

Looks like you're running on Windows @derekschauland

https://serverfault.com/questions/432322/how-to-sleep-in-a-batch-file

Maybe timeout is for you?

judika03 commented 4 years ago

I'm same issue about that, How to solved to create new bucket?

judika03 commented 4 years ago

Error uploading object redis_test.sh: googleapi: Error 403: The project to be billed is associated with an absent billing account., accountDisabled.

I have same issue, I already to adding sleep to create bucket. but same issue. How to solved?

venkykuberan commented 4 years ago

@derekschauland and others, If you are creating a project and adding a storage bucket for the project in the same config, you would hit eventually consistency issue.

1) Project API confirms project is created successfully and billing account association is good. 2) If Storage API didn't get the propagated data, it throws an absent billing account error. It happens intermittently, its not a provider/terraform issue.

A work item has been created to add retry mechanism for storage API calls and its been tracked here.

In the mean time having sleep command on project creation is best bet to get around the "billing account error' on storage API call.

Sample Config

resource "google_project" "demo_project" {
    name = "testing-project"
    project_id = local.project_id
    org_id = local.org_id
    billing_account = var.billing_account_id
    provisioner "local-exec" {
         command = "sleep 10"
     }
}

resource "google_storage_bucket" "demo_bucket" {
    name     = google_project.demo_project.project_id
    force_destroy = true
    depends_on = [google_project.demo_project]
}

As we are tracking this on another issue I am closing this one.

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

hashicorp / terraform-provider-google