1-org step Terraform Plan fails Error 403: The caller does not have permission

sbadakhc commented 4 years ago

Fixed previous permissions issues and now getting error 403.

cft-seed-29c2 master 1-org]$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

module.org_billing_logs.module.project-factory.module.gcloud_deprivilege.random_id.cache: Refreshing state... [id=fP4hHQ]
module.seed_bootstrap.random_id.suffix: Refreshing state... [id=AAU]
module.seed_bootstrap.module.seed_project.module.project-factory.random_id.random_project_id_suffix: Refreshing state... [id=KcI]
module.org_shared_vpc_nonprod.module.project-factory.module.gcloud_delete.random_id.cache: Refreshing state... [id=WUW7Fg]
module.org_shared_vpc_prod.module.project-factory.module.gcloud_delete.random_id.cache: Refreshing state... [id=oN_jhQ]
module.org_billing_logs.module.project-factory.module.gcloud_disable.random_id.cache: Refreshing state... [id=PipnnQ]
module.cloudbuild_bootstrap.module.cloudbuild_project.module.project-factory.random_id.random_project_id_suffix: Refreshing state... [id=U9M]
module.org_shared_vpc_nonprod.module.project-factory.module.gcloud_deprivilege.random_id.cache: Refreshing state... [id=fufKTw]
module.org_audit_logs.module.project-factory.module.gcloud_delete.random_id.cache: Refreshing state... [id=s8ntgg]
module.org_shared_vpc_nonprod.module.project-factory.random_id.random_project_id_suffix: Refreshing state... [id=pUI]
module.org_monitoring_nonprod.module.project-factory.random_id.random_project_id_suffix: Refreshing state... [id=3Xs]
module.org_shared_vpc_nonprod.module.project-factory.module.gcloud_disable.random_id.cache: Refreshing state... [id=-NxuzA]
module.org_billing_logs.module.project-factory.module.gcloud_delete.random_id.cache: Refreshing state... [id=HvN25g]
module.org_monitoring_prod.module.project-factory.module.gcloud_delete.random_id.cache: Refreshing state... [id=7BJwOA]
module.org_monitoring_nonprod.module.project-factory.module.gcloud_deprivilege.random_id.cache: Refreshing state... [id=m6sk3A]
module.org_monitoring_prod.module.project-factory.random_id.random_project_id_suffix: Refreshing state... [id=6pg]
module.org_monitoring_prod.module.project-factory.module.gcloud_deprivilege.random_id.cache: Refreshing state... [id=XEn31w]
module.org_audit_logs.module.project-factory.random_id.random_project_id_suffix: Refreshing state... [id=nLc]
module.org_monitoring_prod.module.project-factory.module.gcloud_disable.random_id.cache: Refreshing state... [id=eqG6Kg]
module.org_audit_logs.module.project-factory.module.gcloud_disable.random_id.cache: Refreshing state... [id=jgWAgQ]
module.org_monitoring_nonprod.module.project-factory.module.gcloud_disable.random_id.cache: Refreshing state... [id=lcZ3VA]
module.org_shared_vpc_prod.module.project-factory.random_id.random_project_id_suffix: Refreshing state... [id=XSM]
module.org_monitoring_nonprod.module.project-factory.module.gcloud_delete.random_id.cache: Refreshing state... [id=aOLbxA]
module.org_billing_logs.module.project-factory.random_id.random_project_id_suffix: Refreshing state... [id=BM8]
module.org_shared_vpc_prod.module.project-factory.module.gcloud_deprivilege.random_id.cache: Refreshing state... [id=7mRcUQ]
module.org_audit_logs.module.project-factory.module.gcloud_deprivilege.random_id.cache: Refreshing state... [id=tIIVKA]
module.cloudbuild_bootstrap.module.cloudbuild_project.module.project-factory.null_resource.preconditions: Refreshing state... [id=5082551985136791431]
module.seed_bootstrap.module.seed_project.module.project-factory.null_resource.preconditions: Refreshing state... [id=8989348742835411969]
module.org_shared_vpc_prod.module.project-factory.module.gcloud_disable.random_id.cache: Refreshing state... [id=OAKzEQ]
data.google_service_account_access_token.default: Refreshing state...

Error: googleapi: Error 403: The caller does not have permission, forbidden

  on providers.tf line 30, in data "google_service_account_access_token" "default":
  30: data "google_service_account_access_token" "default" {

Current permissions associated with my terrform service account:

Billing Account User
Cloud Build Editor
Cloud KMS Admin
Compute Network Admin
Compute Shared VPC Admin
Security Admin
Service Account Admin
Logs Configuration Writer
Organization Policy Administrator
Folder Admin
Organization Viewer
Project Creator
Source Repository Administrator
Storage Admin

rjerrems commented 4 years ago

Hi @sbadakhc - are you running this as your user account or the terraform service account using a key?

By default, if you have not modified the provider configuration, only users in the organization admins group will be able to run the code, which impersonates the service account and generates a short lived credential here https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/1-org/providers.tf#L30

sbadakhc commented 4 years ago

Hi @rjerrems I haven't modified the provider configuration. I can confirm that my user account is a member of the gcp-organization-admins group. I also have the following permissions at the organizational leve for my user accountl:

Support Account
AdministratorOwner
FolderAdmin
OrganizationAdministrator
Quota Viewer

In the terraform.tfvars I have declared the terrform service account created in the 0-bootstrap stage as described in the README.md with the permissions listed in my previous post. I guess this should be working in the way you described but I'm not able authorise. Where is the temporary token kept? Perhaps that is something I can check?

Thankss, Sal.

ericyz commented 4 years ago

Hi @sbadakhc , I think you are missing iam.serviceAccountTokenCreator permission to the terraform service account, which is used to impersonate the terraform service account.

Could you please confirm this permission on the Google IAM service account and also describe how did you run the 0-boostrap?

sbadakhc commented 4 years ago

Hi @ericyz. I did successfully run terraform in the 0-bootstrap step of the example foundation repo before attempting to run it in 1-org. I can confirm my user account was missing the service account token creator role. I will feedback shortly. Thanks for the help chaps.

ericyz commented 4 years ago

@sbadakhc , thanks for checking. If you are running on on your local environment, it might be worth running "gcloud auth application-default login" to refresh your active credentials.

sbadakhc commented 4 years ago

@ericyz will do! I'm actually doing both gcloud auth login and application default login.

sbadakhc commented 4 years ago

So the current state of play is that I'm still seeing the error after starting over from scratch. 0-bootstrap exists without errors. I can confirm the following:

Applied gcloud auth and application login:


Credentials saved to file: [/home/<REDACTED>/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).\

2. User account has the following roles:

Support Account Administrator Service Account Token Creator Owner Folder Admin Organization Administrator Quota Viewer

3. Terraform service account has the following roles:

Access Approval Approver Billing Account User Cloud Build Editor Cloud KMS Admin Compute Network Admin Compute Shared VPC Admin Security Admin Service Account Admin Logs Configuration Writer Organization Policy Administrator Folder Admin Organization Viewer Project Creator Source Repository Administrator Storage Admin


I will see if I can dig up any more info in the mean time.

ericyz commented 4 years ago

@sdharam , you mentioned your user account is part of gcp-organization-admins group (worth confirming again through -:) . After running the 0-bootstrap, the admin group will be granted ServiceAccountTokenCreator role to the terraform service account. So a couple checkpoint below

What is the permission for this admin group after running 0-bootstrap?
Does the admin group has ServiceAccountTokenCreator to terraform service account

sbadakhc commented 4 years ago

Hi @ericyz I can confirm that my user account is a member of the gcp-organization-admins group but that group does not have the ServiceAccountTokenCreator. Running terraform in the 0-bootstrap directory never assigns this role to that group. I have added the role manually for now and run terraform plan in the 1-org dir but still getting the error. Would be possible share a working config for reference? Many thanks, Sal.

ericyz commented 4 years ago

Hi @sbadakhc, using the configure in the repo as it is should work properly. Have you done customization on the 0-bootstrap? I suspect sa_enable_impersonation = true is missing in the 0-bootstrap. If you don't mind, could you please share your config.

sbadakhc commented 4 years ago

Hi @ericyz no problem. I'm really trying to get this to work without changing anything if at all possible. Here is the main.tf in 0-bootstrap dir. Please let me know if you need any more info.

/**
 * Copyright 2020 Google LLC
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

provider "google" {
  version = "~> 3.12.0"
}

provider "google-beta" {
  version = "~> 3.12.0"
}

provider "null" {
  version = "~> 2.1"
}

provider "random" {
  version = "~> 2.2"
}

/*************************************************
  Bootstrap GCP Organization.
*************************************************/

module "seed_bootstrap" {
  source                  = "terraform-google-modules/bootstrap/google"
  version                 = "~> 1.0"
  org_id                  = var.org_id
  billing_account         = var.billing_account
  group_org_admins        = var.group_org_admins
  group_billing_admins    = var.group_billing_admins
  default_region          = var.default_region
  sa_enable_impersonation = true
}

module "cloudbuild_bootstrap" {
  source                  = "terraform-google-modules/bootstrap/google//modules/cloudbuild"
  version                 = "~> 1.0"
  org_id                  = var.org_id
  billing_account         = var.billing_account
  group_org_admins        = var.group_org_admins
  default_region          = var.default_region
  terraform_sa_email      = module.seed_bootstrap.terraform_sa_email
  terraform_sa_name       = module.seed_bootstrap.terraform_sa_name
  terraform_state_bucket  = module.seed_bootstrap.gcs_bucket_tfstate
  sa_enable_impersonation = true
}

ericyz commented 4 years ago

I can confirm that my user account is a member of the gcp-organization-admins group but that group does not have the ServiceAccountTokenCreator.

Hi @sbadakhc , ServiceAccountTokenCreator roles should be granted on the service account level, not org or project level. So you should find this role by choosing _seedproject -> IAM -> service accounts -> select the terraform service account -> show info panel

A few check point below

After running 0-bootstrap, ensure the ServiceAccountTokenCreator is granted to admin group on the service account level. This is to ensure the admin group to have permission to impersonate the terraform service account
In 1-org, I recommend to use google provider with empty configure, and output the user email authenticate with GCP by using https://www.terraform.io/docs/providers/google/d/datasource_google_client_openid_userinfo.html. This to ensure you are running as the user in admin group

sbadakhc commented 4 years ago

Hi @ericyz and thanks for the update. Currently when I run terraform in 0-bootstrap it creates the terrafrom service account at the organisation level. This is inherited by the ctf-seed project but I noticed it doesn't have all the roles that the account has at the organisational level and its missing the service account token creator role entirely at both the organisational and project level. I'm also having to enable the cloudkms and cloudbuild api manually as before.

These a are the terraform service account roles created an a vanilla run at the org level:

Billing Account User
Compute Network Admin
Compute Shared VPC Admin
Security Admin
Service Account Admin
Logs Configuration Writer
Organization Policy Administrator
Folder Admin
Organization Viewer
Project Creator

And these are inherited account roles available in the cft-seed project.

Compute Network Admin
Security Admin
Service Account Admin
Logs Configuration Writer

When I enable the service account token created at the org level its not enable at the project level. If I try to edit account roles at the project level I can't as its greyed out. I can take a look at why this happening but that fact that the role is not granted at the 0-bootstrap stage as expected is the underlying issue.

I should mention that I only have the organization and group admins created when I re run these scripts and I'm not modifying anything other than the tfvars.

Thanks, Sal.

rjerrems commented 4 years ago

One thing to check is that you have run gcloud auth applicaiton-default login? Does gcloud auth list return the user account you are expecting?

sbadakhc commented 4 years ago

Hi @rjerrems I can confirm that I have run gcloud auth application login and indeed I run it with the no browser option. I have multiple google accounts so its necessary to use this option to explicitly choose the correct account.

See the output below.

gcloud auth list
           Credentialed Accounts
ACTIVE  ACCOUNT
*       salim.badakhchani@oneincredible.com

Thanks, Sal.

sbadakhc commented 4 years ago

After installing a number of deps that were failing due to local-exec failures caused by the missing --user arg missing I managed to a get a run through with the following errors:

module.org_shared_vpc_nonprod.module.project-factory.module.gcloud_disable.null_resource.upgrade[0]: Creation complete after 3m9s [id=8539438922138807160]

Error: Error setting billing account "014B4A-C769FD-C8455B" for project "projects/org-audit-logs-d84f": googleapi: Error 400: Precondition check failed., failedPrecondition

  on .terraform/modules/org_audit_logs/terraform-google-project-factory-7.1.0/modules/core_project_factory/main.tf line 96, in resource "google_project" "main":
  96: resource "google_project" "main" {

Error: Error setting billing account "014B4A-C769FD-C8455B" for project "projects/org-billing-logs-fed0": googleapi: Error 400: Precondition check failed., failedPrecondition

  on .terraform/modules/org_billing_logs/terraform-google-project-factory-7.1.0/modules/core_project_factory/main.tf line 96, in resource "google_project" "main":
  96: resource "google_project" "main" {

Error: Error setting billing account "014B4A-C769FD-C8455B" for project "projects/org-monitoring-nonprod-049c": googleapi: Error 400: Precondition check failed., failedPrecondition

  on .terraform/modules/org_monitoring_nonprod/terraform-google-project-factory-7.1.0/modules/core_project_factory/main.tf line 96, in resource "google_project" "main":
  96: resource "google_project" "main" {

Error: Error setting billing account "014B4A-C769FD-C8455B" for project "projects/org-shared-vpc-prod-09a8": googleapi: Error 400: Precondition check failed., failedPrecondition

  on .terraform/modules/org_shared_vpc_prod/terraform-google-project-factory-7.1.0/modules/core_project_factory/main.tf line 96, in resource "google_project" "main":
  96: resource "google_project" "main" {

rjerrems commented 4 years ago

Hi @sbadakhc - you may want to consider removing some of the log output / screen shots which contain your organizations data. Can you indicate where you are stuck now and any changes you have made to the code if any? That latest error looks like a project quota issue, see:

https://github.com/terraform-providers/terraform-provider-google/issues/1892 https://github.com/terraform-google-modules/terraform-example-foundation/issues/20

sbadakhc commented 4 years ago

Hi @rjerrems yep indeed it does. I will capture all the steps I took to get this far and update the ticket. I have made no code changes. This org is used for testing only and will be shelved but I take your point. Thanks, Sal.

rjerrems commented 4 years ago

Hi @sbadakhc - if we can't identify the specific issue here, I think we should probably close of the issue?

sbadakhc commented 4 years ago

Hi @rjerrems. Agree to close this as I seem to get a different error every time. I have not managed to get a complete run through which is disappointing. The steps I'm having to undertake manually are as follows:

After the successful execution of 0-bootstrap I have to configure the following manually.

Add Cloud Build Editor, Cloud KMS Admin, Source Repository Manager, Storage Admin and Service Account Token Administrator roles to the terraform service account at the organisation level. This account is inherited in the other projects and subsequently I cannot assign roles at the project level.
Install httplib2, pyasn1, pytz and uritemplate via pip as the modules fail to install via local-exec due to permissiosns.
Enable enable cloudkms.googleapis.com cloudbuild.googleapis.com on cloud-seed and cloud-build projects.

The above gets me passed 1-org but I get a host of other errors do to with roles and disabled apis in 2-network. Its becoming quite frustrating and I'm not sure why its so flaky. I currently cannot recommend this route to the client given the issues I'm having. Are there are tests we can run to confirm the minimum requirements with regards to roles? I understand that there is a helper script in the project factory demo that goes some way to addressing this but I would suggest we have one at the bootstrap level also.

Thanks, Sal.

rjerrems commented 4 years ago

Hi Sal, all of the required permissions are assigned during the bootstrap phase. This has been tested a number of times in different GCP organizations and generally speaking has worked well. This includes the permissions listed in the project factory module, you should not need to add any permissions manually.

The permissions provided to the service account at the organizational level, are listed here:

https://github.com/terraform-google-modules/terraform-google-bootstrap/blob/master/variables.tf#L83

(you can also see default permissions for the org_admins group in that file)

Sevice account token creator permissions are provided on the service account resource itself, so you will not be able to see these permissions unless you navigate to the service account in the seed project and click the info panel (or use gcloud). The actual code setting that up is here:

https://github.com/terraform-google-modules/terraform-google-bootstrap/blob/master/main.tf#L156

A few things I would recommend you do to get a clean run:

Make sure your account / the terraform service account that gets created has sufficient quota to create projects https://support.google.com/code/contact/project_quota_increase
Double check that gcloud is configured to run as the user in the org_admins group and make sure you run gcloud auth login and gcloud auth application-default login as that user
Clear any locally cached modules under .terraform for each stage and run through it on a mac or linux based machine if possible, as the modules are usually more well tested on these platforms

Failing that, feel free to raise an issue if you are able to reproduce a specific problem and we can look into it - information about the runtime environment (OS, terraform version) and any modifications you have made will help.

terraform-google-modules / terraform-example-foundation

1-org step Terraform Plan fails Error 403: The caller does not have permission #42