Access issues against some API endpoints

zidz commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.0.1 Google Provider 3.75

Affected Resource(s)

data "google_project"
resource "google_iap_brand"

Terraform Configuration Files

provider.tf

...

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "3.75"
    }
  }

  backend "gcs" {
  }
}

provider "google" {
  project = local.project_id
  region  = var.region
}

data "google_project" "gp" {}

output "project_number" {
  value = data.google_project.gp.number
}

...

iap.tf

...

resource "google_iap_brand" "project_brand" {
  support_email     = var.support_email
  application_title = "IAP protected Application"
  project           = local.project_number
}

...

Debug Output

403 during plan, resource "google_iap_brand" https://gist.github.com/zidz/4c3dda5180d93de33a18c19f31e868f6

403 during plan, data "google_project" https://gist.github.com/zidz/01c7b46d77a5941a204895d407e94a38

Panic Output

N/A

Expected Behavior

It should be able to make these API calls as they can be done with the exact same oauth token using curl. Terraform is able to deploy all other resources we use in these projects like buckets, GKE clusters and a whole bunch more. The 403 happens only for the API calls made for the ones mentioned above and in the gists. Terraform vs curl API access verified like this:

For project with IAP API 403

gcloud config set auth/impersonate_service_account $SA_WITH_OWNER_PERMISSION
export GOOGLE_OAUTH_ACCESS_TOKEN="$(gcloud auth print-access-token)"

curl --request GET --header "Authorization: Bearer ${GOOGLE_OAUTH_ACCESS_TOKEN}"  "https://iap.googleapis.com/v1/projects/REDACTED/brands/REDACTED"

terraform init -reconfigure -backend-config="bucket=$STATE_BUCKET" -backend-config="prefix=state/"
terraform plan -var-file="$TFVARS" -out=$CURRENT_PLAN

For project with billingInfo API 403

gcloud config set auth/impersonate_service_account $SA_WITH_OWNER_PERMISSION
export GOOGLE_OAUTH_ACCESS_TOKEN="$(gcloud auth print-access-token)"

curl --request GET --header "Authorization: Bearer ${GOOGLE_OAUTH_ACCESS_TOKEN}"  "https://cloudbilling.googleapis.com/v1/projects/REDACTED/billingInfo"

terraform init -reconfigure -backend-config="bucket=$STATE_BUCKET" -backend-config="prefix=state/"
terraform plan -var-file="$TFVARS" -out=$CURRENT_PLAN

This is NOT a intermittent error.

Actual Behaviour

403 on API call, see gists

Steps to Reproduce

If we run this Terraform code from our laptops the plan works as expected. If run from a GKE node connected to our shared VPC this happens. The shared VPC is routed through a on-prem network before reaching internet.

Important Factoids

N/A

References

N/A

edwardmedia commented 3 years ago

@zidz Terraform and gcloud do not use the exact same methods to handle authentication & authorization. Below is the one recommended for the Terraform. From steps to reproduce you provided, the issue happens on GKE node(s) but not on your local machine(s). Can you compare if both environments have the same environment variables?

https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference#full-reference

zidz commented 3 years ago

Thank you for taking a look into this @edwardmedia

I don't quite understand how they differ regarding to authentication, would you please elaborate a bit? Scrolling down a bit in the provider reference documentation it mentions the GOOGLE_OAUTH_ACCESS_TOKEN environment variable. It describes how you can use the OAuth Authorization: Bearer tokens with Terraform, so that's what we've been using for Terraform since start and is now also used with curl to test the possibility to reach and authenticate to the endpoints mentioned in the gists.

Regarding the environment variables there isn't any relevant environment variables at my laptop that isn't in the pod in GKE. As per the provider reference documentation the only one present on my laptop (and in the pod) that is consumed by Terraform is the GOOGLE_OAUTH_ACCESS_TOKEN.

edwardmedia commented 3 years ago

https://github.com/hashicorp/terraform-provider-google/issues/8671

zidz commented 2 years ago

Hi,

I don't understand the relation between my bug report and issue #8671 .

I don't see that I've mentioned it before or if it does make any difference but I've also downloaded the Service Account key file and did let Terraform authenticate with that one, not relying on Workload Identity in GKE, the plan ended with the very same error.

zidz commented 2 years ago

We found out that when the Shared VPN has Private Google Access enabled, Terraform has trouble to access some endpoints. So to fix that you have to disable that setting or alter the DNS in a suitable way.

github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-google