403 Error unable to authenticate provider

[charlesfused]

I have created a service account as follows

resource "google_service_account" "netapp" {
  account_id   = "netapp"
  display_name = "Service Account for Netapp"
}

resource "google_project_iam_member" "netapp_admin" {
  project = local.project_id
  role    = "roles/netappcloudvolumes.admin"
  member  = "serviceAccount:${google_service_account.netapp.email}"
}

resource "google_project_iam_member" "netapp_sa" {
  project = local.project_id
  role    = "roles/iam.serviceAccountUser"
  member  = "serviceAccount:${google_service_account.netapp.email}"
}

Downloaded JSON key, and configured provider as follows

provider "netapp-gcp" {
  project     = local.project_id
  service_account = "/path/to/netapp_sa.json"
}

When attempting to create any Netapp resource a 403 error is immediately returned.

╷
│ Error: code: 403, message: Authentication failure
│ 
│   with netapp-gcp_storage_pool.xxx,
│   on netapp.tf line 11, in resource "netapp-gcp_storage_pool" "xxx":
│   11: resource "netapp-gcp_storage_pool" "xxx" {

This is the resource I'm trying to create

resource "netapp-gcp_storage_pool" "xxx" {
  name = "xxx"
  provider = netapp-gcp
  region = local.region
  zone = "us-west1-b"
  network = google_compute_network.vpc_network.name
  global_ad_access = false
  size = 1024
  service_level = "StandardSW"
  storage_class = "software"
}

What permissions need to be set in order for the provider to work?

[okrause]

I tested it and it works for me. I removed the "NetApp_sa" resource. It is not needed. Please make sure that "/path/to/netapp_sa.json" is an absolute path. I don't expect that to be the problem, since that would throw a different error. Did you create the service account in the same project as the CVS resources? Maybe you google provider is pointing to a different project?

I also encourage you to use service account impersonation to get rid of SA key files.

More infos: https://cloud.google.com/architecture/partners/netapp-cloud-volumes/api#authenticate_using_a_service_account_key https://registry.terraform.io/providers/NetApp/netapp-gcp/latest/docs#service_account

[charlesfused]

Thanks for responding so quickly! I tried removing the netapp_sa resource and also confirmed path. However still getting 403 denied.

Also tried using credentials via setting the JSON key as runtime variable. And tried using service account impersonation.

In all cases 403 is returned.

Also tried setting project to the project ID and project number, neither helped. Confirmed that the same project is being used everywhere in Terraform.

When TF_LOG=debug is set, the only error shown is 2024-04-20T13:24:40.371-0400 [ERROR] vertex "netapp-gcp_storage_pool.xxx" error: code: 403, message: Authentication failure

I'm not sure what else to try at this point. Are there any other log files?

[okrause]

No. TF_LOG is the only log available as far as I know. Have a look into your key.json file. Is it the right one? Is it for the project you want to access? The service uses HTTPS calls with JWT bearer auth. The error seems to indicate the provider doesn't offer a valid JWT token. The token is created from the SA key using standard Google SDK libraries.

[charlesfused]

Thanks for the response. Confirmed the key.json file is correct. I am able to use it with the google-cloud-netapp Python SDK as well - it just doesn't work with this provider

[okrause]

I am out of ideas. When using a JSON key, the provider passes the key down into standard Google auth libraries to get a JWT token. It is using that key to do APi calls. Nothing really which can fail here. I am out of ideas. Maybe expect one: Did you specify a valid region? project NUMBER and region are part of the API URL used. If one of them is wrong, you are calling invalid URLs, which may result in a misleading error (just guessing here). I recommend enabling TF_LOG=debug and have a very, very careful look at the URL the provider is calling.