search

IBM-Cloud / terraform-provider-ibm

https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs
Mozilla Public License 2.0
341 stars 670 forks source link

Error provisiong VPC kubernetes cluster using service id API key #2128

Closed feliperfmarques closed 3 years ago

feliperfmarques commented 3 years ago

Hi there,

I have been used terraform to provision infrastructure on IBM Cloud. For this, I have three terraform projects:

Initially, I manually created admin resource-group and service id and also setted policies for service id via IBM Cloud UI.

Basically admin terraform project is used for: create global resources in admin resource group, create other resource-groups (prod and dev), create services ids(prod and dev), beyond to set policies for each service id.

I configured IBM Cloud terraform provider using API key created for admin service id. Everything works fine in this project.

In prod and dev terraform projects, I wish to create practically identical resources for each environment, using following structure:

I configured terraform provider using an API key created for prod and dev service ids, the same way I did for the admin project. I can to provision practically all resources, with the exception of kubernetes service(iks module), because I got the following error:

Error: Request failed with status code: 500, ServerErrorResponse: {"incidentID":"603b9405772cd050-GRU","code":"A0002","description":"Could not connect to a backend service. Try again later.","type":"Authentication"}

Below I am attaching the service policies setted in admin project for each service id, according official documentation. 

resource "ibm_iam_service_policy" "terraform_iac_prod_policy_1" {
  iam_service_id = data.ibm_iam_service_id.terraform_iac_prod.service_ids.*.id[0]
  roles          = ["Administrator"]
}

# VPC Permissions
# https://cloud.ibm.com/docs/vpc?topic=vpc-managing-user-permissions-for-vpc-resources
resource "ibm_iam_service_policy" "terraform_iac_prod_policy_2" {
  iam_service_id = data.ibm_iam_service_id.terraform_iac_prod.service_ids.*.id[0]
  roles          = ["Administrator"]
  resources {
    service = "is"
  }
}

# Kubernetes Permissions
# https://cloud.ibm.com/docs/containers?topic=containers-access_reference
resource "ibm_iam_service_policy" "terraform_iac_prod_policy_3" {
  iam_service_id = data.ibm_iam_service_id.terraform_iac_prod.service_ids.*.id[0]
  roles          = ["Administrator", "Manager"]
  resources {
    service = "containers-kubernetes"
  }
}

resource "ibm_iam_service_policy" "terraform_iac_prod_policy_4" {
  iam_service_id = data.ibm_iam_service_id.terraform_iac_prod.service_ids.*.id[0]
  roles          = ["Administrator", "Manager"]
  resources {
    service = "container-registry"
  }
}

# https://cloud.ibm.com/docs/key-protect?topic=key-protect-manage-access
resource "ibm_iam_service_policy" "terraform_iac_prod_policy_5" {
  iam_service_id = data.ibm_iam_service_id.terraform_iac_prod.service_ids.*.id[0]
  roles          = ["Administrator", "Manager"]
  resources {
    service = "kms"
    resource_group_id = data.ibm_resource_group.rg_prod.id
  }
}

resource "ibm_iam_service_policy" "terraform_iac_prod_policy_7" {
  iam_service_id = data.ibm_iam_service_id.terraform_iac_prod.service_ids.*.id[0]
  roles          = ["Administrator", "Manager"]
  resources {
    service = "cloudcerts"
  }
}

However, despite setting all the polices like the documentation says, I can't create the cluster. Any more policy is needed?

Terraform Version

Terraform v0.13.5 provider registry.terraform.io/ibm-cloud/ibm v1.16.1

Terraform Configuration Files

data "ibm_is_vpc" "vpc_1" {
  name = "vpc-${var.globals.environment}-1"
}

data "ibm_is_subnet" "vpc_1_subnet_1_us_south_1" {
  name = "vpc-${var.globals.environment}-1-subnet-1-${local.zone1}"
}

data "ibm_is_subnet" "vpc_1_subnet_1_us_south_2" {
  name = "vpc-${var.globals.environment}-1-subnet-1-${local.zone2}"
}

data "ibm_resource_instance" "kms_1_us_south" {
  name              = "kms-${var.globals.environment}-1-${var.globals.location}"
  service           = "kms"
  location          = var.globals.location
  resource_group_id = var.globals.resource_group
}

data "ibm_kms_key" "kms_1_key_1_us_south" {
  instance_id = data.ibm_resource_instance.kms_1_us_south.guid
  key_name = "kms-${var.globals.environment}-1-${var.globals.location}"
}

locals {
  zone1 = "${var.globals.location}-1"
  zone2 = "${var.globals.location}-2"
}

resource "ibm_container_vpc_cluster" "iks_1_us_south" {
  name                            = "iks-${var.globals.environment}-1-${var.globals.location}"
  vpc_id                          = data.ibm_is_vpc.vpc_1.id
  kube_version                    = "1.18.13"
  flavor                          = "bx2.2x8"
  pod_subnet                      = "172.17.0.0/17"
  service_subnet                  = "172.21.0.0/17"
  worker_count                    = 2
  resource_group_id               = var.globals.resource_group
  disable_public_service_endpoint = false

  worker_labels = {
    "pool" = "default"
  }

  zones {
    subnet_id = data.ibm_is_subnet.vpc_1_subnet_1_us_south_1.id
    name      = local.zone1
  }

  zones {
    subnet_id = data.ibm_is_subnet.vpc_1_subnet_1_us_south_2.id
    name      = local.zone2
  }

  kms_config {
    instance_id      = data.ibm_kms_key.kms_1_key_1_us_south.instance_id
    crk_id           = data.ibm_kms_key.kms_1_key_1_us_south.keys.*.id[0]
    private_endpoint = true
  }

  tags = [var.globals.environment]
}

resource "ibm_container_addons" "iks_1_addons_us_south" {
  depends_on        = [ibm_container_vpc_cluster.iks_1_us_south] 
  cluster           = ibm_container_vpc_cluster.iks_1_us_south.id
  resource_group_id = var.globals.resource_group

  addons {
    name    = "cluster-autoscaler"
    version = "1.0.1"
  }
  addons {
    name    = "alb-oauth-proxy"
    version = "1.0.0"
  }
  addons {
    name    = "istio"
    version = "1.7"
  }
}

Affected Resource(s):

ibm_container_vpc_cluster

Steps to Reproduce

  1. Create a service id
  2. Set service id policies according documentation
  3. Create an service id API key
  4. Use service id API key for configure IBM Cloud terraform provider
  5. Try provision an cluster using ibm_container_vpc_cluster

Actual Behavior

The resource is not created.

Expected Behavior

The resource should be created.

References

Access policies for kubernetes service:

https://cloud.ibm.com/docs/containers?topic=containers-access_reference&locale=pt-BR

hkantare commented 3 years ago

Its an issue from Kubernetes API as per their comments 

This is due to an authentication error with IAM when the cluster create is trying to create an API key, because there is not one yet for the account in eu-gb. This is because IKS does not currently support service to service, so it is unable to create a user api key with a service id.
We have an epic for service to service support. https://bigblue.aha.io/features/SECADV-72
The customer could work around this by creating an api key in eu-gb targeting the resource group they want the cluster in ibmcloud ks api-key reset --region eu-gb

So you should have one user API key created in the region where you are provsioning the cluster

feliperfmarques commented 3 years ago

Helloooo,

Thanks so much @hkantare. You saved me!!! I would never discover it.

I’ll show you the steps I’ve followed for others who have the same issue, basically is what was reported above:

  1. Log in to ibmcloud cli with a user account
  2. Select the resource-group you want to provision the cluster, using command: ibmcloud target -g <resource-group-name>
  3. Reset API key for kubernetes service ibmcloud ks api-key reset --region <cluster-region>
  4. Repeat step 2-3 for all resources-groups you wish to provision clusters
hkantare commented 3 years ago

closing the issue based on above comments