oci_network_load_balancer_backend 404-NotAuthorizedOrNotFound but the backeds are correctly created

garutilorenzo commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version and Provider Version

Terraform v1.1.6 on linux_amd64

provider registry.terraform.io/hashicorp/oci v4.64.0
provider registry.terraform.io/hashicorp/oci v4.76.0
provider oracle/oci 4.76.0
provider registry.terraform.io/hashicorp/template v2.2.0

i've tried also oracle/oci module v.4.64.0 and the latest oracle oci module

Affected Resource(s)

oci_network_load_balancer_backend

Terraform Configuration Files

lb.tf

resource "oci_network_load_balancer_network_load_balancer" "k3s_load_balancer" {
  compartment_id = var.compartment_ocid
  display_name   = var.k3s_load_balancer_name
  subnet_id      = oci_core_subnet.oci_core_subnet11.id

  is_private                     = true
  is_preserve_source_destination = false

  freeform_tags = {
    "provisioner"           = "terraform"
    "environment"           = "${var.environment}"
    "${var.unique_tag_key}" = "${var.unique_tag_value}"
  }
}

resource "oci_network_load_balancer_listener" "k3s_kube_api_listener" {
  default_backend_set_name = oci_network_load_balancer_backend_set.k3s_kube_api_backend_set.name
  name                     = "k3s kube api listener"
  network_load_balancer_id = oci_network_load_balancer_network_load_balancer.k3s_load_balancer.id
  port                     = var.kube_api_port
  protocol                 = "TCP"
}

resource "oci_network_load_balancer_backend_set" "k3s_kube_api_backend_set" {
  health_checker {
    protocol = "TCP"
    port     = var.kube_api_port
  }

  name                     = "k3s kube api backend"
  network_load_balancer_id = oci_network_load_balancer_network_load_balancer.k3s_load_balancer.id
  policy                   = "FIVE_TUPLE"
  is_preserve_source       = true
}

resource "oci_network_load_balancer_backend" "k3s_kube_api_backend" {
  depends_on = [
    oci_core_instance_pool.k3s_servers,
  ]

  count                    = 2
  backend_set_name         = oci_network_load_balancer_backend_set.k3s_kube_api_backend_set.name
  network_load_balancer_id = oci_network_load_balancer_network_load_balancer.k3s_load_balancer.id
  port                     = var.kube_api_port

  target_id = data.oci_core_instance_pool_instances.k3s_servers_instances.instances[count.index].id
}

data.tf

data "oci_core_instance_pool_instances" "k3s_servers_instances" {
  depends_on = [
    oci_core_instance_pool.k3s_servers,
  ]
  compartment_id   = var.compartment_ocid
  instance_pool_id = oci_core_instance_pool.k3s_servers.id
}

instance_pool.tf

resource "oci_core_instance_pool" "k3s_servers" {
  depends_on = [
    oci_identity_dynamic_group.compute_dynamic_group,
    oci_identity_policy.compute_dynamic_group_policy
  ]

  lifecycle {
    create_before_destroy = true
    ignore_changes        = [load_balancers, freeform_tags]
  }

  display_name              = "k3s-servers"
  compartment_id            = var.compartment_ocid
  instance_configuration_id = oci_core_instance_configuration.k3s_server_template.id

  placement_configurations {
    availability_domain = var.availability_domain
    primary_subnet_id   = oci_core_subnet.default_oci_core_subnet10.id
    fault_domains       = var.fault_domains
  }

  size = 2

  freeform_tags = {
    "provisioner"           = "terraform"
    "environment"           = "${var.environment}"
    "${var.unique_tag_key}" = "${var.unique_tag_value}"
    "k3s-cluster-name"      = "${var.cluster_name}"
    "k3s-instance-type"     = "k3s-server"
  }
}

Panic Output

│ Error: 404-NotAuthorizedOrNotFound, Unknown resource Entity of type Backend with key ocid1.instance.oc1.eu-zurich-1.an5heljr5kjm7pycg2zjcqxbsbe26rwvc6szowce7u7qpi2m5gaqxm53bwza.6443 not found 
│ Suggestion: Either the resource has been deleted or service Network Load Balancer Backend need policy to access this resource. Policy reference: https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/policyreference.htm
│ Documentation: https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/network_load_balancer_backend 
│ Request Target: GET https://network-load-balancer-api.eu-zurich-1.oci.oraclecloud.com/20200501/networkLoadBalancers/ocid1.networkloadbalancer.oc1.eu-zurich-1.amaaaaaa5kjm7pyao4axnaunpvssqzrljdopyyuouepvl5dlvsd5d4y7q3ya/backendSets/k3s%20kube%20api%20backend/backends/ocid1.instance.oc1.eu-zurich-1.an5heljr5kjm7pycg2zjcqxbsbe26rwvc6szowce7u7qpi2m5gaqxm53bwza.6443 
│ Provider version: 4.76.0, released on 2022-05-21.  
│ Service: Network Load Balancer Backend 
│ Operation Name: GetBackend 
│ OPC request ID: b55d15500a84243335655841e09c3ab3/2B8A5720ED91321A19ADF40B894FAD48/AC30D14BAA69E1C642354767A3470EA3 
│ 
│ 
│   with module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0],
│   on ../k3slb.tf line 36, in resource "oci_network_load_balancer_backend" "k3s_kube_api_backend":
│   36: resource "oci_network_load_balancer_backend" "k3s_kube_api_backend" {
│ 
╵
╷
│ Error: 404-NotAuthorizedOrNotFound, Unknown resource Entity of type Backend with key ocid1.instance.oc1.eu-zurich-1.an5heljr5kjm7pycfdk2fs3stowtrhca2dmtz2auu6rkx5qlwfrqxurxlhaq.6443 not found 
│ Suggestion: Either the resource has been deleted or service Network Load Balancer Backend need policy to access this resource. Policy reference: https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/policyreference.htm
│ Documentation: https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/network_load_balancer_backend 
│ Request Target: GET https://network-load-balancer-api.eu-zurich-1.oci.oraclecloud.com/20200501/networkLoadBalancers/ocid1.networkloadbalancer.oc1.eu-zurich-1.amaaaaaa5kjm7pyao4axnaunpvssqzrljdopyyuouepvl5dlvsd5d4y7q3ya/backendSets/k3s%20kube%20api%20backend/backends/ocid1.instance.oc1.eu-zurich-1.an5heljr5kjm7pycfdk2fs3stowtrhca2dmtz2auu6rkx5qlwfrqxurxlhaq.6443 
│ Provider version: 4.76.0, released on 2022-05-21.  
│ Service: Network Load Balancer Backend 
│ Operation Name: GetBackend 
│ OPC request ID: 521d57cf4abd2ac90d40ac9d7c1a6ded/8B5292856DD61813672E31F59476D54D/74ABF142B8425A581854E9E355706EC8 
│ 
│ 
│   with module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[1],
│   on ../k3slb.tf line 36, in resource "oci_network_load_balancer_backend" "k3s_kube_api_backend":
│   36: resource "oci_network_load_balancer_backend" "k3s_kube_api_backend" {
│

If i try to apply again:

module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Still creating... [2m41s elapsed]
module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Still creating... [2m51s elapsed]
module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Still creating... [3m1s elapsed]
module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Still creating... [3m11s elapsed]
module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Still creating... [3m21s elapsed]
module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Still creating... [3m31s elapsed]
module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Still creating... [3m41s elapsed]
module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Still creating... [3m51s elapsed]
module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Still creating... [4m1s elapsed]
╷
│ Error: 409-NotAuthorizedOrResourceAlreadyExists, Conflict.  For nlb-id Duplicate backend IP/id + port combinations not allowed: Backend(name=null, ipVersion=Ipv4, ipAddress=null, targetId=ocid1.instance.oc1.eu-zurich-1.an5heljr5kjm7pycfdk2fs3stowtrhca2dmtz2auu6rkx5qlwfrqxurxlhaq, port=6443, weight=1, isDrain=false, isBackup=false, isOffline=false) 
│ Suggestion: The resource is in a conflicted state. Please retry again or contact support for help with service: Network Load Balancer Backend
│ Documentation: https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/network_load_balancer_backend 
│ Request Target: POST https://network-load-balancer-api.eu-zurich-1.oci.oraclecloud.com/20200501/networkLoadBalancers/ocid1.networkloadbalancer.oc1.eu-zurich-1.amaaaaaa5kjm7pyao4axnaunpvssqzrljdopyyuouepvl5dlvsd5d4y7q3ya/backendSets/k3s%20kube%20api%20backend/backends 
│ Provider version: 4.76.0, released on 2022-05-21.  
│ Service: Network Load Balancer Backend 
│ Operation Name: CreateBackend 
│ OPC request ID: de113a2f981f3412fc646ea3667b1edf/42712CB6BCC726E45949054091989B81/70843B5EC104486CD24F74190B741DC2 
│ 
│ 
│   with module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[1],
│   on ../k3slb.tf line 36, in resource "oci_network_load_balancer_backend" "k3s_kube_api_backend":
│   36: resource "oci_network_load_balancer_backend" "k3s_kube_api_backend" {
│ 
╵
╷
│ Error: 409-NotAuthorizedOrResourceAlreadyExists, Conflict.  For nlb-id Duplicate backend IP/id + port combinations not allowed: Backend(name=null, ipVersion=Ipv4, ipAddress=null, targetId=ocid1.instance.oc1.eu-zurich-1.an5heljr5kjm7pycg2zjcqxbsbe26rwvc6szowce7u7qpi2m5gaqxm53bwza, port=6443, weight=1, isDrain=false, isBackup=false, isOffline=false) 
│ Suggestion: The resource is in a conflicted state. Please retry again or contact support for help with service: Network Load Balancer Backend
│ Documentation: https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/network_load_balancer_backend 
│ Request Target: POST https://network-load-balancer-api.eu-zurich-1.oci.oraclecloud.com/20200501/networkLoadBalancers/ocid1.networkloadbalancer.oc1.eu-zurich-1.amaaaaaa5kjm7pyao4axnaunpvssqzrljdopyyuouepvl5dlvsd5d4y7q3ya/backendSets/k3s%20kube%20api%20backend/backends 
│ Provider version: 4.76.0, released on 2022-05-21.  
│ Service: Network Load Balancer Backend 
│ Operation Name: CreateBackend 
│ OPC request ID: 50e0fe689671c85f94d5001fa64469b8/62921F9F81DA9FCC625A918BF0F6872C/8F2367F2069564F8E1629E7A7A0DB7C4 
│ 
│ 
│   with module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0],
│   on ../k3slb.tf line 36, in resource "oci_network_load_balancer_backend" "k3s_kube_api_backend":
│   36: resource "oci_network_load_balancer_backend" "k3s_kube_api_backend" {
│

Actual Behavior

Backend are correctly created but 404-NotAuthorizedOrNotFound error occured

lb-bug-web2

lb-bug-web

Steps to Reproduce

Apply this module

garutilorenzo commented 2 years ago

The same probelm occures if i try to use the private ip ocid as target_id:

│ Error: 404-NotAuthorizedOrNotFound 
│ Provider version: 4.64.0, released on 2022-02-16. This provider is 13 Update(s) behind to current. 
│ Service: Network Load Balancer Backend 
│ Error Message: Unknown resource Entity of type Backend with key ocid1.privateip.oc1.eu-zurich-1.ab5heljrku3dvhwandbamb34s7cthenz4vki52iycqgoj5obfkoexbserhzq.6443 not found 
│ OPC request ID: 73aca760d408945945bcb3ed681d8b8b/EF6781C100C57DB2EF6AC79D40FF19F1/74E7BA9E972DCF1A125522BF258F90E0 
│ Suggestion: Either the resource has been deleted or service Network Load Balancer Backend need policy to access this resource. Policy reference: https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/policyreference.htm
│ 
│ 
│   with module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0],
│   on ../k3slb.tf line 36, in resource "oci_network_load_balancer_backend" "k3s_kube_api_backend":
│   36: resource "oci_network_load_balancer_backend" "k3s_kube_api_backend" {
│ 
╵
╷
│ Error: 404-NotAuthorizedOrNotFound 
│ Provider version: 4.64.0, released on 2022-02-16. This provider is 13 Update(s) behind to current. 
│ Service: Network Load Balancer Backend 
│ Error Message: Unknown resource Entity of type Backend with key ocid1.privateip.oc1.eu-zurich-1.ab5heljryssggfe4rldf26asmvcpjrj2aoktk7fp6yzwdxpngqszpqgugfja.6443 not found 
│ OPC request ID: ba08ecc04d664aa6a44161e9fa453ec3/06469C6E7FA4A2840278859A5C74EB5E/79880A506DB3F22E3DA470E79C826946 
│ Suggestion: Either the resource has been deleted or service Network Load Balancer Backend need policy to access this resource. Policy reference: https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/policyreference.htm
│ 
│ 
│   with module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[1],
│   on ../k3slb.tf line 36, in resource "oci_network_load_balancer_backend" "k3s_kube_api_backend":
│   36: resource "oci_network_load_balancer_backend" "k3s_kube_api_backend" {

garutilorenzo commented 2 years ago

I've found a workaround. With some reverse enginereeng i've inspected the html of the web console and i've found that the name was setted to the instance name (the name if is not provided is automatically generated). Inspecting the page i've found that in the id of the html the name instead was setted to $instance_id:$backend_port

oci-reverse-eng

Then i've tried to import the resurce with:

terraform import module.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[1] "networkLoadBalancers/ocid1.networkloadbalancer.oc1.eu-zurich-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/backendSets/k3s_kube_api_backend/backends/ocid1.instance.oc1.eu-zurich-1.xxxxxxxxxxxxxxxxxxxxxx:6443"

and the import has correctly imported the backend

Changing the backend name to $instance_id:$backend_port fix the problem, but this is only a workaround.

Also, with this workaround the backand name in the web console is always setted to the instance name. I think something has changed in the OCI api and at the moment is not documented.

garutilorenzo commented 2 years ago

UPDATE

The value in the name argument can be any kind of string

name                     = format("%s_%s", "k3s_server", count.index)

or

name                     = data.oci_core_instance_pool_instances.k3s_servers_instances.instances[count.index].display_name

so the name argument is a required argument, the value must be "any kind of string". In the web interface the result is always the same, the instance name is displayed:

oci-backends

This is the working code:

resource "oci_network_load_balancer_backend" "k3s_kube_api_backend" {
  depends_on = [
    oci_core_instance_pool.k3s_servers,
  ]

  count                    = var.k3s_server_pool_size
  backend_set_name         = oci_network_load_balancer_backend_set.k3s_kube_api_backend_set.name
  network_load_balancer_id = oci_network_load_balancer_network_load_balancer.k3s_load_balancer.id
  name                     = data.oci_core_instance_pool_instances.k3s_servers_instances.instances[count.index].display_name
  port                     = var.kube_api_port
  target_id                = data.oci_core_instance_pool_instances.k3s_servers_instances.instances[count.index].id
}

ravinitp commented 1 year ago

Thank you for reporting the issue. We have raised an internal ticket to track this. Our service engineers will get back to you.

oracle / terraform-provider-oci