Namespace resource causes perpetual diffs on `kubernetes.io` labels applied by default by Kubernetes

[mirraxian]

When using the rancher2_namespace label attribute the Provider does not ignore changes applied from Kubernetes itself such as kubernetes.io/metadata.name which is applied by default from Kubernetes 1.21

Leading to perpetual diffs such as this:

Terraform will perform the following actions:
# module.example_namespace_template.rancher2_namespace.namespace will be updated in-place
~ resource "rancher2_namespace" "namespace" {
id = "example"
~ labels = {
- "kubernetes.io/metadata.name" = "example" -> null

This problem was also seen in the default Kubernetes Provider, which has the following note on the Namespace Resource's label attribute](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace#labels)

By default, the provider ignores any labels whose key names end with kubernetes.io. This is necessary because such labels can be mutated by server-side components and consequently cause a perpetual diff in the Terraform plan output. If you explicitly specify any such labels in the configuration template then Terraform will consider these as normal resource attributes and manage them as expected (while still avoiding the perpetual diff problem). For more info info see Kubernetes reference

Besides ignoring kubernetes.io the default Kubernetes provider also provides an ignore_annotations and ignore_labels as seen in this PR

[stefan-sesser]

I have exactly the same issue after upgrading our clusters from v1.20.11-rancher1-1 to v1.23.7-rancher1-1

[euven]

Seeing this too - it's causing headaches, because it's causing unnecessary dependent resource updates too.

[MKlimuszka]

This needs to be reproduced in 2023-Q1-2.6x and 2023-Q1-2.7x. SURE-5085

[stormqueen1990]

This issue is still reproducible in Rancher versions v2.6.10 and v2.7.0, with the rancher/rancher2 Terraform provider on both versions v1.24.1 and v1.25.0, using the following resources in Terraform:

data "rancher2_cluster" "local" {
  name = "local"
}

resource "rancher2_project" "test_project" {
  name = "test-project"
  cluster_id = data.rancher2_cluster.local.id
}

resource "rancher2_namespace" "test_namespace" {
  name       = "test-namespace"
  project_id = rancher2_project.test_project.id
  labels = {
    "test-label" : "test-value",
  }
}

[isaacsanders]

Is it preferred to ignore the kubernetes.io/ labels or to add a facility for ignoring labels more generally? This is affecting my team currently, and I'd be willing to work on fixing this if I can have a better understanding of the acceptance criteria/direction the team wants here.

[samjustus]

transferring this issue to rancher/rancher for ease of tracking

[MKlimuszka]

Adding this back to the triage queue for reproduction in the newest rancher versions.

[gehrkefc]

I confirm I could reproduce that with Rancher v2.9.1:

steps:

1. spin up a rancher build: docker run --privileged -d --restart=always -p 8080:80 -p 8443:443 rancher/rancher:v2.9.1

   2. open https://localhost:8443

   3. get the bootstrap password docker logs $(docker ps | grep rancher/rancher | awk '{print $1}') 2>&1 | grep "Bootstrap Password:"

2. create a token for the admin user in Rancher UI:

   • go to right top side
   • click in the user icon
   • select 'Account & API Keys'
   • then click 'Create API Key' button
   • left the default options and click and 'Create' button

3. create a main.tf file and replace access_key and secret_key in it:

terraform {
  required_providers {
    rancher2 = {
      source = "rancher/rancher2"
      version = "4.2.0"
    }
  }
}

provider "rancher2" {
  api_url   = "https://localhost:8443"
  secret_key = **REPLACE_BY_STEP_4_OBTAINED_VALUE**
  access_key = **REPLACE_BY_STEP_4_OBTAINED_VALUE**
  insecure = true
}

data "rancher2_cluster" "local" {
  name = "local"
}

resource "rancher2_project" "project" {
  name = "project"
  cluster_id = data.rancher2_cluster.local.id
}

resource "rancher2_namespace" "namespace" {
  name = "example"
  project_id = rancher2_project.project.id
  labels = {
    "test-label" = "test-value"
    "kubernetes.io/metadata.name" = "example"
  }
}

6. run terraform commands

   terraform init
   terraform apply

7. remove kubernetes.io/metadata.name = "example" from labels in main.tf file.

8. run terraform apply again and confirm

terraform apply

in this step you should notice the diff indicating the label will be removed. if you confirm it and run it again the label will be there in the diff again. even after confirming it and running it again.

also, it's possible to see it in Rancher UI after removed:

[matttrach]

I am not seeing it here, but as a short term solution has anyone tried using the ignore_changes lifecycle meta?

This is a bit of a too generalized solution, so I think a medium term and longer term solution will also be necessary, but to get anyone who is blocked able to move forward in the short term.

[matttrach]

This is a discrepancy between Terraform's expectations and Kubernetes. Terraform wants to be the source of truth on all changes, but the reality is that APIs change objects without Terraform's permission all the time, especially on labels and annotations. This discrepancy is why the ignore_changes lifecycle meta exists. If the ignore_changes meta fails to ignore labels, then this is a bug, otherwise this is a feature request that in my opinion should be addressed sooner rather than later.

[matttrach]

If this should be a feature the requirements would be:

• A new argument on the resource that allows for selectively ignoring additions, removal, or changes to labels.
• Nice to have: regex search, but basic string contains would be fine.

[prachidamle]

After offline discussion with @matttrach this issue seems to be purely Terraform provider provider area, hence transferring it to TF provider repo cc @caroline-suse-rancher