rancher / terraform-provider-rancher2

Terraform Rancher2 provider
https://www.terraform.io/docs/providers/rancher2/
Mozilla Public License 2.0
260 stars 226 forks source link

error while parsing encryption provider configuration file with RKE secrets encryption custom_config #645

Open jmatsushita opened 3 years ago

jmatsushita commented 3 years ago

Hi there,

I use the following terraform code with the latest version 0.13.0 in a resource "rancher2_cluster_template" "template".template_revisions.cluster_config.rke_config block:

kube_api {
  secrets_encryption_config {
    enabled       = true
    custom_config = <<-EOT
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aesgcm:
      keys:
      - name:abc
        secret: ${random_password.aesgcm_secret.result}
  - identity: {}
EOT
  }
}

When looking at the API object for the cluster template, the secretsEncryptionConfig looks fine:

  "secretsEncryptionConfig": {
    "customConfig": {
      "apiVersion": "apiserver.config.k8s.io/v1",
      "kind": "EncryptionConfiguration",
      "resources": [
        {
          "providers": [
            {
              "aesgcm": {
                "keys": [
                  {
                    "name": "abc",
                    "secret": "123"
                  }
                ]
              }
            },
            {
              "identity": {}
            }
          ],
          "resources": [
            "secrets"
          ]
        }
      ]
    },
    "enabled": true,
    "type": "/v3/schemas/secretsEncryptionConfig"
  },

However the cluster itself seem to have lost the resources value:

{
  "secretsEncryptionConfig": {
    "customConfig": {
      "Resources": null,
      "apiVersion": "apiserver.config.k8s.io/v1",
      "kind": "EncryptionConfiguration"
    },
    "enabled": true,
    "type": "/v3/schemas/secretsEncryptionConfig"
  },
}

I checked that the cluster references the correct cluster_template and cluster_template_revision ids.

The upper case R in Resources in the cluster api object, instead of the lower case r in the cluster template object seems suspicious

And the cluster fails to start and the Rancher UI displays the following error message:

[controlPlane] Failed to bring up Control Plane: [Failed to verify healthcheck: Failed to check https://localhost:6443/healthz for service [kube-apiserver] on host [xxx.xxx.xxx.xxx]: Get "https://localhost:6443/healthz": dial tcp 127.0.0.1:6443: connect: connection refused, log: Error: error while parsing encryption provider configuration file "/etc/kubernetes/ssl/encryption.yaml": error while parsing file: resources: Required value: at least one resources is required]

Indeed when sshing into one of the control plane nodes, I see that:

root@controlplane1:~# more /etc/kubernetes/ssl/encryption.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources: []
jmatsushita commented 3 years ago

It's a reported rancher bug https://github.com/rancher/rancher/issues/31385