Fail to fetch updated ~/.kube/config

[jndeverteuil]

Hi, I'd like to report an issue when we create a ~/.kube/config, the provider will not get the latest file.

In my case, with Terraform, I am:

1. Creating a K3s Master Node (on Proxmox)
2. Fetching the generated kube config
3. Creating a local copy of the kube config

provider "kubectl" {
  config_path = pathexpand("~/.kube/config")
}

resource "proxmox_vm_qemu" "k3s_master" {
...
...
}

data "external" "k3s_kubeconfig" {
  depends_on = [proxmox_vm_qemu.k3s_master]

  program = [
    "/usr/bin/ssh",
    "-o UserKnownHostsFile=/dev/null",
    "-o StrictHostKeyChecking=no",
    "${local.master_node_settings.user}@${local.master_node_ips[0]}",
    "echo '{\"kubeconfig\":\"'$(sudo cat /etc/rancher/k3s/k3s.yaml | base64)'\"}'"
  ]
}

resource "local_sensitive_file" "k3s_kubeconfig" {
  depends_on = [data.external.k3s_kubeconfig]

  content  = replace(base64decode(replace(data.external.k3s_kubeconfig.result.kubeconfig, " ", "")), "server: https://127.0.0.1:6443", "server: https://${local.master_node_ips[0]}:6443")
  filename = pathexpand("~/.kube/config")
}

resource "kubectl_manifest" "test-configmap" {
  depends_on = [local_sensitive_file.k3s_kubeconfig]
  yaml_body = <<YAML
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: test-configmap
  labels:
    app: test-configmap
data:
  test.yml: |-
    this is a test
YAML
}

This will fail:

Error: test-configmap failed to create kubernetes rest client for update of resource: Get "http://localhost/api?timeout=32s": dial tcp 127.0.0.1:80: connect: connection refused

It seems like the kubectl provider initialize itself with the non-existing ~/.kube/config and default to localhost instead of recognizing the updated config file.

If I run terraform apply a second time, then it will work:

kubectl_manifest.test-configmap: Creating...
kubectl_manifest.test-configmap: Creation complete after 0s [id=/api/v1/namespaces/default/configmaps/test-configmap]