Provider errors when passing passing GKE endpoint: rpc error: code = Unknown desc = failed to validate provider configuration

[davidalger]

Terraform, Provider, Kubernetes versions

$ terraform -v
Terraform v0.14.6
+ provider registry.terraform.io/hashicorp/archive v2.0.0
+ provider registry.terraform.io/hashicorp/google v3.56.0
+ provider registry.terraform.io/hashicorp/google-beta v3.56.0
+ provider registry.terraform.io/hashicorp/helm v2.0.2
+ provider registry.terraform.io/hashicorp/kubernetes v2.0.2
+ provider registry.terraform.io/hashicorp/kubernetes-alpha v0.2.1
+ provider registry.terraform.io/hashicorp/random v3.0.1

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-14T05:15:04Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.14-gke.1600", GitCommit:"44dfef46993969762b5df37fe3dda47621476708", GitTreeState:"clean", BuildDate:"2021-01-12T09:20:19Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

Affected Resource(s)

• provider "kubernetes-alpha" on GKE

Terraform Configuration Files

# Provides OAuth token for K8S authentication
data "google_client_config" "default" {}

provider "kubernetes" {
  host                   = google_container_cluster.k8s.endpoint
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(google_container_cluster.k8s.master_auth.0.cluster_ca_certificate)
}

provider "kubernetes-alpha" {
  host                   = google_container_cluster.k8s.endpoint
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(google_container_cluster.k8s.master_auth.0.cluster_ca_certificate)
}

provider "helm" {
  kubernetes {
    host                   = google_container_cluster.k8s.endpoint
    token                  = data.google_client_config.default.access_token
    cluster_ca_certificate = base64decode(google_container_cluster.k8s.master_auth.0.cluster_ca_certificate)
  }
}

resource "helm_release" "traefik" {
  namespace  = "kube-system"
  name       = "traefik"
  chart      = "traefik"
  repository = "https://helm.traefik.io/traefik"
  version    = "9.12.0"

  values = [jsonencode({
    # ClusterIP for internal use with container-native external load balancing
    service = {
      type = "ClusterIP"
      annotations = {
        "cloud.google.com/neg"            = jsonencode({ ingress = true })
        "cloud.google.com/backend-config" = jsonencode({ default = "traefik" })
      }
    }

    # Require annotating to avoid conflict with GKE Ingress which will consume all non-annotated ingress objects
    additionalArguments = [
      "--providers.kubernetesingress.ingressclass=traefik",
    ]

    resources = {
      requests = {
        cpu    = "250m"
        memory = "128Mi"
      }
      limits = {
        cpu    = "500m"
        memory = "512Mi"
      }
    }
  })]

  depends_on = [google_container_cluster.k8s]
}

resource "kubernetes_ingress" "traefik" {
  metadata {
    name      = "traefik"
    namespace = "kube-system"
    annotations = {
      "kubernetes.io/ingress.class" = "gce"
    }
  }

  spec {
    backend {
      service_name = "traefik"
      service_port = "web"
    }
  }

  wait_for_load_balancer = true
  depends_on             = [helm_release.traefik]
}

resource "kubernetes_manifest" "traefik_backend_config" {
  provider = kubernetes-alpha

  manifest = {
    "apiVersion" = "cloud.google.com/v1beta1"
    "kind"       = "BackendConfig"
    "metadata" = {
      "name"      = "traefik"
      "namespace" = "kube-system"
    }
    "spec" = {
      "healthCheck" = {
        "port"        = 9000
        "requestPath" = "/ping"
      }
    }
  }

  depends_on = [helm_release.traefik]
}

Steps to Reproduce

1. terraform apply to spin up a working google_container_cluster resource
2. Add above configuration
3. terraform apply

Expected Behavior

The provider should accept the endpoint output as the existing kubernetes and helm providers do, not requiring a fully qualified URL, but actually accepting a hostname as is implied from the name.

Actual Behavior

Error: rpc error: code = Unknown desc = failed to validate provider configuration

Workaround

Prefix the endpoint with https:// fully qualifying it as a URL and bingo, it all works and applies correctly.

provider "kubernetes-alpha" {
  host                   = "https://${google_container_cluster.k8s.endpoint}"
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(google_container_cluster.k8s.master_auth.0.cluster_ca_certificate)
}

Important Factoids

• The cluster is already up and running, i.e. this error is NOT from attempting to use the provider in the same apply as cluster creation.
• google_container_cluster.k8s.endpoint is an IP address string such as 123.123.123.123

References

• This same bug was reported on another issue, but subsequently closed after the OP reported what he thought (incorrectly) may have caused it: https://github.com/hashicorp/terraform-provider-kubernetes-alpha/issues/124#issuecomment-701161889
• Unrelated outside of the error message being identical, so I'm tagging it here so that hopefully others can more easily find the workaround which took me hours to uncover: https://github.com/hashicorp/terraform-provider-kubernetes-alpha/issues/133

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[michalmedvecky]

The workaround does not work for me.

TF 0.14.7 kubernetes-alpha v0.2.1

provider "kubernetes-alpha" {
  cluster_ca_certificate = base64decode(module.eks.certificate_authority.0.data)
  host                   = "https://${module.eks.endpoint}"
  token                  = data.aws_eks_cluster_auth.eks.token
}

[lawliet89]

With 0.3.1, I get a different error. I have opened a new issue https://github.com/hashicorp/terraform-provider-kubernetes-alpha/issues/174