kubernetes client initialization failed: no Auth Provider found for name "oidc"

[kcighon]

Describe the bug

We use kubernetes on tkgi and authenticate against the cluster with sso to create our KUBECONFIG with

• tkgi login -a <tkgihostfqdn> -k -sso
• export KUBECONFIG=~/.kube/<clustername>.yaml
• tkgi get-kubeconfig <clustername> -a <tkgihostfqdn> -k -sso

The resultant KUBECONFIG looks like:

apiVersion: v1
clusters: 
- cluster: 
      certificate-authority-data: xx....
      server: https://<myclusterfqdn>:8443 
  name: <clustername> 
contexts: 
- context: 
      cluster: <clustername> 
      user: <myoidcusername> 
   name: <clustername>
current-context: <clustername>
kind: Config
references: {}
users:
- name: <myoidcusername> 
   user: 
      auth-provider: 
         config: 
            client-id: pks_cluster_client 
            client-secret: "" 
            id-token: xx.... 
            idp-certificate-authority-data: xx.... 
            idp-issuer-url: https://<tkgihostfqdn>:8443/oauth/token
            refresh-token: xx.... 
         name: oidc 

Running terraform apply gives the error:

│ Error: Kubernetes Client 
│ 
│ with flux_bootstrap_git.this, 
│ on bootstrap.tf line 1, in resource "flux_bootstrap_git" "this": 
│ 1: resource "flux_bootstrap_git" "this" { 
│ 
│ kubernetes client initialization failed: no Auth Provider found for name "oidc" 

Steps to reproduce

Our terraform code is as follows:

terraform {
  required_providers {
    flux = {
      source = "fluxcd/flux"
      version = "1.2.3"
    }
  }
}

provider "flux" {
  kubernetes = {  }
  git = {
    url = var.git_url
    branch = var.git_branch

    # configure credentials
  }
}

resource "flux_bootstrap_git" "this" {
  path        = var.cluster_path
  registry   = var.registry
  version    = "v2.2.3"
}

Expected behavior

With one of our TKGI clusters, this works as expected however the KUBECONFIG is token based only, not OIDC

Screenshots and recordings

No response

Terraform and provider versions

Terraform v1.7.3 fluxcd/flux Provider v1.2.3

Terraform provider configurations

terraform {
  required_providers {
    flux = {
      source = "fluxcd/flux"
      version = "1.2.3"
    }
  }
}

provider "flux" {
  kubernetes = {  }
  git = {
    url = var.git_url
    branch = var.git_branch

    # configure credentials
  }
}

flux_bootstrap_git resource

resource "flux_bootstrap_git" "this" {
  path        = var.cluster_path
  registry   = var.registry
  version    = "v2.2.3"
}

Flux version

v2.2.3

Additional context

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Would you like to implement a fix?

None

[swade1987]

@kcighon I have to admit I have never seen a kubeconfig with this block before

 auth-provider: 
         config: 
            client-id: pks_cluster_client 
            client-secret: "" 
            id-token: xx.... 
            idp-certificate-authority-data: xx.... 
            idp-issuer-url: https://<tkgihostfqdn>:8443/oauth/token
            refresh-token: xx.... 
         name: oidc 

When using oidc to connect previously, I have seen the following:

users:
- name: keycloak
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: kubectl
      args:
      - oidc-login
      - get-token
      - --oidc-issuer-url=https://issuer.example.com
      - --oidc-client-id=YOUR_CLIENT_ID
      - --oidc-client-secret=YOUR_CLIENT_SECRET

Are you able to use the kubeconfig snippet you provided an run kubectl commands without issues?

[kcighon]

@kcighon I have to admit I have never seen a kubeconfig with this block before

 auth-provider: 

         config: 

            client-id: pks_cluster_client 

            client-secret: "" 

            id-token: xx.... 

            idp-certificate-authority-data: xx.... 

            idp-issuer-url: https://<tkgihostfqdn>:8443/oauth/token

            refresh-token: xx.... 

         name: oidc 

When using oidc to connect previously, I have seen the following:

users:

- name: keycloak

  user:

    exec:

      apiVersion: client.authentication.k8s.io/v1beta1

      command: kubectl

      args:

      - oidc-login

      - get-token

      - --oidc-issuer-url=https://issuer.example.com

      - --oidc-client-id=YOUR_CLIENT_ID

      - --oidc-client-secret=YOUR_CLIENT_SECRET

Are you able to use the kubeconfig snippet you provided an run kubectl commands without issues?

Hi @swade1987 - kubectl (and the flux cli) work as normal with this KUBECONFIG.

[swade1987]

@kcighon, I'm interested in whether this Kubeconfig works with the upstream Kubernetes Terraform provider. Additionally, can you please provide me with the full configuration block for the flux provider?

[kcighon]

Thanks @swade1987 . I ran a simple test to create a secret and it failed.

The full provider code is as above and makes use of the fact that KUBE_CONFIG_PATH (required as kubernetes provider does not use KUBECONFIG by default) is set to our KUBECONFIG file.

I've used the below TF code to test just the kubernetes provider:

terraform { 
  required_version = ">=1.3.4"
  required_providers {
   kubernetes = { 
    source = "hashicorp/kubernetes"
    version = "2.26.0"
    }
   }
 } 

provider "kubernetes" { }

resource "kubernetes_secret_v1" "common_secrets" {
   metadata { 
     name = "test" 
     namespace = "flux-system"
   }
   data = {
     test = "test"
   }
   type = "Opaque" 
}

There were no errors and the secret was created as defined. The error reported yesterday appears to be specific to the flux_bootstrap_git resource.

[swade1987]

@kcighon, you stated above that you tested creating a secret, which failed. Did you mean it didn't fail?

It seems like its related to https://github.com/fluxcd/terraform-provider-flux/issues/440

[kcighon]

@swade1987 test creating a secret succeeded showing the kubernetes provider with KUBE_CONFIG_PATH set works if the KUBECONFIG file contains auth-provider name oidc.

flux_bootstrap_git with the same setup (ie using KUBE_CONFIG_PATH) fails with the error in the OP.