kube-cert-agent-controller: { } failed with: could not find a healthy kube-controller-manager on talos linux

rbuffi commented 3 months ago

When we deploy pinniped concierge with kapp or yaml the kube-cert-agent controller does not appear. We use talos linux 1.7

k logs -n pinniped-concierge pinniped-concierge-69b65f6876-8m5h4:

message":"kube-cert-agent-controller: { } failed with: could not find a healthy kube-controller-manager pod (0 candidates): note that this error is the expected behavior for some cluster types, including most cloud provider clusters (e.g. GKE, AKS, EKS)"} {"level":"info","timestamp":"2024-06-25T20:47:54.078299Z","caller":"k8s.io/client-go@v0.30.2/tools/cache/reflector.go:547$cache.(Reflector).list","message":"k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list v1.PriorityLevelConfiguration: the server could not find the requested resource"}

Can you describe how kube-controller-manager is determined?

k get node --show-labels:

NAME STATUS ROLES AGE VERSION LABELS talos-default-controlplane-1 Ready control-plane 20d v1.28.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=talos-default-controlplane-1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane= talos-default-worker-1 Ready 20d v1.28.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=talos-default-worker-1,kubernetes.io/os=linux

k get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-78f679c54d-ltv54 1/1 Running 2 (88m ago) 20d coredns-78f679c54d-zrvjf 1/1 Running 2 (88m ago) 20d kube-apiserver-talos-default-controlplane-1 1/1 Running 0 87m kube-controller-manager-talos-default-controlplane-1 1/1 Running 1 (88m ago) 87m kube-flannel-qgbxl 1/1 Running 1 (88m ago) 19d kube-flannel-vkstq 1/1 Running 2 (88m ago) 20d kube-proxy-kz8p2 1/1 Running 2 (88m ago) 20d kube-proxy-wsh64 1/1 Running 2 (88m ago) 20d kube-scheduler-talos-default-controlplane-1 1/1 Running 1 (88m ago) 87m

What did you expect to happen? Deployment of kube-cert-agent controller and functioning concierge

We use the latest version of pinniped.

cfryanr commented 3 months ago

Hi @rbuffi, thanks for posting your question.

Can you describe how kube-controller-manager is determined?

The Pinniped controller code is looking for a pod in the kube-system namespace which has the standard label component=kube-controller-manager. Does your controller manager pod have that standard label? You could try adding that label as a workaround.

Here is some additional background that may be useful to other people trying to follow this thread. The Pinniped Concierge will try to auto-detect which strategy it should use to allow it to customize end-user authentication into the cluster.

If it can find the kube-controller-manager pod (which is part of a cluster's control plane) using the search described above, then it will try to deploy a special Pinniped agent pod onto the same node. If the agent pod succeeds, then the Concierge will use it to help authenticate users into that cluster. This is the preferred integration strategy, but it does not work on many Kubernetes distributions, especially those where the control plane is invisible.
- If the above fails at any step, then the Concierge will fall back to another strategy called the "impersonation proxy". By default, it will make a load balancer Service to provide ingress into the Concierge impersonation proxy server (although this is configurable). Then when you use pinniped get kubeconfig for that cluster, it will output a kubeconfig where you are actually talking to the cluster through the Concierge impersonation proxy (not directly to the Kuberntes API server), to allow the Concierge to control authentication of any end users who make requests to the cluster that way.
- You can observe the status of these two strategies on the CredentialIssuer resource.
- If you want to configure the impersonation proxy, you can configure that in the spec of the CredentialIssuer resource to override the defaults.

rbuffi commented 3 months ago

Thank you for your fast response. The kube-controllers where indeed not having the correct label. I tried labeling them but still no agent was. deployed. My goal is to authenticate to kubeapps with keycloak and pinniped. I now have set up the impersonation proxy:

apiVersion: v1

items:

- apiVersion: [config.concierge.pinniped.dev/v1alpha1](http://config.concierge.pinniped.dev/v1alpha1)

  kind: CredentialIssuer

  metadata:

    creationTimestamp: "2024-06-25T14:36:04Z"

    generation: 2

    labels:

      app: pinniped-concierge

    name: pinniped-concierge-config

    resourceVersion: "16012020"

    uid: a6b6b570-311b-4b00-9706-71f44671cfa7

  spec:

    impersonationProxy:

      mode: enabled

      service:

        annotations:

          [service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout](http://service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout): "4000"

        type: LoadBalancer

  status:

    strategies:

    - lastUpdateTime: "2024-06-25T14:36:13Z"

      message: could not find a healthy kube-controller-manager pod (0 candidates)

      reason: CouldNotFetchKey

      status: Error

      type: KubeClusterSigningCertificate

    - frontend:

        impersonationProxyInfo:

          certificateAuthorityData: xxxx

          endpoint: https://x.x.x.x/

        type: ImpersonationProxy

      lastUpdateTime: "2024-06-25T22:41:48Z"

      message: impersonation proxy is ready to accept client connections

      reason: Listening`

And jwtauthenticator:

apiVersion: v1

items:

- apiVersion: [authentication.concierge.pinniped.dev/v1alpha1](http://authentication.concierge.pinniped.dev/v1alpha1)

  kind: JWTAuthenticator

  metadata:

    creationTimestamp: "2024-06-26T00:20:50Z"

    generation: 1

    name: jwt-authenticator

    resourceVersion: "16033939"

    uid: ac12cf5c-228d-494c-9f1f-80044a75f01c

  spec:

    audience: kubeapps

    claims:

      groups: groups

      username: email

    issuer: https://kc.testlab.x.x/realms/kubeapps

    tls:

      certificateAuthorityData: xxxx
kind: List

metadata:

  resourceVersion: ""

With this config i'm able to authenticate to kubeapps with keycloak but after authentication i'm being redirected to the login page. In kubeapps auth-proxy logging I see nothing strange:

[10.244.1.1:45115](http://10.244.1.1:45115/) - 372269b1-2a3d-4de1-88b6-31843b95e5e5 - [user@anonymous.nl](mailto:user@anonymous.nl) [2024/06/26 08:31:24] [AuthSuccess] Authenticated via OAuth2: Session{email:user@anonymous user:93424824-a080-4690-ae1d-8346c40efc0e [PreferredUsername:user@anonymous.nl](mailto:PreferredUsername%3Auser@anonymous.nl) token:true id_token:true created:2024-06-26 08:31:24.585448393 +0000 UTC m=+2920.873606365 expires:2024-06-26 08:36:24.500799825 +0000 UTC m=+3220.788957799 refresh_token:true groups:[kubeapps-admin]}

[10.244.1.1:45115](http://10.244.1.1:45115/) - 372269b1-2a3d-4de1-88b6-31843b95e5e5 - - [2024/06/26 08:31:24] 192.168.210.116 GET - "/oauth2/callback?state=7mtfXVKtt4-AbTYHzCvZIlvAizmJ1CdwH-LIu2rPo_s%3A%2F&session_state=96c2dfdb-3722-4d3d-bb52-e54c3d501829&iss=https%3A%2F%2Fkc.testlab.xxx.local%2Frealms%2Fkubeapps&code=4fa74f08-ca24-4193-8bb1-d0db9b293f4f.96c2dfdb-3722-4d3d-bb52-e54c3d501829.cb382bec-bc96-4750-a889-7e34456c8a8d" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/[126.0.0.0](http://126.0.0.0/) Safari/53

But in the apiserver logging I see the following:

I0626 08:56:41.131411       1 handler.go:232] Adding GroupVersion [identity.concierge.pinniped.dev](http://identity.concierge.pinniped.dev/) v1alpha1 to ResourceManager

I0626 08:56:41.144661       1 handler.go:232] Adding GroupVersion [login.concierge.pinniped.dev](http://login.concierge.pinniped.dev/) v1alpha1 to ResourceManager

E0626 08:57:06.728431       1 controller.go:102] loading OpenAPI spec for "[v1alpha1.identity.concierge.pinniped.dev](http://v1alpha1.identity.concierge.pinniped.dev/)" failed with: failed to download [v1alpha1.identity.concierge.pinniped.dev](http://v1alpha1.identity.concierge.pinniped.dev/): resource not found

I0626 08:57:06.728494       1 controller.go:109] OpenAPI AggregationController: action for item [v1alpha1.identity.concierge.pinniped.dev](http://v1alpha1.identity.concierge.pinniped.dev/): Rate Limited Requeue.

E0626 08:57:06.828889       1 controller.go:102] loading OpenAPI spec for "[v1alpha1.login.concierge.pinniped.dev](http://v1alpha1.login.concierge.pinniped.dev/)" failed with: failed to download [v1alpha1.login.concierge.pinniped.dev](http://v1alpha1.login.concierge.pinniped.dev/): resource not found

 1 authentication.go:73] "Unable to authenticate the request" err="invalid bearer token"

When I try to decode the token as described (https://kubeapps.dev/docs/latest/howto/oidc/oauth2oidc-debugging/) I get the following error:

{"alg":"RS256","typ" : "JWT","kid" : "wkF65vug7ZdfpsKzc5Fpt_qCUHNZo_37uwxhDzoU5v8"}base64: invalid input

In the concierge logging I do not see any token requests. Is it possible to raise the loglevel? And is there any logging for the jwtauthenticator / impersonationproxy? Am I maybe missing some step?

thanks in advance,

Ronald

rbuffi commented 3 months ago

I also tried troubleshooting with the pinniped-cli:

pinniped-cli-windows-amd64.exe login oidc --issuer https://kc.testlab.xxx.local/realms/kubeapps --ca-bundle-data XXXX --client-id kubeapps --enable-concierge --concierge-endpoint https://192.168.x.x --concierge-authenticator-name jwt-authenticator --concierge-authenticator-type jwt --scopes openid,groups,email --concierge-ca-bundle-data xxxx

`Wed, 26 Jun 2024 14:30:07 CEST rest/warnings.go:70 Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.

Result: {"kind":"ExecCredential","apiVersion":"client.authentication.k8s.io/v1beta1","spec":{"interactive":false},"status":{"expirationTimestamp":"2024-06-26T12:35:07Z","clientCertificateData":"-----BEGIN CERTIFICATE-----\nCERTIFICATE\n-----END CERTIFICATE-----\n","clientKeyData":"-----BEGIN PRIVATE KEY-----\nKEY\n-----END PRIVATE KEY-----\n"}}`

So it seems pinniped does the job. I now only have to get the authentication with kubeapps working.

cfryanr commented 3 months ago

So it seems pinniped does the job.

Glad you were able to get it working. I'll close this issue for now but please feel free to keep asking questions.

rbuffi commented 3 months ago

I still got a question... do you know if there is any documentation on pinniped-proxy (bundled with kubeapps) there does not seem to be a lot of documentation on it. It can't figure out how to get ik working with kubeapps...

Regards,

Ronald

cfryanr commented 3 months ago

Hi @rbuffi, only that docs that I can find via Google search. That app was developed by the Kubeapps team, so I'm not too familiar with it. Maybe try reaching out to them for help... https://kubeapps.dev/community/

vmware-tanzu / pinniped

kube-cert-agent-controller: { } failed with: could not find a healthy kube-controller-manager on talos linux #2002