Kapp-controller will not work properly in Kubernetes 1.24.

GrahamDumpleton commented 2 years ago

I raised this in a private Slack channel discussion but since not seen any issue created here by others since that discussion, creating one.

What steps did you take:

Install kapp-controller in Kubernetes 1.24.

Try installing a package. Installation fails with the error:

kctrl: Error: packageinstall/xxx (packaging.carvel.dev/v1alpha1) namespace: xxx:
  Reconciling:
    Preparing kapp:
      Expected to find one service account token secret, but found none. Reconcile failed:
        Error (see .status.usefulErrorMessage for details)

What happened:

The problem relates to:

That is, in Kubernetes 1.24, an API token secret is no longer automatically created and listed as a secret against a service account. For in-cluster API tokens for pods a new mechanism is now used to allow token refreshes every hours.

A service account therefore now looks like:

apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-05-13T03:34:02Z"
  name: default
  namespace: lab-k8s-fundamentals-w01-s001
  resourceVersion: "1744"
  uid: ff853727-1989-4f49-b5ab-a0e1ec51944c

Note how there is no secrets property like before.

It seems kapp-controller would look up the API token secret name in secrets of the service account and use the token in the linked secret to do stuff. Since this no longer exists, kapp-controller fails with the error shown above.

What did you expect:

It works.

Anything else you would like to add:

This will likely have implications for where a distinct service account is provided by a user when installing a package, rather than the default one used. For the default, kapp-controller can do something else whereby it has created an API token secret of its own to use and linked it up to the default service account used.

For a user providing a service account that will no longer be sufficient, unless you tell users to also create a separate API token secret linked to the service account. What then needs to happen is that kapp-controller needs to search for any secret in the same namespace with the kubernetes.io/service-account.name annotation giving the name of the service account it is for.

To avoid that, it may or may not be possible to grant RBAC to kapp-controller to create API token secrets and create them for the user. The question is whether that is a good idea and raises security concerns and thus whether force it on the users to create the API secret for the service account they nominate.

So basically it is going to be messy. :-(

Environment:

Kubernetes version (use kubectl version)

1.24

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

stealthybox commented 2 years ago

This is how we solve this by impersonating ServiceAccounts /w Flux via User Impersonation

This means that you don't even need a BearerToken for the account as long as the kapp-controller ServiceAccount has ClusterRoleBinding to a ClusterRole /w the impersonate verb for SA's

As long as we have a way to change the client configuration via Go, this is easy. This needs to be done for every tool that kapp-controller uses that might talk to the k8s cluster.

If we need to do this by exec'ing a CLI that is using client-go, there is a bug that I've been meaning to fix for years with the heuristic in-cluster config that we can prioritize. I have a POC fix, but the code-path in client-go is confusing/hard-to-patch.

A workaround by writing a kube-config within the controller-filesystem to circumvent the heuristics is also perfectly usable without the bug fix.

neil-hickey commented 2 years ago

This should be fixed as part of https://github.com/vmware-tanzu/carvel-kapp-controller/pull/695

We opted for a TokenRequest by KC instead of user impersonation.

carvel-dev / kapp-controller

Kapp-controller will not work properly in Kubernetes 1.24. #687