search

loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.26k stars 398 forks source link

Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden #213

Closed olljanat closed 2 years ago

olljanat commented 2 years ago

Not sure which kind of issues this causes but I noticed when I was testing scenario where k8s with --target-namespace (host cluster is v1.22.3+rke2r1 and vcluster and noticed this error on syncer log:

E1130 15:24:10.731530       1 webhook.go:127] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:vcluster-test-k8s:vc-test" cannot create resource "tokenreview
s" in API group "authentication.k8s.io" at the cluster scope
E1130 15:24:10.731872       1 authentication.go:63] "Unable to authenticate the request" err="[[invalid bearer token, Token has been invalidated], tokenreviews.authentication.k8s.io is forbidden: User \"system:serviceaccount:vcluster-test-k8s:vc-test\" cannot create resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope]"

I see that logic on code https://github.com/loft-sh/vcluster/blob/a76788b12e7349c58874dd49ce4907b6ec1fe86a/pkg/authentication/delegatingauthenticator/delegatingauthenticator.go#L43-L46 so I guess that cluster role system:auth-delegator should be delegated to service account?

FabianKramm commented 2 years ago

@olljanat thanks for creating this issue! Yes this seems odd, looks like some requests cannot be authenticated correctly, however whats strange is that the error message implies the host serviceaccounts does not have these rights, however authentication should happen against the virtual cluster (k3s), so it seems like for some reason the wrong client is used for this request.

Did you notice any specific requests fail?

olljanat commented 2 years ago

Ok. It might be related to #214 too as I did see this on same env with k8s and --target-namespace was in use.

FabianKramm commented 2 years ago

@olljanat should be fixed with v0.5.0-alpha.5, the problem was that for certain requests admission control needs to be checked (pod exec, attach & portforward), but the wrong client could get used for those which resulted in these errors

olljanat commented 2 years ago

Yea. Don't see this error anymore but I still see error like this which does not happen on single namespace solution

E1203 13:28:45.938890       1 controller.go:302] controller-runtime: manager: reconciler group  reconciler kind Node: controller: node: name k8s-test-1 namespace : Reconciler error update node status: get vNode IP: get pod: Pod "test-7686d7474c-5v6zr" not found

This is with v0.5.0-alpha.5

olljanat commented 2 years ago

And one more. Ingress looks to be missing when deploying with --target-namespace even when it is enabled on config.

I think that I will switch to one namespace version for now and test this again later...

FabianKramm commented 2 years ago

@olljanat mhh strange, the error vNode IP: get pod: Pod "test-7686d7474c-5v6zr" not found shouldn't appear anymore on the new version v0.5.0-alpha.6, does this also happen if you create a new cluster with that version?

olljanat commented 2 years ago

My bad. I had --chart-version v0.5.0-alpha.3 parameter still on deployment script. Also ingress deployment issue is most likely problem on my env. Need to investigate it later.