Unauthorized when listing k8s resources in tenant

[StanMichel]

Bug description

I've created a tenant in a clean Kubernetes cluster, by following the docs. Afterwards, I ran the hack/create-user.sh to create a dummy kubeconfig file. With this file exported, I could always list the pods in the newly created tenant. As of today, I get the following error:

Note: This always worked for me until this morning

KUBECONFIG=alice-oil.kubeconfig kubectl get pods
Error from server (Forbidden): pods is forbidden: User "alice" cannot list resource "pods" in API group "" in the namespace "default"

I tested this on a clean Kind & Debian environment.

How to reproduce

Steps to reproduce the behavior:

In a clean Kind environment, I executed the following commands:

1. kubectl apply -f https://raw.githubusercontent.com/clastix/capsule/master/config/install.yaml

2. kubectl create -f - << EOF
   apiVersion: capsule.clastix.io/v1beta2
   kind: Tenant
   metadata:
   name: oil
   spec:
   owners:
   - name: alice
   kind: User
   EOF

   Note: I also tried apiVersion: capsule.clastix.io/v1beta1

3. kubectl get tenants (to verify if the tenant oil is active)

4. ./create-user.sh alice oil

5. KUBECONFIG=alice-oil.kubeconfig kubectl get pods (This results in a 403)

We've tested this on several macos machines. I'm running Sonoma 14.5

Expected behavior

Not being forbidden when I list the pods - or another k8s resource - when logged in as the tenant owner.

Logs

kubectl -n capsule-system logs deploy/capsule-controller-manager
2024/06/06 10:43:49 maxprocs: Updating GOMAXPROCS=1: using minimum allowed GOMAXPROCS
2024-06-06T10:43:49.186Z        INFO    setup   Capsule Version v0.6.2 9c1b948
2024-06-06T10:43:49.186Z        INFO    setup   Build from: https://github.com/projectcapsule/capsule
2024-06-06T10:43:49.186Z        INFO    setup   Build date: 2024-03-28T20:14:03
2024-06-06T10:43:49.186Z        INFO    setup   Go Version: go1.21.8
2024-06-06T10:43:49.186Z        INFO    setup   Go OS/Arch: linux/arm64
2024-06-06T10:43:49.225Z        INFO    controllers.TLS Skipping TLS certificate generation as it is still valid
2024-06-06T10:43:49.225Z        INFO    controllers.TLS Updating caBundle in webhooks and crd
2024-06-06T10:43:49.229Z        INFO    controllers.TLS Updating capsule operator pods
2024-06-06T10:43:49.294Z        INFO    controller-runtime.builder      skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called       {"GVK": "capsule.clastix.io/v1beta1, Kind=Tenant"}
2024-06-06T10:43:49.294Z        INFO    controller-runtime.builder      skip registering a validating webhook, object does not implement admission.Validator or WithValidator wasn't called     {"GVK": "capsule.clastix.io/v1beta1, Kind=Tenant"}
2024-06-06T10:43:49.294Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/convert"}
2024-06-06T10:43:49.294Z        INFO    controller-runtime.builder      Conversion webhook enabled      {"GVK": "capsule.clastix.io/v1beta1, Kind=Tenant"}
2024-06-06T10:43:49.317Z        INFO    setup   skipping setup of Indexer ingress.HostnamePath for object *v1beta1.Ingress      {"error": "failed to get API group resources: unable to retrieve the complete list of server APIs: extensions/v1beta1: the server could not find the requested resource"}
2024-06-06T10:43:49.318Z        INFO    setup   skipping setup of Indexer ingress.HostnamePath for object *v1beta1.Ingress      {"error": "failed to get API group resources: unable to retrieve the complete list of server APIs: networking.k8s.io/v1beta1: the server could not find the requested resource"}
2024-06-06T10:43:49.318Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/pods"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/namespaces"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/ingresses"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/persistentvolumeclaims"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/services"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/tenantresource-objects"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/networkpolicies"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/tenants"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/namespace-owner-reference"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/cordoning"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/nodes"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Registering webhook     {"path": "/defaults"}
2024-06-06T10:43:49.319Z        INFO    setup   starting manager
2024-06-06T10:43:49.319Z        INFO    controller-runtime.metrics      Starting metrics server
2024-06-06T10:43:49.319Z        INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
2024-06-06T10:43:49.319Z        INFO    starting server {"kind": "health probe", "addr": "[::]:10080"}
2024-06-06T10:43:49.319Z        INFO    controller-runtime.webhook      Starting webhook server
2024-06-06T10:43:49.320Z        INFO    controller-runtime.certwatcher  Updated current TLS certificate
2024-06-06T10:43:49.320Z        INFO    controller-runtime.webhook      Serving webhook server  {"host": "", "port": 9443}
2024-06-06T10:43:49.320Z        INFO    controller-runtime.certwatcher  Starting certificate watcher
I0606 10:43:49.425509       1 leaderelection.go:250] attempting to acquire leader lease capsule-system/42c733ea.clastix.capsule.io...
I0606 10:44:07.379255       1 leaderelection.go:260] successfully acquired lease capsule-system/42c733ea.clastix.capsule.io
2024-06-06T10:44:07.381Z        INFO    Starting EventSource    {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "source": "kind source: *v1.Secret"}
2024-06-06T10:44:07.381Z        INFO    Starting EventSource    {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "source": "kind source: *v1.ValidatingWebhookConfiguration"}
2024-06-06T10:44:07.381Z        INFO    Starting EventSource    {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "source": "kind source: *v1.MutatingWebhookConfiguration"}
2024-06-06T10:44:07.381Z        INFO    Starting EventSource    {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "source": "kind source: *v1.CustomResourceDefinition"}
2024-06-06T10:44:07.381Z        INFO    Starting Controller     {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret"}
2024-06-06T10:44:07.382Z        DEBUG   events  capsule-controller-manager-58884475b6-fj56t_b45e0ee1-e567-4ea1-b217-baec1dd034a0 became leader  {"type": "Normal", "object": {"kind":"Lease","namespace":"capsule-system","name":"42c733ea.clastix.capsule.io","uid":"a7b5a715-f021-4f3f-8f05-9e5738900151","apiVersion":"coordination.k8s.io/v1","resourceVersion":"703"}, "reason": "LeaderElection"}
2024-06-06T10:44:07.382Z        INFO    controllers.Rbac        setting up ClusterRoles {"ClusterRole": "capsule-namespace-provisioner"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "source": "kind source: *v1beta2.Tenant"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "source": "kind source: *v1.Namespace"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "source": "kind source: *v1.NetworkPolicy"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "source": "kind source: *v1.LimitRange"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "source": "kind source: *v1.ResourceQuota"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "source": "kind source: *v1.RoleBinding"}
2024-06-06T10:44:07.384Z        INFO    Starting Controller     {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1.ClusterRoleBinding"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "source": "kind source: *v1beta2.CapsuleConfiguration"}
2024-06-06T10:44:07.384Z        INFO    Starting Controller     {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "clusterrole", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRole", "source": "kind source: *v1.ClusterRole"}
2024-06-06T10:44:07.384Z        INFO    Starting Controller     {"controller": "clusterrole", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRole"}
2024-06-06T10:44:07.384Z        INFO    Starting EventSource    {"controller": "endpoints", "controllerGroup": "", "controllerKind": "Endpoints", "source": "kind source: *v1.Endpoints"}
2024-06-06T10:44:07.385Z        INFO    Starting Controller     {"controller": "endpoints", "controllerGroup": "", "controllerKind": "Endpoints"}
2024-06-06T10:44:07.385Z        INFO    Starting EventSource    {"controller": "service", "controllerGroup": "", "controllerKind": "Service", "source": "kind source: *v1.Service"}
2024-06-06T10:44:07.385Z        INFO    Starting Controller     {"controller": "service", "controllerGroup": "", "controllerKind": "Service"}
2024-06-06T10:44:07.385Z        INFO    Starting EventSource    {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2024-06-06T10:44:07.385Z        INFO    Starting Controller     {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod"}
2024-06-06T10:44:07.385Z        INFO    Starting EventSource    {"controller": "endpointslice", "controllerGroup": "discovery.k8s.io", "controllerKind": "EndpointSlice", "source": "kind source: *v1.EndpointSlice"}
2024-06-06T10:44:07.385Z        INFO    Starting Controller     {"controller": "endpointslice", "controllerGroup": "discovery.k8s.io", "controllerKind": "EndpointSlice"}
2024-06-06T10:44:07.385Z        INFO    Starting EventSource    {"controller": "capsuleconfiguration", "controllerGroup": "capsule.clastix.io", "controllerKind": "CapsuleConfiguration", "source": "kind source: *v1beta2.CapsuleConfiguration"}
2024-06-06T10:44:07.385Z        INFO    Starting Controller     {"controller": "capsuleconfiguration", "controllerGroup": "capsule.clastix.io", "controllerKind": "CapsuleConfiguration"}
2024-06-06T10:44:07.385Z        INFO    Starting EventSource    {"controller": "persistentvolume", "controllerGroup": "", "controllerKind": "PersistentVolume", "source": "kind source: *v1.PersistentVolume"}
2024-06-06T10:44:07.385Z        INFO    Starting Controller     {"controller": "persistentvolume", "controllerGroup": "", "controllerKind": "PersistentVolume"}
2024-06-06T10:44:07.386Z        INFO    Starting EventSource    {"controller": "tenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "TenantResource", "source": "kind source: *v1beta2.TenantResource"}
2024-06-06T10:44:07.386Z        INFO    Starting Controller     {"controller": "tenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "TenantResource"}
2024-06-06T10:44:07.386Z        INFO    Starting EventSource    {"controller": "globaltenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "GlobalTenantResource", "source": "kind source: *v1beta2.GlobalTenantResource"}
2024-06-06T10:44:07.386Z        INFO    Starting EventSource    {"controller": "globaltenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "GlobalTenantResource", "source": "kind source: *v1beta2.Tenant"}
2024-06-06T10:44:07.386Z        INFO    Starting Controller     {"controller": "globaltenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "GlobalTenantResource"}
2024-06-06T10:44:07.619Z        INFO    Starting workers        {"controller": "tenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "TenantResource", "worker count": 1}
2024-06-06T10:44:07.619Z        INFO    Starting workers        {"controller": "globaltenantresource", "controllerGroup": "capsule.clastix.io", "controllerKind": "GlobalTenantResource", "worker count": 1}
2024-06-06T10:44:07.620Z        INFO    controllers.Rbac        setting up ClusterRoles {"ClusterRole": "capsule-namespace-deleter"}
2024-06-06T10:44:07.622Z        INFO    controllers.Rbac        setting up ClusterRoleBindings
2024-06-06T10:44:07.623Z        INFO    Starting workers        {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1}
2024-06-06T10:44:07.678Z        INFO    Starting workers        {"controller": "capsuleconfiguration", "controllerGroup": "capsule.clastix.io", "controllerKind": "CapsuleConfiguration", "worker count": 1}
2024-06-06T10:44:07.678Z        INFO    controllers.CapsuleConfiguration        CapsuleConfiguration reconciliation started     {"request.name": "capsule-default"}
2024-06-06T10:44:07.678Z        INFO    controllers.CapsuleConfiguration        CapsuleConfiguration reconciliation finished    {"request.name": "capsule-default"}
2024-06-06T10:44:07.678Z        INFO    Starting workers        {"controller": "endpointslice", "controllerGroup": "discovery.k8s.io", "controllerKind": "EndpointSlice", "worker count": 1}
2024-06-06T10:44:07.680Z        INFO    Starting workers        {"controller": "endpoints", "controllerGroup": "", "controllerKind": "Endpoints", "worker count": 1}
2024-06-06T10:44:07.680Z        INFO    Starting workers        {"controller": "service", "controllerGroup": "", "controllerKind": "Service", "worker count": 1}
2024-06-06T10:44:07.680Z        INFO    Starting workers        {"controller": "persistentvolume", "controllerGroup": "", "controllerKind": "PersistentVolume", "worker count": 1}
2024-06-06T10:44:07.692Z        INFO    Starting workers        {"controller": "tenant", "controllerGroup": "capsule.clastix.io", "controllerKind": "Tenant", "worker count": 1}
2024-06-06T10:44:07.707Z        INFO    controllers.Tenant      Ensuring limit resources count is updated       {"Request.Name": "oil"}
2024-06-06T10:44:07.707Z        INFO    controllers.Tenant      Ensuring all Namespaces are collected   {"Request.Name": "oil"}
2024-06-06T10:44:07.711Z        INFO    controllers.Tenant      Starting processing of Namespaces       {"Request.Name": "oil", "items": 0}
2024-06-06T10:44:07.711Z        INFO    controllers.Tenant      Starting processing of Network Policies {"Request.Name": "oil"}
2024-06-06T10:44:07.711Z        INFO    controllers.Tenant      Starting processing of Limit Ranges     {"Request.Name": "oil", "items": 0}
2024-06-06T10:44:07.711Z        INFO    controllers.Tenant      Starting processing of Resource Quotas  {"Request.Name": "oil", "items": 0}
2024-06-06T10:44:07.711Z        INFO    controllers.Tenant      Ensuring RoleBindings for Owners and Tenant     {"Request.Name": "oil"}
2024-06-06T10:44:07.711Z        INFO    controllers.Tenant      Ensuring Namespace count        {"Request.Name": "oil"}
2024-06-06T10:44:07.714Z        INFO    controllers.Tenant      Tenant reconciling completed    {"Request.Name": "oil"}
2024-06-06T10:44:07.720Z        INFO    Starting workers        {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "worker count": 1}
2024-06-06T10:44:07.720Z        INFO    Starting workers        {"controller": "clusterrole", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRole", "worker count": 1}
2024-06-06T10:44:07.720Z        INFO    Starting workers        {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "worker count": 1}
2024-06-06T10:44:07.721Z        INFO    controllers.Tenant      Ensuring limit resources count is updated       {"Request.Name": "oil"}
2024-06-06T10:44:07.721Z        INFO    controllers.Tenant      Ensuring all Namespaces are collected   {"Request.Name": "oil"}
2024-06-06T10:44:07.724Z        INFO    controllers.TLS Skipping TLS certificate generation as it is still valid        {"Request.Namespace": "capsule-system", "Request.Name": "capsule-tls"}
2024-06-06T10:44:07.724Z        INFO    controllers.TLS Updating caBundle in webhooks and crd   {"Request.Namespace": "capsule-system", "Request.Name": "capsule-tls"}
2024-06-06T10:44:07.724Z        INFO    controllers.Tenant      Starting processing of Namespaces       {"Request.Name": "oil", "items": 0}
2024-06-06T10:44:07.724Z        INFO    controllers.Tenant      Starting processing of Network Policies {"Request.Name": "oil"}
2024-06-06T10:44:07.724Z        INFO    controllers.Tenant      Starting processing of Limit Ranges     {"Request.Name": "oil", "items": 0}
2024-06-06T10:44:07.724Z        INFO    controllers.Tenant      Starting processing of Resource Quotas  {"Request.Name": "oil", "items": 0}
2024-06-06T10:44:07.724Z        INFO    controllers.Tenant      Ensuring RoleBindings for Owners and Tenant     {"Request.Name": "oil"}
2024-06-06T10:44:07.724Z        INFO    controllers.Tenant      Ensuring Namespace count        {"Request.Name": "oil"}
2024-06-06T10:44:07.727Z        INFO    controllers.TLS Updating capsule operator pods  {"Request.Namespace": "capsule-system", "Request.Name": "capsule-tls"}
2024-06-06T10:44:07.727Z        INFO    controllers.Tenant      Tenant reconciling completed    {"Request.Name": "oil"}
2024-06-06T10:44:07.821Z        INFO    controllers.TLS Reconciliation completed, processing back in 4247h58m50.178222333s      {"Request.Namespace": "capsule-system", "Request.Name": "capsule-tls"}

Additional context

• Capsule version: (capsule --version) - the newest
• Helm Chart version: (helm list -n capsule-system) - not installed
• Kubernetes version: (kubectl version):

  kubectl version
  Client Version: v1.29.2
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.29.2

[oliverbaehler]

Hi, there's multiple issues with your approach:

1. We mainly support installation via Helm Chart, for everything else we can't offer support: https://artifacthub.io/packages/helm/projectcapsule/capsule

The command you are expecting to work kubectl get pods rightfully does not work anymore, because the namespace default is not in a tenant, where alice is an owner of.

I recommend you first going through the documentation: https://projectcapsule.dev/docs/tenants/

Alice needs a dedicated namespace:

KUBECONFIG=alice-oil.kubeconfig kubectl create ns oil-dev
KUBECONFIG=alice-oil.kubeconfig kubectl get pod -n oil-dev

[StanMichel]

Creating the dedicated namespaces and listing the pods in that namespace as you suggested didn't work with my configuration, but after a clean installation and installing Capsule throughout helm worked for me!

Thanks for your time to response as quickly as you did :) But might I suggest that the supported approach will be reflected in the documentation? The first option in the Getting started section is to install throughout the YAML file.

[oliverbaehler]

Bad timing from your side :P, we are on the final steps of migrating to the helm chart https://github.com/projectcapsule/capsule/pull/1065

The documentation will also be deprecated