Problem with RBAC permissions

[epstein6]

I am encountering an error trying to run Bridge to Kubernetes on our kubernetes cluster. After configuring and running it on my pod, I get this error:

##################### Environment started. #############################################################
Run /var/folders/0c/f0sfvkkn1wd75wy5dytz7vp4000w4s/T/tmp-43007rERTeMavExAN.env.cmd in your existing console to also get connected.
Stopping workload and cleaning up...
Restore: Deployment 'python-deployment' restored to original state with image '<registry>/public/python-helloworld'.
Bridge to Kubernetes command terminated successfully.
Restoring Deployment patch ...
Restore: Deployment 'python-deployment' restored to original state with image '<registry>/public/python-helloworld'.
Failed to deploy remote restore job for pod deployment 'epstein6/python-deployment' with error: Encountered permissions error. To see which permissions you need to run Bridge To Kubernetes, visit https://aka.ms/bridge-to-k8s-rbac. Error: Forbidden: roles.rbac.authorization.k8s.io "lpkrestorationjob-role-v2" is forbidden: user "system:serviceaccount:epstein6:epstein6-user" (groups=["system:serviceaccounts" "system:serviceaccounts:epstein6" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["extensions"], Resources:["statefulsets"], Verbs:["get" "list" "update" "patch"]}
Stopping workload and cleaning up...
Failed to establish a connection. Error: Failed to deploy remote restore job for pod deployment 'epstein6/python-deployment' with error: Encountered permissions error. To see which permissions you need to run Bridge To Kubernetes, visit https://aka.ms/bridge-to-k8s-rbac. Error: Forbidden: roles.rbac.authorization.k8s.io "lpkrestorationjob-role-v2" is forbidden: user "system:serviceaccount:epstein6:epstein6-user" (groups=["system:serviceaccounts" "system:serviceaccounts:epstein6" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["extensions"], Resources:["statefulsets"], Verbs:["get" "list" "update" "patch"]}

I should be admin as described in the documentation, so I'm not sure what could be wrong:

- apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    creationTimestamp: "2021-08-03T17:05:59Z"
    name: epstein6-sa
    namespace: epstein6
    resourceVersion: "8570032"
    selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/epstein6/rolebindings/epstein6-sa
    uid: 546ca891-f4f1-4a98-87f7-006421004427
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: admin
  subjects:
  - kind: ServiceAccount
    name: epstein6-user
    namespace: epstein6

Any help would be appreciated.

[rakeshvanga]

@epstein6 As the documentation states there are no subjects for user or group mentioned in your RoleBinding. Bridge To Kubernetes uses kubectl to connect to the cluster on your behalf and a RoleBinding for user or group should be specified. Can you try adding user or admin subjects to the RoleBinding?

[epstein6]

Thanks for the quick response. We use the epstein6-user ServiceAccount in our kubeconfig, so that should be correct. We did some more investigation, and it appears that our admin rolebinding doesn't include all the needed permissions for roles and rolebindings. After fixing that, it appears to work. It is supposed to work with the default admin ClusterRole, right?

[rakeshvanga]

@epstein6 Yes, it should work with the default admin ClusterRole.

[awangptc]

The default clusterrole/admin appears to only have these permissions against the extensions api group:

- apiGroups:
  - extensions
  resources:
  - daemonsets
  - deployments
  - deployments/rollback
  - deployments/scale
  - ingresses
  - networkpolicies
  - replicasets
  - replicasets/scale
  - replicationcontrollers/scale
  verbs:
  - create
  - delete
  - deletecollection
  - patch
  - update
- apiGroups:
  - extensions
  resources:
  - daemonsets
  - daemonsets/status
  - deployments
  - deployments/scale
  - deployments/status
  - ingresses
  - ingresses/status
  - networkpolicies
  - replicasets
  - replicasets/scale
  - replicasets/status
  - replicationcontrollers/scale
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - ingresses/status
  verbs:
  - update

So it does appear that the default admin ClusterRole is missing some permissions here. Is there any work to clarify the requirement?

[amsoedal]

Hi @awangptc, at the time that we were investigating RBAC guidance, we confirmed that having the admin role in the namespace was sufficient for being able to run Bridge. Are you running into a specific issue?

[awangptc]

@amsoedal See the original post on this issue. I'm seeing the exact same thing

Failed to deploy remote restore job for pod deployment 'epstein6/python-deployment' with error: Encountered permissions error. To see which permissions you need to run Bridge To Kubernetes, visit https://aka.ms/bridge-to-k8s-rbac. Error: Forbidden: roles.rbac.authorization.k8s.io "lpkrestorationjob-role-v2" is forbidden: user "system:serviceaccount:epstein6:epstein6-user" (groups=["system:serviceaccounts" "system:serviceaccounts:epstein6" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["extensions"], Resources:["statefulsets"], Verbs:["get" "list" "update" "patch"]}
Stopping workload and cleaning up...
Failed to establish a connection. Error: Failed to deploy remote restore job for pod deployment 'epstein6/python-deployment' with error: Encountered permissions error. To see which permissions you need to run Bridge To Kubernetes, visit https://aka.ms/bridge-to-k8s-rbac. Error: Forbidden: roles.rbac.authorization.k8s.io "lpkrestorationjob-role-v2" is forbidden: user "system:serviceaccount:epstein6:epstein6-user" (groups=["system:serviceaccounts" "system:serviceaccounts:epstein6" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["extensions"], Resources:["statefulsets"], Verbs:["get" "list" "update" "patch"]}

If you look at the clusterrole/admin permissions I posted, those permissions on stateful set are not granted to that role, so there is a disconnect somewhere.

[amsoedal]

@awangptc OK thanks for clarifying. What is the version on your K8s cluster? I see from the StatefulSet documentation that the object has moved to the "apps" api group, rather than "extensions". It looks like the admin role does support interacting with StatefulSets, but not under the "extension" api group anymore:

> kubectl describe clusterrole admin
Name:         admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources                                       Non-Resource URLs  Resource Names  Verbs
  ---------                                       -----------------  --------------  -----
  statefulsets.apps/scale                         []                 []              [create delete deletecollection patch update get list watch]
  statefulsets.apps                               []                 []              [create delete deletecollection patch update get list watch]

It looks like we may need to do a release to remove the reference on our side.

[amsoedal]

@awangptc I've logged a bug to fix this on our side. In the meantime, can you please apply this clusterrole (it should get aggregated to the admin role) as a workaround?

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: extensions-admin
  labels:
    # Add these permissions to the "admin" default role
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["extensions"]
  resources: ["statefulsets"]
  verbs: ["get", "list", "update", "patch"]