Inconsistent results with kubectl and curl for SelfSubjectAccessReview api calls

[lake-of-dreams]

What happened:

1. We create a kind cluster 18.8 version.
2. Setup a service account

   apiVersion: v1
   kind: ServiceAccount
   metadata:
   name: myserviceaccount
   namespace: default

3. Created an impersonator clusterrole that allows impersonating users and groups

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
   name: impersonate-myserviceaccount
   rules:
   - apiGroups: [""]
   resources: ["users", "groups"]
   verbs: ["impersonate"]

4. Assign this role to myserviceaccount

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
   name: impersonate-myserviceaccount
   roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: impersonate-myserviceaccount
   subjects:
   - kind: ServiceAccount
   name: myserviceaccount
   namespace: default

5. Now we wanted to test if this serviceaccount can only impersonate users and groups as assigned by RBAC above and can not impersonate any other service account. Similar to doing

   kubectl auth can-i impersonate serviceaccounts --token=<token-created-in-secret-for-myserviceaccount> --v=10

6. When we execute the command above, we get allowed as true meaning that the serviceaccount can impersonate other serviceaccounts and which would mean our RBAC is not setup correctly.

   
   I0504 18:01:30.495678   13926 request.go:1068] Request Body: {"kind":"SelfSubjectAccessReview","apiVersion":"authorization.k8s.io/v1","metadata":{"creationTimestamp":null},"spec":{"resourceAttributes":{"namespace":"default","verb":"impersonate","resource":"serviceaccounts"}},"status":{"allowed":false}}
   I0504 18:01:30.495895   13926 round_trippers.go:423] curl -k -v -XPOST  -H "User-Agent: kubectl/v1.18.2 (darwin/amd64) kubernetes/52c56ce" -H "Accept: application/json, */*" -H "Authorization: Bearer <redacted-token-created-in-secret-for-myserviceaccount>" -H "Content-Type: application/json" 'https://127.0.0.1:58114/apis/authorization.k8s.io/v1/selfsubjectaccessreviews'
   I0504 18:01:30.804568   13926 round_trippers.go:443] POST https://127.0.0.1:58114/apis/authorization.k8s.io/v1/selfsubjectaccessreviews 201 Created in 308 milliseconds
   I0504 18:01:30.804674   13926 round_trippers.go:449] Response Headers:
   I0504 18:01:30.804728   13926 round_trippers.go:452]     Cache-Control: no-cache, private
   I0504 18:01:30.804761   13926 round_trippers.go:452]     Content-Type: application/json
   I0504 18:01:30.804785   13926 round_trippers.go:452]     Content-Length: 497
   I0504 18:01:30.804824   13926 round_trippers.go:452]     Date: Tue, 04 May 2021 12:31:30 GMT
   I0504 18:01:30.804942   13926 request.go:1068] Response Body: {"kind":"SelfSubjectAccessReview","apiVersion":"authorization.k8s.io/v1","metadata":{"creationTimestamp":null,"managedFields":[{"manager":"kubectl","operation":"Update","apiVersion":"authorization.k8s.io/v1","time":"2021-05-04T12:31:30Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:resourceAttributes":{".":{},"f:namespace":{},"f:resource":{},"f:verb":{}}}}}]},"spec":{"resourceAttributes":{"namespace":"default","verb":"impersonate","resource":"serviceaccounts"}},"status":{"allowed":true}}

7. However if we execute the curl command directly printed above, it  gives the expected result
```bash
curl -k -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.18.2 (darwin/amd64) kubernetes/52c56ce" -H "Authorization: Bearer <redacted-token-created-in-secret-for-myserviceaccount>" 'https://127.0.0.1:58114/apis/authorization.k8s.io/v1/selfsubjectaccessreviews' -d '{"kind":"SelfSubjectAccessReview","apiVersion":"authorization.k8s.io/v1","metadata":{"creationTimestamp":null},"spec":{"resourceAttributes":{"namespace":"default","verb":"impersonate","resource":"serviceaccounts"}},"status":{"allowed":false}}'

* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7ff236808200)
> POST /apis/authorization.k8s.io/v1/selfsubjectaccessreviews HTTP/2
> Host: 127.0.0.1:58114
> Accept: application/json, */*
> Content-Type: application/json
> User-Agent: kubectl/v1.18.2 (darwin/amd64) kubernetes/52c56ce
> Authorization: Bearer <redacted-token-created-in-secret-for-myserviceaccount>
> Content-Length: 242
> 
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 201 
< cache-control: no-cache, private
< content-type: application/json
< content-length: 498
< date: Tue, 04 May 2021 12:40:07 GMT
< 
{"kind":"SelfSubjectAccessReview","apiVersion":"authorization.k8s.io/v1","metadata":{"creationTimestamp":null,"managedFields":[{"manager":"kubectl","operation":"Update","apiVersion":"authorization.k8s.io/v1","time":"2021-05-04T12:40:07Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:resourceAttributes":{".":{},"f:namespace":{},"f:resource":{},"f:verb":{}}}}}]},"spec":{"resourceAttributes":{"namespace":"default","verb":"impersonate","resource":"serviceaccounts"}},"status":{"allowed":false}}
* Connection #0 to host 127.0.0.1 left intact
* Closing connection 0

allowed is false which is expected result.

8. We tried the same on other hosted clusters on Oracle OCI but could not reproduce this issue. However on kind its reproducible every time, i.e. with kubectl the result is "allowed":true and on a subsequent curl command with same headers and body , its "allowed" : false

What you expected to happen: Since the same api is being called from both kubectl and curl and with same headers/body , it should return same result.

How to reproduce it (as minimally and precisely as possible): Create the serviceaccount, cluster role and cluster role binding and try to execute kubectl followed by curl as mentioned above

Anything else we need to know?: This works consistently on other hosted Kubernetes clusters and issue is only on kind

Environment:

• kind version: (use kind version): kind v0.10.0 go1.15.7 darwin/amd64 but the cluster was created using v1.18.8

  $ kind create cluster --config - <<EOF
  kind: Cluster
  apiVersion: kind.x-k8s.io/v1alpha4
  nodes:
  - role: control-plane
  image: kindest/node:v1.18.8@sha256:f4bcc97a0ad6e7abaf3f643d890add7efe6ee4ab90baeb374b4f41a4c95567eb
  kubeadmConfigPatches:
    - |
      kind: ClusterConfiguration
      apiServer:
        extraArgs:
          "service-account-issuer": "kubernetes.default.svc"
          "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
  EOF

• Kubernetes version: (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-09-14T07:44:34Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Docker version: (use docker info):

  
  bash-3.2$ docker info
  Client:
  Context:    default
  Debug Mode: false
  Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.6.0)

Server: Containers: 11 Running: 3 Paused: 0 Stopped: 8 Images: 22 Server Version: 20.10.5 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec init version: de40ad0 Security Options: seccomp Profile: default Kernel Version: 5.10.25-linuxkit Operating System: Docker Desktop OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 1.941GiB Name: docker-desktop ID: 33ZG:JI56:DXHW:46UM:VKBQ:HQLJ:B5FL:WFAV:O2NU:RDZI:4GGW:A4HK Docker Root Dir: /var/lib/docker Debug Mode: false HTTP Proxy: http.docker.internal:3128 HTTPS Proxy: http.docker.internal:3128 Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

- OS (e.g. from `/etc/os-release`):
```bash
macOS Catalina
v10.15.7

[BenTheElder]

KIND doesn't implement RBAC etc. What you are seeing is a vanilla unpatched kubernetes 1.18.8 API server from upstream which is where all of this is implemented.

Also worth noting: the supported image we supply for 1.18.x with kind v0.10.0 is 1.18.15, see the release notes.

This works consistently on other hosted Kubernetes clusters and issue is only on kind

It would be helpful to be more specific about this.

[enj]

kubectl auth can-i impersonate serviceaccounts --token=<token-created-in-secret-for-myserviceaccount> --v=10

That command is running as the cluster-admin certificate in your kubeconfig which has precedence over the SA token.

[enj]

xref: https://github.com/kubernetes/kubernetes/issues/99603

cc @ankeesler

[lake-of-dreams]

kubectl auth can-i impersonate serviceaccounts --token=<token-created-in-secret-for-myserviceaccount> --v=10

That command is running as the cluster-admin certificate in your kubeconfig which has precedence over the SA token.

Can you please suggest an alternative way so that the token will take precedence? @enj

[enj]

@lake-of-dreams create a new context in your kubeconfig that uses the SA token:

kubectl config get-clusters # figure out what value to set for KIND_CLUSTER_NAME
kubectl config set-credentials sa_token --token=SA_TOKEN_HERE
kubectl config set-context sa_token_ctx --cluster=KIND_CLUSTER_NAME --user=sa_token
kubectl config use-context sa_token_ctx
kubectl auth can-i impersonate serviceaccounts # no need to specify --token now

[lake-of-dreams]

@enj I followed the steps you suggested please but I get following error on last line

bash-3.2$ kubectl auth can-i impersonate serviceaccounts
error: You must be logged in to the server (Unauthorized)

[enj]

That error message means that you failed to authenticate. Perhaps the token has expired.

[lake-of-dreams]

Thanks a lot @enj , it worked. Thanks much for all the help.

[BenTheElder]

Thanks @enj!