search

cyberark / kubeletctl

A client for kubelet
Apache License 2.0
713 stars 81 forks source link

failed to execute `exec` command #17

Closed greenhandatsjtu closed 1 year ago

greenhandatsjtu commented 3 years ago

Summary

I can successfully execute run command, but I failed to execute exec command with error unable to upgrade connection: Unauthorized

Steps to Reproduce

Steps to reproduce the behavior:

  1. login to any node
  2. try execute ls command in a container 
    ./kubeletctl --cacert /etc/kubernetes/pki/ca.crt --cert /var/lib/kubelet/pki/kubelet-client-current.pem --key /var/lib/kubelet/pki/kubelet-client-current.pem -s 10.10.94.147  exec -p hello-world -c hello ls

Expected Results

It should print directory contents under /

Actual Results (including error logs, if applicable)

Failed with error unable to upgrade connection: Unauthorized image

While I can successfully execute run command:

./kubeletctl --cacert /etc/kubernetes/pki/ca.crt --cert /var/lib/kubelet/pki/kubelet-client-current.pem --key /var/lib/kubelet/pki/kubelet-client-current.pem -s 10.10.94.147 run -p hello-world -c hello ls

image

Reproducible

Version/Tag number

Environment setup

Our K8s nodes are running on VM in local machines.

g3rzi commented 3 years ago

Hi,

I tried to reproduce it but for me both commands worked.
Can you please share the settings of the kubelet from 10.10.94.147?
It should be in this file: 

/var/lib/kubelet/config.yaml

I am especially interesting in the configuration that under authentication: and authorization:.

Did you use "hello-world" as the name of the image? Because when I used it, it was in completed status: 

root:~# kubectl run --image hello-world hello-world
pod/hello-world created
root:~# kubectl get pods
NAME          READY   STATUS      RESTARTS   AGE
hello-world   0/1     Completed   0          3s

My test was on:
I also noticed that kubeletctl: 1.7 Kubernetes: v1.18.0 Docker: 19.03.08

My next goal is to test it on Kubernetes version from v1.18.3 and above, and also Docker version from 19.03.14 and above.

To sum up what I need in order to help me to reproduce it:

  1. Content of /var/lib/kubelet/config.yaml (or at least the authentication and authorization fields.
  2. The full kubectl command you used to create the container or the YAML of the container. If this is a sensitive YAML, just write in the sensitive fields
greenhandatsjtu commented 3 years ago

Hi, thanks for your reply!

Here's /var/lib/kubelet/config.yaml of 10.10.94.147

apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/ssl/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
clusterDNS:
- 169.254.25.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

And here's my YAML to create hello-world Pod, it uses nginx:1.19 image:

apiVersion: v1
kind: Pod
metadata:
  name: hello-world
spec:
  containers:
    - name: hello
      image: nginx:1.19
g3rzi commented 3 years ago

Just a small update, my colleague @yanivyakobovich has been able to reproduced it on Kubernetes v1.21.0.
It seems that some change in Kubernetes affects it, so we will need to dig a little bit further.

greenhandatsjtu commented 3 years ago

Great! Glad to hear that! I'll try to figure out the reason too.

g3rzi commented 3 years ago

Hi @greenhandatsjtu,

@yanivyakobovich checked this issue and this is what he found: In the following Kubernetes versions: 1.20.0, 1.18.3 and 1.18.1, when kubelet config set like that:

apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: true
...
authorization:
  mode: AlwaysAllow

Which is insecure configurations, you can use the run and exec command, they work. But when he changed the kubelet config to (like yours):

apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
...
authorization:
  mode: Webhook

He wasn't able to execute run and exec: exec: 

[*] Using KUBECONFIG environment variable
[*] You can ignore it by modifying the KUBECONFIG environment variable, file "~/.kube/config" or use the "-i" switch
unable to upgrade connection: Unauthorized

run:

[*] Using KUBECONFIG environment variable
[*] You can ignore it by modifying the KUBECONFIG environment variable, file "~/.kube/config" or use the "-i" switch
[*] The reponse failed with status: 403
[*] Message: Forbidden (user=system:node:ip-172-31-35-212, verb=create, resource=nodes, subresource=prox

Therefore, it seems that this issue is related to the permissions you are using. Can you please provide us with the permissions of the certificate? (see the command I used below)

Because when I checked on my Kubernetes cluster, it seems that they don't have permissions:

root@ubuntu:~# kubectl --certificate-authority=/etc/kubernetes/pki/ca.crt --client-certificate=/var/lib/kubelet/pki/kubelet-client-current.pem --client-key=/var/lib/kubelet/pki/kubelet-client-current.pem auth can-i --list
Resources                                                       Non-Resource URLs   Resource Names   Verbs
selfsubjectaccessreviews.authorization.k8s.io                   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io                    []                  []               [create]
certificatesigningrequests.certificates.k8s.io/selfnodeclient   []                  []               [create]
                                                                [/api/*]            []               [get]
                                                                [/api]              []               [get]
                                                                [/apis/*]           []               [get]
                                                                [/apis]             []               [get]
                                                                [/healthz]          []               [get]
                                                                [/healthz]          []               [get]
                                                                [/livez]            []               [get]
                                                                [/livez]            []               [get]
                                                                [/openapi/*]        []               [get]
                                                                [/openapi]          []               [get]
                                                                [/readyz]           []               [get]
                                                                [/readyz]           []               [get]
                                                                [/version/]         []               [get]
                                                                [/version/]         []               [get]
                                                                [/version]          []               [get]
                                                                [/version]          []               [get]

NOTE: if you will use the above command, you might receive the error:

Error in configuration:
* client-cert-data and client-cert are both specified for kubernetes-admin. client-cert-data will override.
* client-key-data and client-key are both specified for kubernetes-admin; client-key-data will override

The workaround is to comment the client-certificate-data and client-key-data on your kubeconfig file and try again.

I suppose you don't have permissions either, but I was surprised that the run command worked for you, I would expect it won't work too unless your permissions are different from the one I saw.

greenhandatsjtu commented 3 years ago

Hi @g3rzi , thanks for your reply!

Here's the permissions of the certificate of the node 10.10.94.170 (because node 10.10.94.140 hasn't install kubectl yet)

root@node1:/home/ubuntu# kubectl --certificate-authority=/etc/kubernetes/pki/ca.crt --client-certificate=/var/lib/kubelet/pki/kubelet-client-current.pem --client-key=/var/lib/kubelet/pki/kubelet-client-current.pem auth can-i --list
Resources                                                       Non-Resource URLs   Resource Names   Verbs
nodes/log                                                       []                  []               [*]
nodes/metrics                                                   []                  []               [*]
nodes/proxy                                                     []                  []               [*]
nodes/spec                                                      []                  []               [*]
nodes/stats                                                     []                  []               [*]
leases.coordination.k8s.io                                      []                  []               [create delete get patch update]
csinodes.storage.k8s.io                                         []                  []               [create delete get patch update]
nodes                                                           []                  []               [create get list watch patch update]
certificatesigningrequests.certificates.k8s.io                  []                  []               [create get list watch]
events                                                          []                  []               [create patch update]
pods/eviction                                                   []                  []               [create]
serviceaccounts/token                                           []                  []               [create]
tokenreviews.authentication.k8s.io                              []                  []               [create]
localsubjectaccessreviews.authorization.k8s.io                  []                  []               [create]
selfsubjectaccessreviews.authorization.k8s.io                   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io                    []                  []               [create]
subjectaccessreviews.authorization.k8s.io                       []                  []               [create]
certificatesigningrequests.certificates.k8s.io/selfnodeclient   []                  []               [create]
pods                                                            []                  []               [get list watch create delete]
configmaps                                                      []                  []               [get list watch]
secrets                                                         []                  []               [get list watch]
services                                                        []                  []               [get list watch]
runtimeclasses.node.k8s.io                                      []                  []               [get list watch]
csidrivers.storage.k8s.io                                       []                  []               [get list watch]
persistentvolumeclaims/status                                   []                  []               [get patch update]
                                                                [/api/*]            []               [get]
                                                                [/api]              []               [get]
                                                                [/apis/*]           []               [get]
                                                                [/apis]             []               [get]
                                                                [/healthz]          []               [get]
                                                                [/healthz]          []               [get]
                                                                [/livez]            []               [get]
                                                                [/livez]            []               [get]
                                                                [/openapi/*]        []               [get]
                                                                [/openapi]          []               [get]
                                                                [/readyz]           []               [get]
                                                                [/readyz]           []               [get]
                                                                [/version/]         []               [get]
                                                                [/version/]         []               [get]
                                                                [/version]          []               [get]
                                                                [/version]          []               [get]
endpoints                                                       []                  []               [get]
persistentvolumeclaims                                          []                  []               [get]
persistentvolumes                                               []                  []               [get]
volumeattachments.storage.k8s.io                                []                  []               [get]
nodes/status                                                    []                  []               [patch update]
pods/status                                                     []                  []               [patch update]

And when I try to use kubeletctl to run and exec a pod on node 10.10.94.170, the results are same as before.

root@node1:/home/ubuntu# ./kubeletctl --cacert /etc/kubernetes/pki/ca.crt --cert /var/lib/kubelet/pki/kubelet-client-current.pem --key /var/lib/kubelet/pki/kubelet-client-current.pem -s 10.10.94.170  run -p get-sa -c ubuntu ls
[*] Using KUBECONFIG environment variable
[*] You can ignore it by modifying the KUBECONFIG environment variable, file "~/.kube/config" or use the "-i" switch
app
bin
boot
dev
etc
home
lib
lib32
lib64
libx32
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

root@node1:/home/ubuntu# ./kubeletctl --cacert /etc/kubernetes/pki/ca.crt --cert /var/lib/kubelet/pki/kubelet-client-current.pem --key /var/lib/kubelet/pki/kubelet-client-current.pem -s 10.10.94.170  exec -p get-sa -c ubuntu ls
[*] Using KUBECONFIG environment variable
[*] You can ignore it by modifying the KUBECONFIG environment variable, file "~/.kube/config" or use the "-i" switch
unable to upgrade connection: Unauthorized

image

It's weird, but now I know that there must be something wrong with my configs. I'll continue trying to figure out the reason by myself , thank you guys again! Helps a lot!

g3rzi commented 3 years ago

Hey, thanks for these information!

The certificate have lots of permissions, more than I expected, this is probably the reason that the run command worked and exec didn't. By the way, the only difference between these commands is that the exec opens a stream and the other doesn't. So, everything you do with run you can do with exec and vice versa.

But if you really want to find the reason, we need to find the permissions. My guess, I didn't check it, that if you will add the following permissions it will work:

resources: ["pods/exec"]
verbs: ["create", "get"]

But this is only a guess because I think that these are the permissions that it needs to open a stream.

g3rzi commented 1 year ago

We tried with different credentials:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"name":"kubiscan-clusterrole"},"rules":[{"apiGroups":["*"],"resources":["nodes/spec","nodes/log","nodes/metrics","nodes/proxy","nodes/stats"],"verbs":["*"]}]}
  creationTimestamp: "2022-09-05T13:28:15Z"
  name: kubiscan-clusterrole
  resourceVersion: "315714"
  uid: d9d258b0-16ee-4123-9076-8b13e676257c
rules:
- apiGroups:
  - '*'
  resources:
  - nodes/spec
  - nodes/log
  - nodes/metrics
  - nodes/proxy
  - nodes/stats
  verbs:
  - '*'

And also with:

rules:
- apiGroups:
  - '*'
  resources:
  - '*'

None of them gave us the permissions to use exec. The only thing that helps us is to change the kubelet config (NOT RECOMMENDED): 

apiVersion: kubelet.config.k8s.io/v1beta1 
authentication: 
  anonymous: 
    enabled: true 
    ... 
authorization: 
    mode: AlwaysAllow

Reference:
https://www.deepnetwork.com/blog/2020/01/13/kubelet-api.html
https://9to5answer.com/quot-kubectl-exec-quot-results-in-quot-error-unable-to-upgrade-connection-unauthorized-quot