Misconfigured kubeletctl or the port config is ignored

ryuzakyl commented 3 years ago

Summary

Due to a possible misconfiguration on my side of kubeletctl or perhaps another reason, the port being used to comunicate with the kubelet API is not correct. The port being used (39261) is the cluster port specified on my kubeconfig file (see Environment setup section).

Steps to Reproduce

Steps to reproduce the behavior:

Download the precompiled kubeletctl binary with:

$ curl -LO https://github.com/cyberark/kubeletctl/releases/download/v1.6/kubeletctl_linux_amd64 && chmod a+x ./kubeletctl_linux_amd64 && mv ./kubeletctl_linux_amd64 /usr/local/bin/kubeletctl

Try to check worker node kubelet's health:
```
$ kubeletctl -s 172.18.0.3 healthz
```
:eyes: NOTE: The target worker node has the IP 172.18.0.3.

Expected Results

Get the proper output from the kubelet. In this case the endpoint tested was healthz. This is the output obtained using curl instead of kubeletctl:

$ curl -k https://172.18.0.2:10250/healthz
ok

Actual Results (including error logs, if applicable)

Using the default port for kubelet (port 10250) or setting it manually both result in the wrong address being used.

With the default port:

$ kubeletctl -s 172.18.0.3 healthz
[*] Using KUBECONFIG environment variable
[*] You can ignore it by modifying the KUBECONFIG environment variable, file "~/.kube/config" or use the "-i" switch
[*] Failed to run HTTP request with error: Get "https://172.18.0.3:39261/healthz/": dial tcp 172.18.0.3:39261: connect: connection refused

$ kubeletctl -s 172.18.0.3 --port 10250 healthz
[*] Using KUBECONFIG environment variable
[*] You can ignore it by modifying the KUBECONFIG environment variable, file "~/.kube/config" or use the "-i" switch
[*] The reponse failed with status: 404
[*] Message: 404 page not found

Reproducible

[x] Always
[ ] Sometimes
[ ] Non-Reproducible

Version/Tag number

Product version is the following:

$ kubeletctl version
[*] Using KUBECONFIG environment variable
[*] You can ignore it by modifying the KUBECONFIG environment variable, file "~/.kube/config" or use the "-i" switch

 _           _           _                         _  
| |         | |         | |         _          _  | | 
| |  _ _   _| |__  _____| | _____ _| |_ ____ _| |_| | 
| |_/ ) | | |  _ \| ___ | || ___ (_   _) ___|_   _) | 
|  _ (| |_| | |_) ) ____| || ____| | |( (___  | |_| | 
|_| \_)____/|____/|_____)\_)_____)  \__)____)  \__)\_)

Author: Eviatar Gerzi
Version: 1.6

Environment setup

Running on local development box:

$ uname -a
Linux <box-name> 5.0.0-32-generic #34~18.04.2-Ubuntu SMP Thu Oct 10 10:36:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes version and cluster info:

$ kubectl cluster-info 
Kubernetes master is running at https://127.0.0.1:39261
KubeDNS is running at https://127.0.0.1:39261/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Config file pointed by $KUBECONFIG env var:

$ cat $KUBECONFIG
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: BASE64_CERTIFICATE_AUTHORITY_DATA
    server: https://127.0.0.1:39261
  name: kind-cluster-name
contexts:
- context:
    cluster: kind-cluster-name
    user: kind-cluster-name
  name: kind-cluster-name
current-context: kind-cluster-name
kind: Config
preferences: {}
users:
- name: kind-cluster-name
  user:
    client-certificate-data: BASE64_ENCODED_CERTIFICATE_DATA
    client-key-data: BASE64_ENCODED_KEY_DATA

Additional Information

The kubernetes flavor used for this scenario is KinD

The extra configuration for the worker nodes is the following:

- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        anonymous-auth: "true"
        authorization-mode: "AlwaysAllow"

This is mainly to allow unauthenticated requests to the kubelet api.

g3rzi commented 3 years ago

@ryuzakyl Thank you for the informative response! I see the problem, I will fix it. I will update you once it will be ready.

g3rzi commented 3 years ago

@ryuzakyl Can you try:

kubeletctl -s 172.18.0.3 -i healthz

There is the -i switch that ignores the config file.

ryuzakyl commented 3 years ago

I had also previously tried with the -i switch, but didn't include it on the initial bug report. I suspect the URL being queried is not the proper one and that's why I'm getting the HTTP 404.

$ kubeletctl -s 172.18.0.3 -i healthz
[*] The reponse failed with status: 404
[*] Message: 404 page not found

It would be nice to have some sort of verbose mode (-v or -vv, etc.) to know which URLs are being queried and thus, having a better understanding of the kubelet API.

g3rzi commented 3 years ago

I see what is the problem. I debugged it and if there is not input arguments it creates a url like that: https://<node_ip>:10250/healthz/ instead of https://<node_ip>:10250/healthz/ (without the last /).

Can you please try a differnt command? Like that:

kubeletctl -s 172.18.0.3 -i pods

Does this commands works?

Regarding the verbose, this is a great idea, I will add it to the TODO list.

g3rzi commented 3 years ago

You can try version 1.7:
https://github.com/cyberark/kubeletctl/releases/tag/v1.7

ryuzakyl commented 3 years ago

It works!!! :thumbsup: :ok_hand: :clap:

Healthcheck:

$ kubeletctl -s 172.18.0.3 -i healthz
ok

Node pods:

$ kubeletctl -s 172.18.0.3 -i pods
┌─────────────────────────────────────────────────────────┐
│                    Pods from Kubelet                    │
├───┬─────────────────────────┬─────────────┬─────────────┤
│   │ POD                     │ NAMESPACE   │ CONTAINERS  │
├───┼─────────────────────────┼─────────────┼─────────────┤
│ 1 │ kube-proxy-6jtcx        │ kube-system │ kube-proxy  │
│   │                         │             │             │
├───┼─────────────────────────┼─────────────┼─────────────┤
│ 2 │ kindnet-886fl           │ kube-system │ kindnet-cni │
│   │                         │             │             │
├───┼─────────────────────────┼─────────────┼─────────────┤
│ 3 │ coredns-74ff55c5b-r5llh │ kube-system │ coredns     │
│   │                         │             │             │
└───┴─────────────────────────┴─────────────┴─────────────┘

kubelet config:

$ kubeletctl -s 172.18.0.3 -i configz
{
  "kubeletconfig": {
    "enableServer": true,
    "staticPodPath": "/etc/kubernetes/manifests",
    "syncFrequency": "1m0s",
    "fileCheckFrequency": "20s",
    "httpCheckFrequency": "20s",
    "address": "0.0.0.0",
    "port": 10250,
    "tlsCertFile": "/var/lib/kubelet/pki/kubelet.crt",
    "tlsPrivateKeyFile": "/var/lib/kubelet/pki/kubelet.key",
    "rotateCertificates": true,
    "authentication": {
      "x509": {
        "clientCAFile": "/etc/kubernetes/pki/ca.crt"
      },
      "webhook": {
        "enabled": true,
        "cacheTTL": "2m0s"
      },
      "anonymous": {
        "enabled": true
      }
    },
    "authorization": {
      "mode": "AlwaysAllow",
      "webhook": {
        "cacheAuthorizedTTL": "5m0s",
        "cacheUnauthorizedTTL": "30s"
      }
    },
    "registryPullQPS": 5,
    "registryBurst": 10,
    "eventRecordQPS": 5,
    "eventBurst": 10,
    "enableDebuggingHandlers": true,
    "healthzPort": 10248,
    "healthzBindAddress": "127.0.0.1",
    "oomScoreAdj": -999,
    "clusterDomain": "cluster.local",
    "clusterDNS": ["10.96.0.10"],
    "streamingConnectionIdleTimeout": "4h0m0s",
    "nodeStatusUpdateFrequency": "10s",
    "nodeStatusReportFrequency": "5m0s",
    "nodeLeaseDurationSeconds": 40,
    "imageMinimumGCAge": "2m0s",
    "imageGCHighThresholdPercent": 100,
    "imageGCLowThresholdPercent": 80,
    "volumeStatsAggPeriod": "1m0s",
    "cgroupRoot": "/kubelet",
    "cgroupsPerQOS": true,
    "cgroupDriver": "cgroupfs",
    "cpuManagerPolicy": "none",
    "cpuManagerReconcilePeriod": "10s",
    "topologyManagerPolicy": "none",
    "topologyManagerScope": "container",
    "runtimeRequestTimeout": "2m0s",
    "hairpinMode": "promiscuous-bridge",
    "maxPods": 110,
    "podPidsLimit": -1,
    "resolvConf": "/etc/resolv.conf",
    "cpuCFSQuota": true,
    "cpuCFSQuotaPeriod": "100ms",
    "nodeStatusMaxImages": 50,
    "maxOpenFiles": 1000000,
    "contentType": "application/vnd.kubernetes.protobuf",
    "kubeAPIQPS": 5,
    "kubeAPIBurst": 10,
    "serializeImagePulls": true,
    "evictionHard": {
      "imagefs.available": "0%",
      "nodefs.available": "0%",
      "nodefs.inodesFree": "0%"
    },
    "evictionPressureTransitionPeriod": "5m0s",
    "enableControllerAttachDetach": true,
    "makeIPTablesUtilChains": true,
    "iptablesMasqueradeBit": 14,
    "iptablesDropBit": 15,
    "failSwapOn": false,
    "containerLogMaxSize": "10Mi",
    "containerLogMaxFiles": 5,
    "configMapAndSecretChangeDetectionStrategy": "Watch",
    "enforceNodeAllocatable": ["pods"],
    "volumePluginDir": "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/",
    "providerID": "kind://docker/tfm-k8s/tfm-k8s-worker",
    "logging": {
      "format": "text"
    },
    "enableSystemLogHandler": true,
    "shutdownGracePeriod": "0s",
    "shutdownGracePeriodCriticalPods": "0s"
  }
}

cyberark / kubeletctl