search

NimbleArchitect / kubectl-ice

Kubectl-ice is an open-source tool for Kubernetes users to monitor and optimize container resource usage. Features include usage breakdowns for pods and containers, making scaling and optimization easier. The tool is compatible with major cloud providers and is actively developed by a community of contributors
Apache License 2.0
247 stars 9 forks source link

displaying node name and selected node label(s) the pod is placed on #7

Closed dmitry-mightydevops closed 2 years ago

dmitry-mightydevops commented 2 years ago

it would be nice to include the node name the pod is scheduled on OR the label's value of the node the pod is placed on.

the reason: when I change affinity or nodeSelector (especially in the case of complex affinity/antiaffinity rules) I have to double check that the pod was placed in a correct node. Typically by doing:

kg pod NAME -o yaml
kdpo NAME

Which is kinda time consuming. It would be great if we can

k ice status --node --nl eks.amazonaws.com/nodegroup

which will output the name of the node and the value of the label eks.amazonaws.com/nodegroup associated with the node the pod is placed on. So we will get 2 new columns:

node                                         eks.amazonaws.com/nodegroup
ip-10-110-4-137.ec2.internal                 staging-ops-v2

current output

➜ k ice status                                         
T  PODNAME                             CONTAINER              READY  STARTED  RESTARTS  STATE       REASON     EXIT-CODE  SIGNAL  TIMESTAMP                      MESSAGE
S  aws-node-27r6g                      aws-node               true   true     0         Running     -          -          -       2022-06-15 22:27:53 -0500 CDT  -
I  aws-node-27r6g                      aws-vpc-cni-init       true   -        0         Terminated  Completed  0          0       2022-06-15 22:27:50 -0500 CDT  -
S  aws-node-8f6rc                      aws-node               true   true     0         Running     -          -          -       2022-06-16 00:31:56 -0500 CDT  -
I  aws-node-8f6rc                      aws-vpc-cni-init       true   -        0         Terminated  Completed  0          0       2022-06-16 00:31:53 -0500 CDT  -
S  aws-node-hk5l2                      aws-node               true   true     0         Running     -          -          -       2022-06-15 22:27:56 -0500 CDT  -
I  aws-node-hk5l2                      aws-vpc-cni-init       true   -        0         Terminated  Completed  0          0       2022-06-15 22:27:53 -0500 CDT  -
S  aws-node-p8vpt                      aws-node               true   true     0         Running     -          -          -       2022-06-15 22:27:45 -0500 CDT  -
I  aws-node-p8vpt                      aws-vpc-cni-init       true   -        0         Terminated  Completed  0          0       2022-06-15 22:27:42 -0500 CDT  -
S  coredns-84888f77cf-hl85g            coredns                true   true     0         Running     -          -          -       2022-06-16 17:42:10 -0500 CDT  -
S  ebs-csi-controller-cf8d4fd68-9p62b  csi-attacher           true   true     0         Running     -          -          -       2022-06-16 00:33:22 -0500 CDT  -
S  ebs-csi-controller-cf8d4fd68-9p62b  csi-provisioner        true   true     0         Running     -          -          -       2022-06-16 00:33:20 -0500 CDT  -
S  ebs-csi-controller-cf8d4fd68-9p62b  csi-resizer            true   true     0         Running     -          -          -       2022-06-16 00:33:24 -0500 CDT  -
S  ebs-csi-controller-cf8d4fd68-9p62b  ebs-plugin             true   true     0         Running     -          -          -       2022-06-16 00:33:05 -0500 CDT  -
S  ebs-csi-controller-cf8d4fd68-9p62b  liveness-probe         true   true     0         Running     -          -          -       2022-06-16 00:33:24 -0500 CDT  -
S  ebs-csi-node-f47vk                  ebs-plugin             true   true     0         Running     -          -          -       2022-06-16 00:32:12 -0500 CDT  -
S  ebs-csi-node-f47vk                  liveness-probe         true   true     0         Running     -          -          -       2022-06-16 00:32:14 -0500 CDT  -
S  ebs-csi-node-f47vk                  node-driver-registrar  true   true     0         Running     -          -          -       2022-06-16 00:32:13 -0500 CDT  -
S  ebs-csi-node-fdpjl                  ebs-plugin             true   true     0         Running     -          -          -       2022-06-15 22:28:05 -0500 CDT  -
S  ebs-csi-node-fdpjl                  liveness-probe         true   true     0         Running     -          -          -       2022-06-15 22:28:09 -0500 CDT  -
S  ebs-csi-node-fdpjl                  node-driver-registrar  true   true     0         Running     -          -          -       2022-06-15 22:28:07 -0500 CDT  -
S  ebs-csi-node-lzfqh                  ebs-plugin             true   true     0         Running     -          -          -       2022-06-15 22:28:13 -0500 CDT  -
S  ebs-csi-node-lzfqh                  liveness-probe         true   true     0         Running     -          -          -       2022-06-15 22:28:15 -0500 CDT  -
S  ebs-csi-node-lzfqh                  node-driver-registrar  true   true     0         Running     -          -          -       2022-06-15 22:28:14 -0500 CDT  -
S  ebs-csi-node-ntfq8                  ebs-plugin             true   true     0         Running     -          -          -       2022-06-15 22:28:12 -0500 CDT  -
S  ebs-csi-node-ntfq8                  liveness-probe         true   true     0         Running     -          -          -       2022-06-15 22:28:16 -0500 CDT  -
S  ebs-csi-node-ntfq8                  node-driver-registrar  true   true     0         Running     -          -          -       2022-06-15 22:28:13 -0500 CDT  -
S  kube-proxy-9cw8f                    kube-proxy             true   true     0         Running     -          -          -       2022-06-15 22:27:38 -0500 CDT  -
S  kube-proxy-dq7q7                    kube-proxy             true   true     0         Running     -          -          -       2022-06-15 22:27:46 -0500 CDT  -
S  kube-proxy-fkq8q                    kube-proxy             true   true     0         Running     -          -          -       2022-06-16 00:31:53 -0500 CDT  -
S  kube-proxy-z4wvs                    kube-proxy             true   true     0         Running     -          -          -       2022-06-15 22:27:49 -0500 CDT  -
NimbleArchitect commented 2 years ago

I have added the --show-node flag to print the node name in the column output

NimbleArchitect commented 2 years ago

this took some doing... I have node labels working on my minikube setup but I'm unable to test with multiple nodes so would appreciate some feedback.

I have added --node-label and --pod-label flags they take a single label name and output the label as a new column (see output below)

currently you can't use the sort and match flags with the node-labels and pod-labels though

kubectl-ice status  -A --show-node --node-label "minikube.k8s.io/version"
T  NAMESPACE    NODE      PODNAME                           CONTAINER                minikube.k8s.io/version  READY  STARTED  RESTARTS  STATE    REASON  EXIT-CODE  SIGNAL  AGE
S  kube-system  minikube  coredns-78fcd69978-gtg8c          coredns                  v1.24.0                  true   true     26        Running  -       -          -       4d9h
S  kube-system  minikube  etcd-minikube                     etcd                     v1.24.0                  true   true     26        Running  -       -          -       4d9h
S  kube-system  minikube  kube-apiserver-minikube           kube-apiserver           v1.24.0                  true   true     6         Running  -       -          -       4d9h
S  kube-system  minikube  kube-controller-manager-minikube  kube-controller-manager  v1.24.0                  true   true     6         Running  -       -          -       4d9h
S  kube-system  minikube  kube-proxy-4p6q8                  kube-proxy               v1.24.0                  true   true     26        Running  -       -          -       4d9h
S  kube-system  minikube  kube-scheduler-minikube           kube-scheduler           v1.24.0                  true   true     6         Running  -       -          -       4d9h
S  kube-system  minikube  metrics-server-77c99ccb96-z86xc   metrics-server           v1.24.0                  true   true     33        Running  -       -          -       4d9h
S  kube-system  minikube  storage-provisioner               storage-provisioner      v1.24.0                  true   true     43        Running  -       -          -       4d9h
$ kubectl-ice status  -A --show-node --node-label "beta.kubernetes.io/os" --pod-label "component"
T  NAMESPACE    NODE      PODNAME                           CONTAINER                beta.kubernetes.io/os  component                READY  STARTED  RESTARTS  STATE    REASON  EXIT-CODE  SIGNAL  AGE
S  kube-system  minikube  coredns-78fcd69978-gtg8c          coredns                  linux                  -                        true   true     26        Running  -       -          -       4d9h
S  kube-system  minikube  etcd-minikube                     etcd                     linux                  etcd                     true   true     26        Running  -       -          -       4d9h
S  kube-system  minikube  kube-apiserver-minikube           kube-apiserver           linux                  kube-apiserver           true   true     6         Running  -       -          -       4d9h
S  kube-system  minikube  kube-controller-manager-minikube  kube-controller-manager  linux                  kube-controller-manager  true   true     6         Running  -       -          -       4d9h
S  kube-system  minikube  kube-proxy-4p6q8                  kube-proxy               linux                  -                        true   true     26        Running  -       -          -       4d9h
S  kube-system  minikube  kube-scheduler-minikube           kube-scheduler           linux                  kube-scheduler           true   true     6         Running  -       -          -       4d9h
S  kube-system  minikube  metrics-server-77c99ccb96-z86xc   metrics-server           linux                  -                        true   true     33        Running  -       -          -       4d9h
S  kube-system  minikube  storage-provisioner               storage-provisioner      linux                  -                        true   true     43        Running  -       -          -       4d9h
$ kubectl-ice status  -A --show-node --node-label "beta.kubernetes.io/os" --pod-label "component" --tree --details
NAMESPACE    NODE      beta.kubernetes.io/os  component                NAME                                  READY  STARTED  RESTARTS  STATE    REASON  EXIT-CODE  SIGNAL  TIMESTAMP            MESSAGE
kube-system  minikube  linux                  -                        Pod/coredns-78fcd69978-gtg8c          -      -        0         Running  -       -          -       2022-05-20 09:40:06  -
kube-system  minikube  linux                  -                        └─Container/coredns                   true   true     26        Running  -       -          -       2022-06-20 08:55:56  -
kube-system  minikube  linux                  etcd                     Pod/etcd-minikube                     -      -        0         Running  -       -          -       2022-06-06 09:10:17  -
kube-system  minikube  linux                  etcd                     └─Container/etcd                      true   true     26        Running  -       -          -       2022-06-20 08:55:50  -
kube-system  minikube  linux                  kube-apiserver           Pod/kube-apiserver-minikube           -      -        0         Running  -       -          -       2022-05-23 10:52:33  -
kube-system  minikube  linux                  kube-apiserver           └─Container/kube-apiserver            true   true     6         Running  -       -          -       2022-06-20 08:55:50  -
kube-system  minikube  linux                  kube-controller-manager  Pod/kube-controller-manager-minikube  -      -        0         Running  -       -          -       2022-06-06 09:11:58  -
kube-system  minikube  linux                  kube-controller-manager  └─Container/kube-controller-manager   true   true     6         Running  -       -          -       2022-06-20 08:55:50  -
kube-system  minikube  linux                  -                        Pod/kube-proxy-4p6q8                  -      -        0         Running  -       -          -       2022-05-20 09:40:06  -
kube-system  minikube  linux                  -                        └─Container/kube-proxy                true   true     26        Running  -       -          -       2022-06-20 08:55:56  -
kube-system  minikube  linux                  kube-scheduler           Pod/kube-scheduler-minikube           -      -        0         Running  -       -          -       2022-06-18 10:25:00  -
kube-system  minikube  linux                  kube-scheduler           └─Container/kube-scheduler            true   true     6         Running  -       -          -       2022-06-20 08:55:50  -
kube-system  minikube  linux                  -                        Pod/metrics-server-77c99ccb96-z86xc   -      -        0         Running  -       -          -       2022-05-20 09:40:17  -
kube-system  minikube  linux                  -                        └─Container/metrics-server            true   true     33        Running  -       -          -       2022-06-20 08:55:56  -
kube-system  minikube  linux                  -                        Pod/storage-provisioner               -      -        0         Running  -       -          -       2022-05-20 09:40:06  -
kube-system  minikube  linux                  -                        └─Container/storage-provisioner       true   true     43        Running  -       -          -       2022-06-20 08:56:38  -
dmitry-mightydevops commented 2 years ago

@NimbleArchitect it would be nice to skip "I" containers in the same way you did with cpu/memory output (i.e. -i arg to the cli)

➜ kubectl-ice status --show-node --node-label "workload" --pod-label "app.kubernetes.io/component"
T  NODE                         PODNAME                           CONTAINER  workload  app.kubernetes.io/component  READY  STARTED  RESTARTS  STATE       REASON     EXIT-CODE  SIGNAL  AGE
S  ip-10-120-3-53.ec2.internal  prod-backend-v1-7bb984f4d4-jk9df  backend    apps      backend                      true   true     0         Running     -          -          -       10h
I  ip-10-120-3-53.ec2.internal  prod-backend-v1-7bb984f4d4-jk9df  install    apps      backend                      true   -        0         Terminated  Completed  0          0       10h
S  ip-10-120-3-53.ec2.internal  prod-frontend-v1-5696b46985-nv7vd frontend   apps      frontend                     true   true     0         Running     -          -          -       10h
NimbleArchitect commented 2 years ago

is that just removing the T column or removing init containers from the table output?

if you just want to remove the init containers from the output you can use --match "T!=I" which means only output the line if the "T" column does not equal "I" (uppercase i).

edit: the match flag doesn't work correctly atm, I'll work on a fix soon.

dmitry-mightydevops commented 2 years ago

didn't work for me

➜ kubectl-ice status --show-node --node-label "workload" --pod-label "app.kubernetes.io/component" --match "T!=I"
T  NODE                         PODNAME                              CONTAINER            workload  app.kubernetes.io/component  READY  STARTED  RESTARTS  STATE       REASON     EXIT-CODE  SIGNAL  AGE
S  ip-10-110-4-28.ec2.internal  staging-backend-v1-57dd8d95f4-lpnf2  backend              apps      backend                      true   true     0         Running     -          -          -       11d
I  ip-10-110-4-28.ec2.internal  staging-backend-v1-57dd8d95f4-lpnf2  get-config-template  apps      backend                      true   -        0         Terminated  Completed  0          0       11d
I  ip-10-110-4-28.ec2.internal  staging-backend-v1-57dd8d95f4-lpnf2  install              apps      backend                      true   -        0         Terminated  Completed  0          0       11d

so I expect this in outcome to skip terminated init containers:

➜ kubectl-ice status --show-node --node-label "workload" --pod-label "app.kubernetes.io/component" --tree | sed -n '/Terminated/!p' 
NODE                         workload  app.kubernetes.io/component  NAME                                     READY  STARTED  RESTARTS  STATE       REASON     EXIT-CODE  SIGNAL  AGE
ip-10-110-4-28.ec2.internal  apps      backend                      Pod/staging-backend-v1-57dd8d95f4-lpnf2  -      -        0         Running     -          -          -       11d
ip-10-110-4-28.ec2.internal  apps      backend                      └─Container/backend                      true   true     0         Running     -          -          -       11d
NimbleArchitect commented 2 years ago

@dmitry-mightydevops first let me say a massive thank you for your donation. Thank you I really appreciate it.

Also I have just pushed a large update that fixes the match flag and hides the Type column, if it's needed again -T will show the container type.

let me know if you have any issues or problems

dmitry-mightydevops commented 2 years ago

you're very welcome!

confirmed that it worked as expected:

➜  kubectl-ice status --show-node --node-label "workload" --pod-label "app.kubernetes.io/component" --match "STATE!=Terminated"
NODE                         PODNAME                              CONTAINER            workload  app.kubernetes.io/component  READY  STARTED  RESTARTS  STATE       REASON     EXIT-CODE  SIGNAL  AGE
ip-10-110-4-28.ec2.internal  staging-backend-v1-57dd8d95f4-lpnf2  backend              apps      backend                      true   true     0         Running     -          -          -       17d