CrashLoopBackOff

[filipdadgar]

Awesome stuff, but I have the below issue. So the pod is in crash loop.

kubectl apply -f deploy/manifest.yaml deployment.apps/k8spurger created service/k8spurger-svc created serviceaccount/k8spurger-sa created Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole clusterrole.rbac.authorization.k8s.io/k8spurger-cluster-role created Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v 1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBindin g clusterrolebinding.rbac.authorization.k8s.io/k8spurger-rb created

The error:

Getting unused secret it may take couple of minute.. Traceback (most recent call last): File "K8sPurger.py", line 324, in main("svc") File "K8sPurger.py", line 32, in main GetUsedResources(v1) File "K8sPurger.py", line 125, in GetUsedResources UsedConfigMap.append([volume.config_map_ref.name, i.metadata.namespace])

[yogeshkk]

Thanks @filipdadgar

This was fixed bug so Some how you are getting old image.

So instant of latest I have pinned image tag and pushed new image yogeshkunjir/k8spurger:v0.35 Please try this image and let me know if sorted.

[filipdadgar]

Thank you @yogeshkk for swift response.

It lists some of the unused stuff = awesome!

But then the pod goes to Error and the log produces:

Getting unused Ingress it may take couple of minute.. Not able to reach Kubernetes cluster check Kubeconfig Traceback (most recent call last): File "K8sPurger.py", line 230, in DefinedIngress ApiResponce = V1beta1Api.list_ingress_for_all_namespaces(watch=False) File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/extensions_v1beta1_api.py", line 3385, in list_ingress_for_all_namespaces return self.list_ingress_for_all_namespaces_with_http_info(**kwargs) # noqa: E501 File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/extensions_v1beta1_api.py", line 3488, in list_ingress_for_all_namespaces_with_http_info return self.api_client.call_api( File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 239, in GET return self.request("GET", url, File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 233, in request raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'b82da55a-a966-4ab6-bfe1-acf83e15543f', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'd3d58154-35af-4e03-9f01-bab3322ac780', 'Date': 'Tue, 22 Jun 2021 06:35:18 GMT', 'Content-Length': '336'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"ingresses.extensions is forbidden: User \"system:serviceaccount:default:k8spurger-sa\" cannot list resource \"ingresses\" in API group \"extensions\" at the cluster scope","reason":"Forbidden","details":{"group":"extensions","kind":"ingresses"},"code":403}

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "K8sPurger.py", line 321, in main("svc") File "K8sPurger.py", line 49, in main DefinedIngress(v1beta1Api) File "K8sPurger.py", line 233, in DefinedIngress raise RuntimeError(e) RuntimeError: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'b82da55a-a966-4ab6-bfe1-acf83e15543f', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'd3d58154-35af-4e03-9f01-bab3322ac780', 'Date': 'Tue, 22 Jun 2021 06:35:18 GMT', 'Content-Length': '336'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"ingresses.extensions is forbidden: User \"system:serviceaccount:default:k8spurger-sa\" cannot list resource \"ingresses\" in API group \"extensions\" at the cluster scope","reason":"Forbidden","details":{"group":"extensions","kind":"ingresses"},"code":403}

[yogeshkk]

Ohh. look like K8S version bit old (Ingress is now part of networking.k8s.io). But no problem I have added permission to handle it. Can you please try again I have updated role.

[filipdadgar]

Looks good now, running!

k8spurger-c6d5bf58f-d4s9l 1/1 Running 0 2m52s

Though the dashboard isnt updating; No Data.

[yogeshkk]

Might be Prometheus issue with Promethus or Grafana.

Can you see "k8s_unused_resources" in Prometheus UI ?

Also I have updated image to add curl. Can you redeploy and then exec into pod and run below command to see if app is exposing data to narrow down the issue.

~/p/K8sPurger> kubectl exec -it k8spurger-76fb9fd54c-lmz8d sh
/app # curl localhost:8000
# HELP python_gc_objects_collected_total Objects collected during gc
<Removed few line>
process_cpu_seconds_total 1.4
# HELP k8s_unused_resources show unused resources in k8s
# TYPE k8s_unused_resources gauge
k8s_unused_resources{name="k8s-spot-price-monitor",namespaces="kube-addons",type="Service Account"} 1.0
k8s_unused_resources{name="namespace-configuration-controller",namespaces="kube-addons",type="Service Account"} 

[filipdadgar]

Had to install curl with apt-get.

Then curl works:

TYPE k8s_unused_resources gauge k8s_unused_resources{name="kube-prometheus-stack-admission",namespaces="default",type="Secrets"} 1.0 k8s_unused_resources{name="prometheus-kube-prometheus-admission",namespaces="default",type="Secrets"} 1.0 k8s_unused_resources{name="regcred",namespaces="default",type="Secrets"} 1.0 k8s_unused_resources{name="sh.helm.release.v1.nfs-subdir-external-provisioner.v1",namespaces="default",type="Secrets"} 1.0 k8s_unused_resources{name="kubernetes-dashboard-csrf",namespaces="kubernetes-dashboard",type="Secrets"} 1.0 k8s_unused_resources{name="kubernetes-dashboard-key-holder",namespaces="kubernetes-dashboard",type="Secrets"} 1.0 k8s_unused_resources{name="prometheus-operator-kube-p-admission",namespaces="monitoring",type="Secrets"} 1.0

[yogeshkk]

Gr8. I have added annotation for promethues to scrape but some how it is not scraping.

      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: /metrics

You need to configure Prometheus to scrape pod for metrics. Below might help. https://stackoverflow.com/questions/41725767/how-to-scrape-pod-level-info-using-prometheus-kubernetes

[filipdadgar]

Awesome, will add a scrape. Will update you during the day. :)

[yogeshkk]

@filipdadgar Are you ok, If i close this issue as K8sPurger is exporting metrics.

[filipdadgar]

Hi @yogeshkk , sorry for lag. Yes it works! Thank you for awesome work!