search

sighupio / fury-kubernetes-monitoring

Kubernetes Fury Distribution Monitoring core module: Monitor the status of your Kubernetes Cluster and its applications
BSD 3-Clause "New" or "Revised" License
47 stars 6 forks source link

Node Exporter 1.3.1 will break node metrics with prometheus-adapter is used #150

Closed lzecca78 closed 3 months ago

lzecca78 commented 1 year ago

Hi! during a setup with module version 2.0.1 we've found out that node-exporter 1.3.1 have some issues when node metrics are requested (grafana dashboards that ask for them and kubectl top nodes). Once node-exporter is upgraded to 1.5.0 (at least), everything has back to work.

nutellinoit commented 1 year ago

Thank you @lzecca78 , we are in the process of releasing the new version of the module with node-exporter 1.5.0

ralgozino commented 1 year ago

I can't replicate the issue, could you give us some more details please?

This is what I did:

  1. Created a 1.24 cluster with kind:
kind create cluster --image kindest/node:v1.24.7

  1. Cloned monitoring v2.0.1

  2. Deployed the following components without any patches:

    1. prometheus-operator
    2. prometheu-operated
    3. prometheus-adapter
    4. node-exporter
    5. kube-state-metrics
    6. kube-proxy-metrics
    7. kubeadm-sm

Wait a few minutes and kubectl top nodes should be working:

❯ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.9", GitCommit:"9710807c82740b9799453677c977758becf0acbb", GitTreeState:"clean", BuildDate:"2022-12-08T10:15:09Z", GoVersion:"go1.18.9", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.7", GitCommit:"e6f35974b08862a23e7f4aad8e5d7f7f2de26c15", GitTreeState:"clean", BuildDate:"2022-10-26T15:06:14Z", GoVersion:"go1.18.7", Compiler:"gc", Platform:"linux/amd64"}
❯ kubectl top nodes
NAME                CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
124-control-plane   247m         4%     2515Mi          31%
❯ kubectl describe pod -n monitoring node-exporter-x6kj2|grep -i 'image:'
    Image:         registry.sighup.io/fury/alpine:3
    Image:         registry.sighup.io/fury/prometheus/node-exporter:v1.3.1
    Image:         quay.io/brancz/kube-rbac-proxy:v0.12.0

In any case, the next release of the monitoring module will include node-exporter v1.5.0

lzecca78 commented 1 year ago

Hi @ralgozino sorry for the delay! We have that issue with Kubernetes EKS 1.22. The answer that wer had from kubectl top nodes was metrics not available yet but forever. Once we've patched the version to 1.5.0 in a couple of minutes all the metrics were there.

ralgozino commented 3 months ago

fixed in #129