Breaks v1.27.x clusters

[SammyA]

What happened?: Upgraded kubernetes to v1.27.x. Various k8s-clients started crashing (helm, k9s, metallb, ...). Seems like prom-adapter returns malformed metrics (see https://github.com/kubernetes/kubernetes/pull/116603). Uninstalling prom-adapter fixes the crashes.

Environment:

• prometheus-adapter version: v0.10.0
• prometheus version: v2.43.1
• Kubernetes version (use kubectl version): v1.27.1
• Cloud provider or hardware configuration: kubeadm

Steps to reproduce

• Install kuberntest cluster v1.27.x
• Install kube-prom
• Install prometheus-adapter (from helm chart, with --set prometheus.url=http://kube-prom-kube-prometheus-prometheus)
• try accessing cluster with k9s version <= v0.27.3
• or try installing a helm chart
• or restart metallb-speaker

[sirantd]

Even internal cluster components affected by it would be great if can be fixed.

https://github.com/aws/amazon-vpc-cni-k8s/issues/2393

[jmriebold]

This is affecting our clusters as well, everything from AWS VPC CNI to Helm to Flux.

CC @dgrisonnet @olivierlemasle

[dgrisonnet]

/assign

[dgrisonnet]

/triage accepted

[colinrgodsey]

PR for the helm chart updates: https://github.com/prometheus-community/helm-charts/pull/3641