kubernetes-retired / kube-aws

[EOL] A command-line tool to declaratively manage Kubernetes clusters on AWS
Apache License 2.0
1.12k stars 295 forks source link

Unable to scrape node metrics after upgrade to 0.13 #1907

Closed mixecan closed 4 years ago

mixecan commented 4 years ago

After upgrading our cluster using kube-aws 0.13 we found prometheus unable to scrape node metrics through kube-proxy. It looks like something in the kubernetes authorization mechanism or RBAC permission changed but we couldn't find anything.

The error that we get is

Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=metrics)

We confirmed that the issue is not related to prometheus as it occurs also using the admin account. Steps to reproduce are

  1. start proxy kubectl proxy
  2. pick a node kubectl get nodes
  3. get node metrics curl http://localhost:8001/api/v1/nodes/<nodename>/proxy/metrics

How can this be fixed?

mixecan commented 4 years ago

Hi, I started a brand new cluster with the latest release (0.16.4) and the problem still exists. Anyone with the same issue?