search

kubernetes-sigs / dashboard-metrics-scraper

Container to scrape, store, and retrieve a window of time from the Metrics Server.
Apache License 2.0
87 stars 39 forks source link

Metrics scraper using recycled tokens in kubernetes 1.21 #42

Closed shalver closed 2 years ago

shalver commented 3 years ago

Metric scraper version: v1.0.6

The Bound Service Account Tokens feature is now enabled by default in kubernetes 1.21

Service account tokens are now regularly regenerated and replaced on pods. Kubernetes currently extends the life of the token after they are swapped, but also increments the serviceaccount_stale_tokens_total of the kube-apiserver and also audits the information about the offender. The dashboard-metrics-scraper pod is using these tokens past their life and is being flagged in audit log (note the annotation: authentication.k8s.io/stale-token). Moving up to go 1.15 or greater may possible correct this problem. 

{
  "level": "Metadata",
  "auditID": "41b24987-fc6f-468c-ac17-0b990d96d214",
  "stage": "RequestReceived",
  "requestURI": "/apis/metrics.k8s.io/v1beta1/nodes",
  "verb": "list",
  "user": {
    "username": "system:serviceaccount:kube-system:kubernetes-dashboard",
    "uid": "90021a00-b991-497b-9b70-93c657e6c569",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:kube-system",
      "system:authenticated"
    ],
    "extra": {
      "authentication.kubernetes.io/pod-name": [
        "dashboard-metrics-scraper-79f744b7dd-jpld7"
      ],
      "authentication.kubernetes.io/pod-uid": [
        "d3e55f47-b195-41f2-b52e-a14d567d4782"
      ]
    }
  },
  "sourceIPs": [
    "172.18.137.0"
  ],
  "userAgent": "metrics-sidecar/v0.0.0 (linux/amd64) kubernetes/$Format",
  "objectRef": {
    "resource": "nodes",
    "apiGroup": "metrics.k8s.io",
    "apiVersion": "v1beta1"
  },
  "requestReceivedTimestamp": "2021-04-13T18:58:14.483224Z",
  "stageTimestamp": "2021-04-13T18:58:14.483224Z",
  "annotations": {
    "authentication.k8s.io/stale-token": "subject: system:serviceaccount:kube-system:kubernetes-dashboard, seconds after warning threshold: 82321"
  }
}
fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

shalver commented 3 years ago

/remove-lifecycle rotten

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

shalver commented 3 years ago

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

shalver commented 2 years ago

/remove-lifecycle stale

maciaszczykm commented 2 years ago

/lifecycle frozen

vgaddavcg commented 2 years ago

@maciaszczykm we are using EKS 1.21 and see metrics scraper is using stale token as described by @shalver . We need to update the kubernetes client-go version to 0.15.7 or higher.

current version: k8s.io/client-go v0.0.0-20190222093734-6e378217e628

BoundServiceAccountTokenVolume graduated to beta and is enabled by default in Kubernetes version 1.21. This feature improves security of service account tokens by allowing workloads running on Kubernetes to request JSON web tokens that are audience, time, and key bound. Service account tokens now have an expiration of one hour. In previous Kubernetes versions, they didn't have an expiration. This means that clients that rely on these tokens must refresh the tokens within an hour. The following Kubernetes client SDKs refresh tokens automatically within the required time frame:

Go v0.15.7 and later

Python v12.0.0 and later

Java v9.0.0 and later

JavaScript v0.10.3 and later

Ruby master branch

Haskell v0.3.0.0

C# v7.0.5 and later