Pod is still using old tokens after upgrade

oganekoi commented 2 years ago

Hi,

I am trying this: https://github.com/fluent/fluentd-kubernetes-daemonset/pull/1367 However, the token does not seem to be refreshed. I am using fluentd-kubernetes-daemonset:v1.14.6-debian-cloudwatch-1.1 in AWS EKS 1.22 and I have checked, if it is using stale tokens.

When the API server receives requests with tokens that are older than one hour, then it annotates the pod with annotations.authentication.k8s.io/stale-token. In my case I can see the following annotation. E.g.:

annotations.authentication.k8s.io/stale-token subject: system:serviceaccount:amazon-cloudwatch:fluentd, seconds after warning threshold: 14655

I have also confirmed in the pod logs that the version of fluent-plugin-kubernetes_metadata_filter is 2.11.1.

Pod Logs:

2022-06-30 03:21:52 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.11.1'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2022-06-30 03:21:52 +0000 [info]: gem 'fluentd' version '1.14.6'

Fluentd Kubernetes Daemonset Version Info

fluentd-kubernetes-daemonset:v1.14.6-debian-cloudwatch-1.1

Cluster Details

AWS EKS 1.22 fluentd-kubernetes-daemonset:v1.14.6-debian-cloudwatch-1.1 deployed as Daemonset

Steps to reproduce issue

Enable EKS Audit Logs

Query CW Insights (select cluster log group):

fields @timestamp
| filter @logStream like /kube-apiserver-audit/
| filter @message like /seconds after warning threshold/
| parse @message "subject: *, seconds after warning threshold:*\"" as subject, elapsedtime

ashie commented 2 years ago

In my understanding, the fix in fluent-plugin-kubernetes_metadata_filter v2.11.1 is just a work around, not fully resolve the issue in desired way. It refreshes a token only when it's really expired. See also: https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/pull/337#issuecomment-1167807275

ashie commented 2 years ago

So I'm considering to help releasing new version of kubeclient (https://github.com/ManageIQ/kubeclient/issues/561#issuecomment-1150856671) but I also don't have enough time for it now...

oganekoi commented 2 years ago

@ashie Thank you for the clear explanation, I understand the current situation very well. I am relieved to hear that the tokens are refreshed when they expire and the operation itself seems to be fine. I understand that you are developing a kubeclient to refresh old tokens. We will continue to monitor this issue closely. I will close this issue. Thank you very much.

PettitWesley commented 2 years ago

Hey, I implemented the workaround. What was stated in this issue is correct. It refreshes reactively, only when the token is expired.

ashie commented 2 years ago

Thanks following up :+1:

jorge-fabric commented 1 year ago

Hello @PettitWesley @ashie 👋 the token still refreshes reactively, correct? So there is no fix to avoid the stale token warnings in EKS for now?

Btw, I'm running an EKS 1.21 and the following fluentd plugins versions:

2022-10-21 18:00:08 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluentd.conf"
2022-10-21 18:00:08 +0000 [info]: gem 'fluentd' version '1.15.2'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.5'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.13.0'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'

PettitWesley commented 1 year ago

@jorge-gyant Yea, the change I contributed (in 2.11.1 IIRC) is reactive.

There was some discussion that making a change in the underlying ruby kube client library would allow proactive refreshes, I am not sure if anyone has worked on that.

ashie commented 1 year ago

kubeclient 4.10.0 supports refreshing the token on every request (when bearer_token_file is set), so that we can fix the issue by updating kubeclient. We'll update it in the next release.

whereisaaron commented 1 year ago

@jorge-gyant I had the same problem, even with latest fluentd, so I switched to fluentbit as the k8s client there can renew tokens before they expire. Looks like it renews every 12-13 minutes. No more stale token alerts 👏

[2022/10/30 08:52:18] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2022/10/30 09:05:08] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2022/10/30 09:18:08] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2022/10/30 09:30:08] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2022/10/30 09:45:08] [ info] [filter:kubernetes:kubernetes.0]  token updated

dhayhak commented 1 year ago

Hi @ashie,

so that we can fix the issue by updating kubeclient. We'll update it in the next release.

Do you know which release will have the kubeclient token fix? I'm currently testing with fluentd-daemonset v1.15.3 and I can't confirm which kubeclient version it is using.

Thanks!

ashie commented 1 year ago

v1.15.3 images use kubeclient 4.10.1. e.g.) https://github.com/fluent/fluentd-kubernetes-daemonset/blob/449777964f839bddcee2743fe463fce22103735d/docker-image/v1.15/debian-s3/Gemfile.lock#L94

fluent / fluentd-kubernetes-daemonset