daniel-spray
commented
2 years ago This issue was resolved in this release https://github.com/cloudability/metrics-agent/releases/tag/v2.9
Make sure your agent's are running on version 2.9 or higher. Are you still seeing this issue with a metrics-agent 2.9 or higher?
Bug Description
On k8s clusters running 1.21+ mounted service account tokens can be created with an expiry. I've started noticing agents getting into states with errors like failed to set up sandbox container ...: Unauthorized.
BoundServiceAccountTokenVolume Kubernetes feature (graduated to stable in 1.22) improves security of service account tokens by requiring a one hour expiry time, over the previous default of no expiration. This means that metric-agent should refetch service account tokens periodically.
How can you resolve the issue?
To make the transition to time bound service account tokens easier, Kubernetes has updated the below official versions of client SDKs to automatically refetch tokens before the one hour expiration:
Go v0.15.7 and later Python v12.0.0 and later Java v9.0.0 and later Javascript v0.10.3 and later Ruby master branch Haskell v0.3.0.0