Improve the jitter value for kubelet's serviceAccountToken projected volume source

[ahmedtd]

The current jittering logic in kubelet token_manager refreshes the token once it has 20% of its lifetime remaining, with 10 seconds of jitter.

Experimentally, this doesn't seem to be enough to really blunt spikes of refresh requests that come from many workloads having a synced refresh. This can occur when:

• A large deployment is scaled up. All the new replicas will have synced-up refresh traffic.
• A kubelet restarts. Because kubelet only holds the tokens in an in-memory cache, it will need to acquire new tokens for all pods on the node after it restarts. After that, all of the pods will have synced-up refresh traffic.

As an example, here's serviceaccount/token API traffic from a real cluster. Note how there is a repeating spike every 48 minutes, which is what we would expect from the "refresh after 80%" logic, and the default setting of token lifetime for 1 hour. Effective jittering logic would be noticeably smoothing these peaks.

We should increase the maxJitter value. Perhaps 5 minutes would be a good value.

/sig auth /kind bug

[aramase]

/triage accepted

[aramase]

Need to change the jitter value to take in the token expiration into consideration. It can't be a fixed 5 mins because user can request a token with 10mins expiry.

[yoonhyunwoo]

Would it not be possible to use a dynamic assignment like the following? This approach has shown quite promising results in our simple tests.
pkg/kubelet/token/token_manager.go

182 maxJitter := time.Duration(*tr.Spec.ExpirationSeconds*10/100) * time.Second
183 
184 jitter := time.Duration(rand.Float64()*maxJitter.Seconds()) * time.Second

Dynamic jitter vs. existing jitter: