Closed ZihanJiang96 closed 8 months ago
Attention: 12 lines
in your changes are missing coverage. Please review.
Comparison is base (
a70d012
) 69.78% compared to head (308e7b1
) 69.87%.:exclamation: Current head 308e7b1 differs from pull request most recent head 3d43b6a. Consider uploading reports for the commit 3d43b6a to get more accurate results
Files | Patch % | Lines |
---|---|---|
pkg/service/server.go | 57.14% | 9 Missing and 3 partials :warning: |
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
Issue
When we terminate a large mount of nodes at the same time, let's 600 nodes, lifecycle-manager can only process 75 node events per minute, which means
600/75=8
min. If we set the ASG Lifecycle hook's heartbeat timeout seconds to 300s, then some of the node events will never get processed and after the 300s timeout, the node will get terminated by ASG directly without proper drain, which leads to pod ungraceful shutdown.Fixes/Improvements
QPS
from 5 to 600,Burst
from 10 to 2000nodeMetadataMap
to cache the node instanceID --> node name info (reduce the k8s client list call).getNodeByName()
is used during node drain/delete. it should use the Get() API instead of List()After the improvements, lifecycle-manager could handle 200 node messages per minute.