caseydavenport
commented
2 months ago @MageekChiu thanks for raising this.
watcherSyncer will wait for 1001 updates and call "ws.callbacks.OnUpdates(updates)" to invoke the onUpdate above
I assume you're looking at this code? https://github.com/projectcalico/calico/blob/77ecdcda90c0a8a38ae247067169523484c6c8e3/libcalico-go/lib/backend/watchersyncer/watchersyncer.go#L156-L166
If so, that only waits for a batch of 1000 updates if we are receiving a rapid fire of updates - if the loop ever executes and there is not a pending update to process, we will send what we have rather than waiting. I don't think this should be a problem, and this logic is used extensively throughout Calico so I am pretty confident it's not waiting here.
I suggest that we just periodically call syncAllAutoHostendpoints of autoHostEndpointController to make sure we have all the expected hostendpoint
This won't help us spot missing nodes, since this logic runs based on the cache of nodes we have received from the syncer. We definitely need to run this off the cache instead of polling nodes, though.
I think we need to dig into why it's actually taking so long to process the new node. My suspicion is that we're actually hitting client side rate limiting here because we aren't using the syncer for HostEndpoint objects, and instead going straight to the API. For a large number of host endpoints, this will cause us to self rate limit requests.
A fix for that would be to stop using the client here: https://github.com/projectcalico/calico/blob/77ecdcda90c0a8a38ae247067169523484c6c8e3/kube-controllers/pkg/controllers/node/hostendpoints.go#L52
And instead replace the Get() and List() calls with a cache fed by the syncer, much like we are doing for nodes.
Do you have a full log from the kube-controllers pod while it is in this state I could look at?
MageekChiu
In a large k8s cluster(say more than 100 nodes), when a node joins the cluster, it relies on its hostendpoint(auto created by kube-controller) to get proper network connectivity(cause we use label based policy), but it might take many minutes, even hours to see the hostendpoint created, leading to the unavailability of this node.
Expected Behavior
I expect the hostendpoint to be created within a predictable timeframe
Possible Solution
I've walked through relevant codes and done some test and here are some of my observations：
To summarize, the event-driven mechanism we have now for autoHostEndpointController spends most of its time checking useless updates, causing truly useful updates(like a new node) to be unable to get dealt with in a timely manner(cause we don't know how many updates are before the new creation update and thus don't know how long does it take before this update can be dealt with). I suggest we use a periodic-checking mechanism to make sure all the hostendpoints can become what's expected within a predictable timeframe
A workaround is to restart the calico-kube-controller, which would start a full resync and the hostendpoint can be created right away, but I don't think it's an elegant solution
Steps to Reproduce (for bugs)