huntergregory
commented
1 month ago This issue can actually occur for NetPol without CIDR rule. Most likely to occur for NetPol with CIDR rule or MatchExpressions. Yet could happen for other IPSets if they are not associated with any Pod IP.
NOTE: starting in v1.5.36, NPM will auto-restart to mitigate this issue and other non-retriable errors when detected
Similar issue: #2977
EDIT (9/5/24): based on https://github.com/Azure/azure-container-networking/issues/2963#issuecomment-2332545759
Summary
Under this race condition, NPM is incapacitated (unable to process this or future NetworkPolicies) until it is restarted.
This race can occur when editing or deleting a NetworkPolicy with "enough" rules if the kernel is slow to process nftables/ipset changes. It seems like the issue only happens right after NPM's garbage collector marks "enough" IPSets for deletion.
Symptoms
Unexpected connectivity after editing/deleting a NetPol.
NPM logs continually show this failure:
Mitigation
Restart all NPM Pods (NPM will automatically restart in v1.5.36 and above).
To Avoid the Issue
Cause
NPM translates Pod labels and namespace labels into kernel IPSets (which are referenced by kernel firewall rules). NPM has a 5-minute background thread that marks unreferenced IPSets for deletion. NPM listens for changes to Pods, namespaces, and NetworkPolicies. Right after "enough" IPSets are marked for deletion, if the next change is an edit or deletion of a NetworkPolicy with "enough" rules, then NPM may hit a race condition. At this point, NPM deletes the Policy firewall rules, then tries to delete both the IPSets marked for deletion and some IPSets from the policy. Sometimes the kernel still thinks that firewall rules reference the policy IPsets (hence the log line
Set cannot be destroyed: it is in use by a kernel component), so after 5 attempts, NPM will fail to delete all the IPSets. However, NPM may have deleted some of the IPSets marked for deletion. The previous "slow kernel" issue is temporary, but when NPM retries after some time, if 5+ IPSets were already deleted previously, then NPM will remain incapacitated (unable to process this or future NetworkPolicies) until it is restarted.Theoretically, the garbage collection precondition is not required, although in practice, I cannot reproduce without satisfying this precondition (even with a NetworkPolicy with 240 CIDR rules).
Reproducing
Steps
Logs