Http requests to the loopback address inside Windows containers seem to randomly drop when using Ingress rules with namespace selectors

[jeffreykeagbine]

We have a series of Windows pods that are configured with network policies to only be accessible to pods in specific namespaces. The windows containers are accessed via a REST API, and for certain requests will call a windows service local to the container via a Http request that will do processing and then send it to another windows service local to the container via another http request to do final handling. Only the initial rest API is exposed outside of the container.

This works pretty consistently when running locally both outside of a container environment and in a local container. However, while running in our cluster, we're seeing about 5-10% of request arbitrarily failing at one of these levels, with logs suggesting that the calling service never sees a response and the called service believing it sent everything back correctly. After some investigation on different environments to try to narrow it down, we've found that it seems to be linked to our Network Policy with Ingress rules based on Namespace selectors. Removing the Ingress policy or swapping to IP based rules looks to cause the failures to disappear.

Looking through documentation, we found this know limitation of windows pods that seems similar to our issue, but we're seeing this with requests internal to the pod rather than pod to pod. If it's the case that it's the same root cause, confirmation would be appreciated.

Expected Behavior

Http requests inside a windows container should consistently work when that container is subject to ingress rules limiting calling to specific namespaces.

Current Behavior

Http requests are sporadically dropped in Windows containers when calling http endpoints internal to the container when subject to a Network Policy that limits ingresses to specific namespaces.

Possible Solution

Confirm this is the same root cause as the known limitation for pod to pod communication linked above and the ball is in Microsoft's court or fix whatever is interfering in the local Http Requests if another issue.

Steps to Reproduce (for bugs)

1. Create a windows container with two Rest APIs running, one exposed externally and one only accessible internally listening to a port on 127.0.0.1. Have the external API call the internal Web API when called.
2. Create a Network Policy that sets an Ingress Policy that only allows specific namespaces that applies to a pod that is running the container created in step one.
3. Repeatedly call the API. The requests will sporadically fail.

Context

We are trying to import customer data from third party products to a legacy system. The sporadic network failures on imports makes it difficult for us to tell if the import failed due to issues with the data or other dependencies and needs to be retried or if it failed sporadically and an attempt to reimport would cause unwanted data duplication.

Your Environment

• Calico version: v3.24.0
• Orchestrator version: Kubernetes 1.27.7 via AKS
• Operating System: Windows Server 2019, Kernel Version 10.0.17763.5329 Windows Server 2022, Kernel Version 10.0.20348.2340

[coutinhop]

@jeffreykeagbine would you be able to provide more detailed steps to reproduce, i.e. some exact images for this scenario, yamls for policies, etc?

Alternatively, would you be able to provide the output of pktmon when the issue happens? That would help determining if we're seeing the same issue as mentioned in our docs...

Let me also tag @song-jiang to see if he has any other insights/questions on how to approach this...

[song-jiang]

@jeffreykeagbine The known limitation in our doc could have an impact on a windows container traffic (internal or external) when policy that applies to a pod contains selectors and the set of endpoints that those selectors match changes .

In your case, if you have an ingress rule to allow or deny access from some namespaces, and if pods being created/deleted in those namespaces, this could upset the traffic for the windows container in a few seconds based on our observations.

[jeffreykeagbine]

@coutinhop We don't have a public container that showcases this, but we've been able to get this to replicate in a simpler container with the ASP.NET program/dockerfile in this zip file where it runs two copies of a an API and a script repeatedly calls one to call the other. Inside that Zip is also a facsimile of the network policy we were using with names swapped out to be generic.

InternalCommunication.zip

I'm working on getting our environment set up again to replicate with pktmon running and will post those results when I have them.

@song-jiang If it can also happen for internal windows container traffic as you're stating here, it seems likely to me that it would be what we're running into. While we haven't been able to prove that out completely, it would align with the evidence we've been able to collect.

[song-jiang]

@jeffreykeagbine The way to prove it is to switch to an IP range based policy model. Say you have three namespaces, assign three different ippools to each namespace and apply a network policy to deny traffic from those three ip ranges.

In same situation that pods being created/deleted in those namespaces, there's no update to Windows ACL rules and you shouldn't see traffic being dropped again.