Closed george-angel closed 6 years ago
@george-angel hm, yeah this is an interesting one.
I'm actually a bit fuzzy on the implementation details of kubectl proxy, but my understanding is that the requests will not go through your ingress controllers, which is probably why the second NP above isn't working as expected.
As you've discovered, since the pods are using host networking Kubernetes network policy doesn't apply to them the way it does to other pods.
One option might be to use a Calico GlobalNetworkPolicy
instead of k8s policies in order to make it a cluster-wide behavior instead of per-namespace. See here
You can also use host endpoints or network sets to help simplify the definition a bit, but neither of those are perfect since they aren't fully automated and still require someone to create the resources. Though, if you've got some sort of node config management in place then it could simply be a matter of adding a section to your terraform / ansible / etc to create the above.
Thanks @caseydavenport fixed using GlobalNetworkPolicies, for reference: https://github.com/utilitywarehouse/tf_kube_ignition/pull/34
Expected Behavior
Given a basic NetworkPolicy that allow traffic within namespace, we would like to have kubectl-proxy functionality to be able to reach pods in that namespace.
Policy above allows all traffic within the namespace, traffic from
sys-prom
namespace, and traffic fromsys-ingress-priv
- which is a namespace that contains internally facing ingress controllers.With that policy applied:
request fails. We also tried allowing traffic from
kube-system
namespace, but the result is the same, request times out.If we delete the NetworkPolicy, the request returns 200. We also tcpdumped all traffic on masters and workers to capture this:
master:
worker:
Our api-server pods are running using host's networking:
So they don't have a separate IP in the pod range (
10.2.0.0/16
) - so the request src address is10.2.X.0
.We can solve the problem by allowing kubelet/flannel/pods using host's network with following policy:
But that doesn't feel good, and it needs to be applied to every namespace.
Is this something that is supposed to work? Is it the specifics of your setup that create this odd situation?
Your Environment
calico/node:v3.0.4 calico/cni:v2.0.3 coreos/flannel:v0.10.0 Container Linux by CoreOS 1688.5.3 (Rhyolite)