Provide mechanism to modify established connections

[0m1xa]

I faced the situation when deletion (and maybe creation) of the policy doesn't affect the pod(s). I guess the main problem is that calico doesn't modify established connections. And if the pod has a persistent connection then changes will not affect it.

Expected Behavior

Modify the established connections as well.

Current Behavior

The state of the established connections doesn't change.

Context

It is a great example to use reloader.

• Create a default deny ingress/egress policy
• Deploy reloader in your cluster according to the docs
• Deploy a simple application and attach a secret/configMap, annotate the deployment with reloader annotations
• Try to make changes in the secret/configMap attached to the deployment.
• See error logs with unavailable kube-api from the reloader pod(s)
• Add allow policy for the reloader

  apiVersion: projectcalico.org/v3
  kind: NetworkPolicy
  metadata:
  name: allow-api-access
  namespace: test
  spec:
  selector: all()
  egress:
  - action: Allow
    destination:
      services:
        name: kubernetes
        namespace: default

• Make changes in the secret/configMap again
• See no errors in the reloader pod(s)
• Delete the allow policy
• Make changes in the secret/configMap one more time
• It's expected to see the error messages in the reloader pod(s) with the unavailability of the kube-api. But in the real world nothing changes to the reloader pod(s). They continue to communicate with the kube-api.

Your Environment

• Calico version 3.23.2
• Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes 1.21 (EKS), kubernetes 1.24.1 (minikube)
• Operating System and version: Amazon Linux2 (EKS), Ubuntu 20.04.4 (minikube)

[caseydavenport]

I think it's worth pointing out here that the Kubernetes NetworkPolicy API is intended to operate on connections and not packets: https://kubernetes.io/docs/concepts/services-networking/network-policies/

So, the current behavior is the "correct" one in terms of the Kubernetes API. However, I can imagine it might be desirable to have a way to invalidate existing allowed connections as a separate feature.

[0m1xa]

@caseydavenport Exactly! I'd like to see an additional feature to invalidate connections.

[sanmai-NL]

@caseydavenport Can you cite where is says that unequivocally?

[caseydavenport]

@sanmai-NL just searching for "connection" in that linked document in my previous comment brings up several examples, but these say it best:

NetworkPolicies apply to a connection with a pod on one or both ends

and

There are two sorts of isolation for a pod: isolation for egress, and isolation for ingress. They concern what connections may be established.

[sanmai-NL]

I think that indeed suggests but certainly doesn't assert exclusive applicability to connections. Indeed, that would be inconsistent since NetworkPolicys can cover UDP, which is not connection-oriented.

[caseydavenport]

Yes, UDP as a protocol isn't connection oriented, but there are still distinct UDP connections and UDP flows identified by the kernel and tracked via connection tracking (conntrack on Linux) much like TCP. NetworkPolicy applies to them similarly - e.g., you don't need to allow return traffic explicitly, it is identified as part of that UDP "connection" and allowed implicitly.

I think the discussion you're having in https://github.com/kubernetes/kubernetes/issues/120525 covers this pretty well.

• NetworkPolicy applies to connections rather than individual packets.
• Changing a NetworkPolicy has no defined impact on existing connections previously allowed by the policy.

It seems the stance from SIG network is now "the behavior is undefined", in which case the answer here remains that Calico's implementation does not impact any existing connections, and that it's a reasonable enhancement request to have an opt-in way to invalidate connections instead.

[zoglam]

Did you found another way to solve the problem? @0m1xa Maybe without using calico

[0m1xa]

Did you found another way to solve the problem? @0m1xa Maybe without using calico

@zoglam Yes, I did. Cilium :satisfied: