search

k8snetworkplumbingwg / multi-networkpolicy-iptables

MultiNetworkPolicy iptable based implementation
Apache License 2.0
13 stars 20 forks source link

cannot express policy with only a `namespaceSelector` #14

Closed maiqueb closed 2 years ago

maiqueb commented 2 years ago

When attempting to configure the following policy (which just uses a namespaceSelector in the policy's from clause), the ...-FROM rules (the ones marked w/ 0x20000) are not generated:

apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: vmi-zero-trust-secondary-network
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/policy-for: wb-vm-network
spec:
  podSelector:
    labelSelector:
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels: 
          role: allowed-namespace
    ports:
    - protocol: TCP
      port: 80

Looking at the code, it seems the rules generator always requires a podSelector for the rules to be generated.

When reading the Kubernetes network policy documentation, it can be seen that using the namespaceSelector without a podSelector is a valid option.

I can also say that when an empty pod selector is added to the from clause provided above, the expected rule is generated.

s1061123 commented 2 years ago

Hi @maiqueb , thank you for reporting the issue.

I could reproduce it and make a fix candidate. Could you please try this image, quay.io/s1061123/multi-networkpolicy-iptables:test ? (this image is for iptable-legacy based one. Please let me know if you need iptables-nft based image)

maiqueb commented 2 years ago

Hi @maiqueb , thank you for reporting the issue.

I could reproduce it and make a fix candidate. Could you please try this image, quay.io/s1061123/multi-networkpolicy-iptables:test ? (this image is for iptable-legacy based one. Please let me know if you need iptables-nft based image)

@s1061123 even worse now: I have no rules at all. This is the iptables output I get from the policy provided in this issue:

root@kind-worker:/# nsenter -t 26128 -n iptables-legacy -L -v 
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MULTI-INGRESS  all  --  net1   any     anywhere             anywhere            

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MULTI-EGRESS  all  --  any    net1    anywhere             anywhere            

Chain MULTI-EGRESS (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain MULTI-INGRESS (1 references)
 pkts bytes target     prot opt in     out     source               destination

And yes, I'm sure I'm using the correct image:

kubectl get ds -nkube-system multi-networkpolicy-ds-amd64 -ojsonpath="{ @.spec.template.spec.containers[0].image }"
quay.io/s1061123/multi-networkpolicy-iptables:test%
s1061123 commented 2 years ago

oh, sorry to let you know wrong image. Please use 'quay.io/s1061123/multi-networkpolicy-iptables:test2', not 'quay.io/s1061123/multi-networkpolicy-iptables:test'

s1061123 commented 2 years ago

hmm... wait a sec.... will let you know the right image.

s1061123 commented 2 years ago

Hmm.. You use right image (so 'quay.io/s1061123/multi-networkpolicy-iptables:test' is right image), however, your result is different from mine.

With this image and following yaml (in Fedora36+k8s 1.23.6), I could have right iptables output (which contains the pod which matches namespace selector), as following.

Could you please share the following information to reproduce it?

---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-net1
spec:
  config: '{
            "cniVersion": "0.3.1",
            "type": "macvlan",
            "capabilities": { "ips": true },
            "master": "eth1",
            "mode": "bridge",
            "ipam": {
                "type": "static"
            }
        }'
---
apiVersion: v1
kind: Pod
metadata:
  name: centos
  labels:
    role: client
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
            { "name": "macvlan-net1", "ips": ["10.1.1.11/24"] }
    ]'
spec:
  containers:
  - name: main
    image: docker.io/centos/tools:latest
    command:
    - /sbin/init
    securityContext:
      privileged: true
---
apiVersion: v1
kind: Pod
metadata:
  name: centos-not-client
  labels:
    role: not-client
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
            { "name": "macvlan-net1", "ips": ["10.1.1.12/24"] }
    ]'
spec:
  containers:
  - name: main
    image: docker.io/centos/tools:latest
    command:
    - /sbin/init
    securityContext:
      privileged: true
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    role: http
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
            { "name": "macvlan-net1", "ips": ["10.1.1.13/24"] }
    ]'
spec:
  containers:
  - name: main
    image: docker.io/s1061123/centos-tools-nginx:latest
    securityContext:
      privileged: true
[tohayash@tohayash-srv multi-networkpolicy-iptables-s1061123]$ cat ~/src/centos-pods/multi-networkpolicy/test220422/sample-policy.yml
apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: policy-test1
  annotations:
    k8s.v1.cni.cncf.io/policy-for: macvlan-net1
spec:
  podSelector:
    matchLabels:
  policyTypes:
    - ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: default
    ports:
    - port: 80
      protocol: TCP
$ kubectl exec -it nginx  -- bash
[root@nginx /]# iptables-save
# Generated by iptables-save v1.4.21 on Fri Apr 22 13:35:25 2022
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Fri Apr 22 13:35:25 2022
# Generated by iptables-save v1.4.21 on Fri Apr 22 13:35:25 2022
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -i net1 -j RETURN
COMMIT
# Completed on Fri Apr 22 13:35:25 2022
# Generated by iptables-save v1.4.21 on Fri Apr 22 13:35:25 2022
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:MULTI-0-INGRESS - [0:0]
:MULTI-0-INGRESS-0-FROM - [0:0]
:MULTI-0-INGRESS-0-PORTS - [0:0]
:MULTI-EGRESS - [0:0]
:MULTI-INGRESS - [0:0]
-A INPUT -i net1 -j MULTI-INGRESS
-A OUTPUT -o net1 -j MULTI-EGRESS
-A MULTI-0-INGRESS -j MARK --set-xmark 0x0/0x30000
-A MULTI-0-INGRESS -j MULTI-0-INGRESS-0-PORTS
-A MULTI-0-INGRESS -j MULTI-0-INGRESS-0-FROM
-A MULTI-0-INGRESS -m mark --mark 0x30000/0x30000 -j RETURN
-A MULTI-0-INGRESS -j DROP
-A MULTI-0-INGRESS-0-FROM -s 10.1.1.11/32 -i net1 -j MARK --set-xmark 0x20000/0x20000
-A MULTI-0-INGRESS-0-FROM -s 10.1.1.12/32 -i net1 -j MARK --set-xmark 0x20000/0x20000
-A MULTI-0-INGRESS-0-FROM -s 10.1.1.13/32 -i net1 -j MARK --set-xmark 0x20000/0x20000
-A MULTI-0-INGRESS-0-PORTS -i net1 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x10000/0x10000
-A MULTI-INGRESS -i net1 -m comment --comment "policy:policy-test1 net-attach-def:default/macvlan-net1" -j MULTI-0-INGRESS
COMMIT
# Completed on Fri Apr 22 13:35:25 2022
[root@nginx /]#
s1061123 commented 2 years ago

I also test with kind and get correct iptables result with my fix candidate, hence I will proceed the fix.

$ kind version
kind v0.12.0 go1.17.8 linux/amd64
$ kubectl get ds -nkube-system multi-networkpolicy-ds-amd64 -ojsonpath="{ @.spec.template.spec.containers[0].image }"
quay.io/s1061123/multi-networkpolicy-iptables:test
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-net1
spec:
  config: '{
            "cniVersion": "0.3.1",
            "type": "macvlan",
            "capabilities": { "ips": true },
            "master": "eth1",
            "mode": "bridge",
            "ipam": {
                "type": "static"
            }
        }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-net2
spec:
  config: '{
            "cniVersion": "0.3.1",
            "type": "macvlan",
            "capabilities": { "ips": true },
            "master": "eth0",
            "mode": "bridge",
            "ipam": {
                "type": "static"
            }
        }'
---
apiVersion: v1
kind: Pod
metadata:
  name: centos
  labels:
    role: client
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
            { "name": "macvlan-net1", "ips": ["10.1.1.11/24"] },
            { "name": "macvlan-net2", "ips": ["22.1.1.11/24"] }
    ]'
spec:
  containers:
  - name: main
    image: docker.io/centos/tools:latest
    command:
    - /sbin/init
    securityContext:
      privileged: true
---
apiVersion: v1
kind: Pod
metadata:
  name: centos-not-client
  labels:
    role: not-client
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
            { "name": "macvlan-net1", "ips": ["10.1.1.12/24"] },
            { "name": "macvlan-net2", "ips": ["22.1.1.12/24"] }
    ]'
spec:
  containers:
  - name: main
    image: docker.io/centos/tools:latest
    command:
    - /sbin/init
    securityContext:
      privileged: true
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    role: http
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
            { "name": "macvlan-net1", "ips": ["10.1.1.13/24"] },
            { "name": "macvlan-net2", "ips": ["22.1.1.13/24"] }
    ]'
spec:
  containers:
  - name: main
    image: docker.io/s1061123/centos-tools-nginx:latest
    securityContext:
      privileged: true
---
apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: policy-test1
  annotations:
    k8s.v1.cni.cncf.io/policy-for: macvlan-net1
spec:
  podSelector:
    matchLabels:
      role: http
  policyTypes:
    - ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: default
    ports:
    - port: 80
      protocol: TCP
$ kubectl exec -it nginx -- bash
[root@nginx /]# iptables-save
# Generated by iptables-save v1.4.21 on Fri Apr 29 16:15:37 2022
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Fri Apr 29 16:15:37 2022
# Generated by iptables-save v1.4.21 on Fri Apr 29 16:15:37 2022
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -i net2 -j RETURN
-A PREROUTING -i net1 -j RETURN
COMMIT
# Completed on Fri Apr 29 16:15:37 2022
# Generated by iptables-save v1.4.21 on Fri Apr 29 16:15:37 2022
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:MULTI-0-INGRESS - [0:0]
:MULTI-0-INGRESS-0-FROM - [0:0]
:MULTI-0-INGRESS-0-PORTS - [0:0]
:MULTI-EGRESS - [0:0]
:MULTI-INGRESS - [0:0]
-A INPUT -i net2 -j MULTI-INGRESS
-A INPUT -i net1 -j MULTI-INGRESS
-A OUTPUT -o net2 -j MULTI-EGRESS
-A OUTPUT -o net1 -j MULTI-EGRESS
-A MULTI-0-INGRESS -j MARK --set-xmark 0x0/0x30000
-A MULTI-0-INGRESS -j MULTI-0-INGRESS-0-PORTS
-A MULTI-0-INGRESS -j MULTI-0-INGRESS-0-FROM
-A MULTI-0-INGRESS -m mark --mark 0x30000/0x30000 -j RETURN
-A MULTI-0-INGRESS -j DROP
-A MULTI-0-INGRESS-0-FROM -s 10.1.1.13/32 -i net1 -j MARK --set-xmark 0x20000/0x20000
-A MULTI-0-INGRESS-0-FROM -s 10.1.1.11/32 -i net1 -j MARK --set-xmark 0x20000/0x20000
-A MULTI-0-INGRESS-0-FROM -s 10.1.1.12/32 -i net1 -j MARK --set-xmark 0x20000/0x20000
-A MULTI-0-INGRESS-0-PORTS -i net1 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x10000/0x10000
-A MULTI-INGRESS -i net1 -m comment --comment "policy:policy-test1 net-attach-def:default/macvlan-net1" -j MULTI-0-INGRESS
COMMIT
# Completed on Fri Apr 29 16:15:37 2022
maiqueb commented 2 years ago

Hmm.. You use right image (so 'quay.io/s1061123/multi-networkpolicy-iptables:test' is right image), however, your result is different from mine.

I'll reply with the data you're requesting below once I can find the time to work on this.

With this image and following yaml (in Fedora36+k8s 1.23.6), I could have right iptables output (which contains the pod which matches namespace selector), as following.

Could you please share the following information to reproduce it?

  • Linux distribution name + version
  • Kubernetes version
  • Container runtime version
  • net-attach-def yaml file
  • Pod yaml file
  • multi-networkpolicy yaml (if you update policy from above one)
s1061123 commented 2 years ago

At least, I reproduce the issue from my side. If you don't mind that, let me proceed the issue fix first. Then you can repro with latest image again when you have a time and file another issue if you still have one.

What do you think about it?

maiqueb commented 2 years ago

Well, I am unsure. As I said in this comment, I have worse results now (less rules installed).

But, I think you can open (and should) create the PR so I can try to make sense of the results I had.

EDIT: and yes, I can always open another bug with the new behavior.