search

k8snetworkplumbingwg / multus-cni

A CNI meta-plugin for multi-homed pods in Kubernetes
Apache License 2.0
2.36k stars 585 forks source link

defaultNetworks always added even when k8s.v1.cni.cncf.io/networks says otherwise #1187

Closed akunszt closed 10 months ago

akunszt commented 10 months ago

What happened:

First of all, this is my first round with Multus CNI so this very likely just an oversight from my end. I read all the docs but they are very-very confusing about how to configure exactly and which directories meant for which cause.

We are running our clusters in AWS on EC2 instances. We are already using aws-vpc-cni but as that does not support dual-stack then we want to introduce a simple ipvlan CNI as well only for IPv6. It is working nicely but we cannot define the CNI configuration in the CRD as every node has a different CNI configuration.

As this change could cause unexpected issues we would like to be able to opt-out on a per-pod basis.

We are using this daemon configuration (the hiya is just our fancy name for a very specific ipvlan configuration):

{
        "chrootDir": "/hostroot",
        "clusterNetwork": "aws-cni",
        "cniConfigDir": "/host/etc/cni/net.d",
        "cniVersion": "0.3.1",
        "defaultNetworks": [
                "hiya"                                                                                                                                                       
        ],
        "logLevel": "verbose",
        "logToStderr": true,
        "multusAutoconfigDir": "/host/etc/cni/net.d",
        "multusConfigFile": "auto",
        "multusNamespace": "multus-cni",
        "namespaceIsolation": false,
        "readinessindicatorfile": "/host/etc/cni/net.d/10-aws.conflist",
        "socketDir": "/host/run/multus/",
        "systemNamespaces": [
                "aws-vpc-cni",
                "hiya-cni",
                "multus-cni"
        ]
}

As a test I wanted to create a pod only using aws-cni.

My goal is to find an annotation which makes pods to run only with aws-cni@eth0 for a few exceptions and have the default aws-cni@eth0, hiya@net1 for the rest of the pods.

Can this be achieved? Is there a documentation about the available annotations and what they does? For it looked like that the k8s.v1.cni.cncf.io/networks just appended to the configuration stored in the daemon-config.json file.

What you expected to happen:

The pod receive IP only from the aws-vpc-cni when using k8s.v1.cni.cncf.io/networks: multus-cni/aws-cni annotation.

How to reproduce it (as minimally and precisely as possible):

Create a daemon-config.json with defaultNetworks and add a k8s.v1.cni.cncf.io/networks annotation to the pod.

Anything else we need to know?:

I tried to keep the /etc/cni/net.d "empty" an use only CRDs but in that case Multus CNI refused to start.

Environment:

It is not available on the host's filesystem, this is from the running container. The files in net.d are exactly the same as the ones on the host as the /etc/cni/net.d was mounted into /etc/cni/multus/net.d too.

# find /etc/cni/
/etc/cni/
/etc/cni/multus
/etc/cni/multus/net.d
/etc/cni/multus/net.d/00-multus.conf
/etc/cni/multus/net.d/10-aws.conflist
/etc/cni/multus/net.d/20-hiya-cni.conflist
/etc/cni/net.d
/etc/cni/net.d/multus.d
/etc/cni/net.d/multus.d/..2023_11_16_13_22_57.484744670
/etc/cni/net.d/multus.d/..2023_11_16_13_22_57.484744670/daemon-config.json
/etc/cni/net.d/multus.d/..data
/etc/cni/net.d/multus.d/daemon-config.json
apiVersion: v1
items:
- apiVersion: k8s.cni.cncf.io/v1
  kind: NetworkAttachmentDefinition
  metadata:
    creationTimestamp: "2023-11-16T13:00:20Z"
    generation: 1
    name: aws-cni
    namespace: multus-cni
    resourceVersion: "322700112"
    uid: 3347cd89-c9d8-4165-b76c-4b6979594594
- apiVersion: k8s.cni.cncf.io/v1
  kind: NetworkAttachmentDefinition
  metadata:
    creationTimestamp: "2023-11-16T13:00:20Z"
    generation: 1
    name: hiya
    namespace: multus-cni
    resourceVersion: "322700117"
    uid: b188d5ea-f0b1-4232-b1f8-feb7ef494949
kind: List
metadata:
  resourceVersion: ""
apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "multus-cni/aws-cni",
          "interface": "eth0",
          "ips": [
              "10.32.0.210"
          ],
          "default": true,
          "dns": {}
      },{
          "name": "multus-cni/hiya",
          "interface": "net1",
          "ips": [
              "2a05:d018:20e:c400:d32f::1b"
          ],
          "mac": "02:7c:be:95:01:ab",
          "dns": {},
          "gateway": [
              "\u003cnil\u003e"
          ]
      },{
          "name": "multus-cni/aws-cni",
          "interface": "net2",
          "ips": [
              "10.32.0.94"
          ],
          "dns": {}
      }]
    k8s.v1.cni.cncf.io/networks: multus-cni/aws-cni
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"k8s.v1.cni.cncf.io/networks":"multus-cni/aws-cni"},"name":"akunszt-test","namespace":"akunszt"},"spec":{"containers":[{"args":["-c","while true; do sleep 2; done"],"command":["/bin/sh"],"image":"hiya.fi/ext/alpine:3.15","name":"shell"}],"restartPolicy":"Never"}}
  creationTimestamp: "2023-11-16T13:58:15Z"
  labels:
    tag.hiya.com/environment: dev
    tag.hiya.com/required-tags-origin: infrastructure-as-code
    tag.hiya.com/system: kubernetes
    tag.hiya.com/system-component: kubernetes-namespace
    tag.hiya.com/team: org-coretech
  name: akunszt-test
  namespace: akunszt
  resourceVersion: "322739918"
  uid: 26850ae0-1c8a-40c1-9c4e-106c6493e92c
spec:
  containers:
  - args:
    - -c
    - while true; do sleep 2; done
    command:
    - /bin/sh
    image: hiya.fi/ext/alpine:3.15
    imagePullPolicy: IfNotPresent
    name: shell
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-hhtvb
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: false
  nodeName: ip-10-32-10-4.eu-west-1.compute.internal
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-hhtvb
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-11-16T13:58:15Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-11-16T13:58:19Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-11-16T13:58:19Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-11-16T13:58:15Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://23f58d14f9f0f543ccb124c63205184f34e7f00fa36bc3c42d68eb691f81dc3c
    image: hiya.fi/ext/alpine:3.15
    imageID: hiya.fi/ext/alpine@sha256:6f7de56eee2ae8d16dd436236269ab457ee80116574a362dda3836ae2873a3fb
    lastState: {}
    name: shell
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-11-16T13:58:18Z"
  hostIP: 10.32.10.4
  phase: Running
  podIP: 10.32.0.210
  podIPs:
  - ip: 10.32.0.210
  qosClass: BestEffort
  startTime: "2023-11-16T13:58:15Z"
2023-11-16T13:22:58Z [verbose] server configured with chroot: /hostroot
2023-11-16T13:22:58Z [verbose] Generated MultusCNI config: {"capabilities":{"portMappings":true},"cniVersion":"0.3.1","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","clusterNetwork":"/host/etc/cni/net.d/10-aws.conflist","readinessindicatorfile":"/host/etc/cni/net.d/10-aws.conflist","type":"multus-shim","socketDir":"/host/run/multus/"}
2023-11-16T13:22:58Z [verbose] started to watch file /host/etc/cni/net.d/10-aws.conflist
...
2023-11-16T13:58:15Z [verbose] ADD starting CNI request &{ContainerID:125bc20c21fd51a0adfda6727e6d9403ee18d29aee56f13bfa9845e2639e0e23 Netns:/var/run/netns/cni-b506ba5b-dbac-decb-ba7f-e98e971e26e9 IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=akunszt;K8S_POD_NAME=akunszt-test;K8S_POD_INFRA_CONTAINER_ID=125bc20c21fd51a0adfda6727e6d9403ee18d29aee56f13bfa9845e2639e0e23;K8S_POD_UID=26850ae0-1c8a-40c1-9c4e-106c6493e92c Path: StdinData:[123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 49 48 45 97 119 115 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 108 111 103 84 111 83 116 100 101 114 114 34 58 116 114 117 101 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 114 101 97 100 105 110 101 115 115 105 110 100 105 99 97 116 111 114 102 105 108 101 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 49 48 45 97 119 115 46 99 111 110 102 108 105 115 116 34 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 44 10 9 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 9 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 32 34 97 119 115 45 99 110 105 34 44 10 9 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 9 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 9 34 100 101 102 97 117 108 116 78 101 116 119 111 114 107 115 34 58 32 91 10 9 9 34 104 105 121 97 34 10 9 93 44 10 9 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 9 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 9 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 9 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 9 34 109 117 108 116 117 115 78 97 109 101 115 112 97 99 101 34 58 32 34 109 117 108 116 117 115 45 99 110 105 34 44 10 9 34 110 97 109 101 115 112 97 99 101 73 115 111 108 97 116 105 111 110 34 58 32 102 97 108 115 101 44 10 9 34 114 101 97 100 105 110 101 115 115 105 110 100 105 99 97 116 111 114 102 105 108 101 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 49 48 45 97 119 115 46 99 111 110 102 108 105 115 116 34 44 10 9 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 10 9 34 115 121 115 116 101 109 78 97 109 101 115 112 97 99 101 115 34 58 32 91 10 9 9 34 97 119 115 45 118 112 99 45 99 110 105 34 44 10 9 9 34 104 105 121 97 45 99 110 105 34 44 10 9 9 34 109 117 108 116 117 115 45 99 110 105 34 10 9 93 10 125 10]}
2023-11-16T13:58:15Z [verbose] Add: akunszt:akunszt-test:26850ae0-1c8a-40c1-9c4e-106c6493e92c:multus-cni/aws-cni(aws-cni):eth0 {"cniVersion":"0.4.0","interfaces":[{"name":"pod49384573cee"},{"name":"eth0","sandbox":"/var/run/netns/cni-b506ba5b-dbac-decb-ba7f-e98e971e26e9"}],"ips":[{"version":"4","interface":1,"address":"10.32.0.210/32"}],"dns":{}}
I1116 13:58:15.912913   83496 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"akunszt", Name:"akunszt-test", UID:"26850ae0-1c8a-40c1-9c4e-106c6493e92c", APIVersion:"v1", ResourceVersion:"322739879", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [10.32.0.210/32] from multus-cni/aws-cni
2023-11-16T13:58:17Z [verbose] Add: akunszt:akunszt-test:26850ae0-1c8a-40c1-9c4e-106c6493e92c:multus-cni/hiya(hiya):net1 {"cniVersion":"1.0.0","interfaces":[{"name":"net1","mac":"02:7c:be:95:01:ab","sandbox":"/var/run/netns/cni-b506ba5b-dbac-decb-ba7f-e98e971e26e9"}],"ips":[{"interface":0,"address":"2a05:d018:20e:c400:d32f::1b/64","gateway":"2a05:d018:20e:c400::1"}],"routes":[{"dst":"::/0"}],"dns":{}}
I1116 13:58:17.977652   83496 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"akunszt", Name:"akunszt-test", UID:"26850ae0-1c8a-40c1-9c4e-106c6493e92c", APIVersion:"v1", ResourceVersion:"322739879", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add net1 [2a05:d018:20e:c400:d32f::1b/64] from multus-cni/hiya
2023-11-16T13:58:18Z [verbose] Add: akunszt:akunszt-test:26850ae0-1c8a-40c1-9c4e-106c6493e92c:multus-cni/aws-cni(aws-cni):net2 {"cniVersion":"0.4.0","interfaces":[{"name":"pod49384573cee"},{"name":"net2","sandbox":"/var/run/netns/cni-b506ba5b-dbac-decb-ba7f-e98e971e26e9"}],"ips":[{"version":"4","interface":1,"address":"10.32.0.94/32"}],"dns":{}}
I1116 13:58:18.034736   83496 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"akunszt", Name:"akunszt-test", UID:"26850ae0-1c8a-40c1-9c4e-106c6493e92c", APIVersion:"v1", ResourceVersion:"322739879", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add net2 [10.32.0.94/32] from multus-cni/aws-cni
2023-11-16T13:58:18Z [verbose] ADD finished CNI request {ContainerID:125bc20c21fd51a0adfda6727e6d9403ee18d29aee56f13bfa9845e2639e0e23 Netns:/var/run/netns/cni-b506ba5b-dbac-decb-ba7f-e98e971e26e9 IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=akunszt;K8S_POD_NAME=akunszt-test;K8S_POD_INFRA_CONTAINER_ID=125bc20c21fd51a0adfda6727e6d9403ee18d29aee56f13bfa9845e2639e0e23;K8S_POD_UID=26850ae0-1c8a-40c1-9c4e-106c6493e92c Path: StdinData:[123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 49 48 45 97 119 115 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 108 111 103 84 111 83 116 100 101 114 114 34 58 116 114 117 101 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 114 101 97 100 105 110 101 115 115 105 110 100 105 99 97 116 111 114 102 105 108 101 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 49 48 45 97 119 115 46 99 111 110 102 108 105 115 116 34 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 44 10 9 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 9 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 32 34 97 119 115 45 99 110 105 34 44 10 9 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 9 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 9 34 100 101 102 97 117 108 116 78 101 116 119 111 114 107 115 34 58 32 91 10 9 9 34 104 105 121 97 34 10 9 93 44 10 9 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 9 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 9 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 9 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 9 34 109 117 108 116 117 115 78 97 109 101 115 112 97 99 101 34 58 32 34 109 117 108 116 117 115 45 99 110 105 34 44 10 9 34 110 97 109 101 115 112 97 99 101 73 115 111 108 97 116 105 111 110 34 58 32 102 97 108 115 101 44 10 9 34 114 101 97 100 105 110 101 115 115 105 110 100 105 99 97 116 111 114 102 105 108 101 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 49 48 45 97 119 115 46 99 111 110 102 108 105 115 116 34 44 10 9 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 10 9 34 115 121 115 116 101 109 78 97 109 101 115 112 97 99 101 115 34 58 32 91 10 9 9 34 97 119 115 45 118 112 99 45 99 110 105 34 44 10 9 9 34 104 105 121 97 45 99 110 105 34 44 10 9 9 34 109 117 108 116 117 115 45 99 110 105 34 10 9 93 10 125 10]}, result: "{\"Result\":{\"cniVersion\":\"1.0.0\",\"interfaces\":[{\"name\":\"pod49384573cee\"},{\"name\":\"eth0\",\"sandbox\":\"/var/run/netns/cni-b506ba5b-dbac-decb-ba7f-e98e971e26e9\"}],\"ips\":[{\"interface\":1,\"address\":\"10.32.0.210/32\"}],\"dns\":{}}}", err: <nil>
s1061123 commented 10 months ago

defaultNetworks makes target net-attach-def is 'default', this means, all pod have this net-attach-def.

k8s.v1.cni.cncf.io/networks: multus-cni/aws-cni - results aws-cni@eth0, hiya@net1, aws-cni@net2 which was unexpected, especially the hiya@net1 part So in this case, clusterNetwork + hiya (defaultNetwork) + aws-cni is expected results.

k8s.v1.cni.cncf.io/networks: "" - results aws-cni@eth0, hiya@net1 This is same as no annotation, k8s.v1.cni.cncf.io/networks.

v1.multus-cni.io/default-network: multus-cni/aws-cni - results aws-cni@eth0, hiya@net1, it was ignored? "v1.multus-cni.io/default-network" just replaces clusterNetwork hence aws-cni is eth0. (but the annotation is not recommend to use, we don't officially support that annotation).

I guess that you want to have the net-attach-def

Currently we don't have such option yet. We only have 'defaultNetwork' (always enabled secondary network interface) and 'clusterNetwork' (of course enabled for cluster network/primary network interface)