Bridge interface down - Githubissues

rayjanoka commented 1 year ago

Hey there, I can't seem to get my networking to function.

I'm trying to setup a network configuration that is migratable. We are using multus with cilium, and I'm trying to just use the bridge plugin as a secondary interface.

But whatever I try I can't seem to get the bridge plugin to work with virtink while the bridge plugin does seem to work with non-virtink pods.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: test-bridge
  namespace: virtink-system
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "bridge",
      "bridge": "mybr0",
      "ipam": {
          "type": "host-local",
          "subnet": "192.168.12.0/24",
          "rangeStart": "192.168.12.10",
          "rangeEnd": "192.168.12.200",
          "gateway": "192.168.12.1"
       }
    }

apiVersion: virt.virtink.smartx.com/v1alpha1
kind: VirtualMachine
metadata:
  name: ubuntu-container-rootfs
spec:
  instance:
    memory:
      size: 1Gi
    kernel:
      image: smartxworks/virtink-kernel-5.15.12
      cmdline: "console=ttyS0 root=/dev/vda rw"
    disks:
      - name: ubuntu
      - name: cloud-init
    interfaces:
      #- name: pod
      - name: migration
        bridge: {}
  volumes:
    - name: ubuntu
      containerRootfs:
        image: smartxworks/virtink-container-rootfs-ubuntu
        size: 4Gi
    - name: cloud-init
      cloudInit:
        userData: |-
          #cloud-config
          password: password
          chpasswd: { expire: False }
          ssh_pwauth: True
  networks:
    #- name: pod
    #  pod: {}
    - name: migration
      multus:
        networkName: test-bridge

the net0 interface is always DOWN

/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: net0-nic@if46: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-net0 state UP group default
    link/ether 52:54:00:af:e1:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::5054:ff:feaf:e17a/64 scope link
       valid_lft forever preferred_lft forever
3: br-net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 52:54:00:af:e1:7a brd ff:ff:ff:ff:ff:ff
    inet 169.254.200.1/30 brd 169.254.200.3 scope global br-net0
       valid_lft forever preferred_lft forever
    inet6 fe80::90ab:8bff:fe1d:a85/64 scope link
       valid_lft forever preferred_lft forever
4: net0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
    link/ether 1a:cc:07:82:31:d8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.12.18/24 brd 192.168.12.255 scope global net0
       valid_lft forever preferred_lft forever
5: tap-net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-net0 state UP group default qlen 1000
    link/ether e6:7e:f9:99:fa:8c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e47e:f9ff:fe99:fa8c/64 scope link
       valid_lft forever preferred_lft forever
44: eth0@if45: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 32:29:fc:06:24:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 240.1.77.208/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::3029:fcff:fe06:2457/64 scope link
       valid_lft forever preferred_lft forever

I can reach the pod network/internet, but not the pods on the bridge plugin network.

➜ k exec -it vm-ubuntu-container-rootfs-62mz7 sh

/ # ip route
default via 240.1.77.88 dev eth0 mtu 1450
169.254.200.0/30 dev br-net0 proto kernel scope link src 169.254.200.1
240.1.77.88 dev eth0 scope link

/ # cat /var/run/virtink/dnsmasq/br-net0.conf
port=0
interface=br-net0
bind-interfaces
dhcp-range=192.168.12.24,static,255.255.255.0
dhcp-host=52:54:00:6f:7a:51,192.168.12.24,infinite
dhcp-option=option:classless-static-route,192.168.12.0/24,0.0.0.0
dhcp-option=option:dns-server,172.16.13.10
dhcp-option=option:domain-search,virtink-system.svc.cluster.local,svc.cluster.local,cluster.local,maas
dhcp-authoritative
shared-network=br-net0,192.168.12.24

/ # ping google.com
PING google.com (142.250.9.139): 56 data bytes
64 bytes from 142.250.9.139: seq=0 ttl=107 time=1.230 ms
^C
--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.230/1.230/1.230 ms

/ # ping 192.168.12.17
PING 192.168.12.17 (192.168.12.17): 56 data bytes
^C
--- 192.168.12.17 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
/ #

rayjanoka commented 1 year ago

well I may have figured it out with the help of another thread.

I added isGateway: true here:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: test-bridge
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "bridge",
      "bridge": "mybr0",
      "isGateway": true,
      "ipam": {
          "type": "host-local",
          "subnet": "192.168.12.0/24",
          "rangeStart": "192.168.12.10",
          "rangeEnd": "192.168.12.200",
          }
    }

fengye87 commented 1 year ago

@rayjanoka Glad to know your problem has been resolved. I'm going to close this issue, feel free to re-open it if your problem resists.

smartxworks / virtink

Bridge interface down #86