More Flexible Network Configuration

The network configuration for bare-metal nodes needs to be different from node to node. The way the network bonds are configured must reflect the actual cabling setup. For identification purposes MAC addresses are pulled from a central "repository".

Additionally, the different node types have different network setup.

Current implementation uses a CRD ExternalNode to allow for network configuration synonymously to the Bootcfg Ignition Template. Looks like this:

apiVersion: kubernikus.sap.cc/v1
kind: ExternalNode
metadata:
  name: master0.cc.bla.cloud.sap

spec:
  ipxe: 12:3A:7D:6A:48:F1
  networks:
    - name: eth0
      match:
        macAddress: 12:df:2f:67:1e:85
      network:
        lldp: "yes"
        dhcp: "no"
        bond: bond1
    - name: eth1
      match:
        macAddress: 70:df:2f:67:1e:86
      network:
        lldp: "yes"
        dhcp: "no"
        bond: bond1
    - name: eth2
      match:
        macAddress: 12:df:2f:66:dc:d5
      network:
        lldp: "yes"
        dhcp: "no"
        bond: bond2
    - name: eth3
      match:
        macAddress: 12:df:2f:66:dc:d6
      network:
        lldp: "yes"
        dhcp: "no"
        bond: bond2
    - name: bond1
      match:
        name: bond1
      network:
        dhcp: "no"
    - name: bond2
      match:
        name: bond2
      network:
        dhcp: "no"
        address: 1.4.7.2/29
        gateway: 1.4.7.1
        dns: ["1.2.9.200", "1.2.9.201"]
        domains: bla..cloud.sap
  netdevs:
    - name: bond1
      netdev:
        name: bond1
        kind: bond
        mtuBytes: 9000
      bond:
        mode: 802.3ad 
        mimMonitorSec: 1s
        lacpTransmitRate: fast
        upDelaySec: 3s
        downDelaySec: 3s
        minLinks: 1
    - name: bond2
      netdev:
        name: bond2
        kind: bond
        mtuBytes: 9000
      bond:
        mode: 802.3ad 
        mimMonitorSec: 1s
        lacpTransmitRate: fast
        upDelaySec: 3s
        downDelaySec: 3s
        minLinks: 1

https://github.com/sapcc/kubernikus/blob/73317a6d301a2fec889d295259b76048a5b4d1b5/pkg/templates/baremetal_node_1.10.go#L144-L212

It generates equivalent Ignition. Here the relevant network config:

  "networkd": {
    "units": [
      {
        "name": "eth0.network",
        "contents": "[Match]\nMACAddress=12:df:2f:67:1e:85\n[Network]\nDHCP=no\nLLDP=yes\nBond=bond1\n"
      },
      {
        "name": "eth1.network",
        "contents": "[Match]\nMACAddress=12:df:2f:67:1e:86\n[Network]\nDHCP=no\nLLDP=yes\nBond=bond1\n"
      },
      {
        "name": "eth2.network",
        "contents": "[Match]\nMACAddress=12:df:2f:66:dc:d5\n[Network]\nDHCP=no\nLLDP=yes\nBond=bond2\n"
      },
      {
        "name": "eth3.network",
        "contents": "[Match]\nMACAddress=12:df:2f:66:dc:d6\n[Network]\nDHCP=no\nLLDP=yes\nBond=bond2\n"
      },
      {
        "name": "bond1.network",
        "contents": "[Match]\nName=bond1\n[Network]\nDHCP=no\n"
      },
      {
        "name": "bond2.network",
        "contents": "[Match]\nName=bond2\n[Network]\nDHCP=no\nAddress=1.4.7.2/29\nGateway=1.4.7.1\nDNS=1.2.9.200,1.2.9.201\nDomains=kubernetes.bla.cloud.sap\n"
      },
      {
        "name": "bond1.netdev",
        "contents": "[NetDev]\nName=bond1\nKind=bond\nMTUBytes=9000\n[Bond]\nMode=802.3ad\nMIMMonitorSec=1s\nLACPTransmitRate=fast\nUpDelaySec=3s\nDownDelaySec=3s\nMinLinks=1\n"
      },
      {
        "name": "bond2.netdev",
        "contents": "[NetDev]\nName=bond1\nKind=bond\nMTUBytes=9000\n[Bond]\nMode=802.3ad\nMIMMonitorSec=1s\nLACPTransmitRate=fast\nUpDelaySec=3s\nDownDelaySec=3s\nMinLinks=1\n"
      }
    ]
  },

In order to retire the whole region.yaml config, I would imagine that we create a NetboxExternalNodeOperator that fishes in Netbox for the MAC addresses and types of the nodes. It also holds the logic to distinguish the different types of nodes that require slightly different configuration. The Ignition template in Kubernikus can be generic, while the CCloud specifics are outsourced to the NetboxExternalNodeOperator.

sapcc / kubernikus

More Flexible Network Configuration #278