network bond on bare metal tectonic

[sorinel]

Hello,

We have 10 BareMetal servers for Tectonic Setup, the distribution is: 1 server for Matchbox 3 controllers 6 workers

Each server has 4 NICs which we are trying to bond to a single bonded interface. Is there a way to setup bonded NICs in Tectonic?

Our approach regarding bonding on Tectonic is:

1. Install CoreOS without machbox on all 9 machines with a custom Ignition config where all 9 servers are in the same etcd cluster.
2. Run Tectonic Installer on laptop and download assets.zip file
3. Upload assets.zip to one of the CoreOS Controllers
4. Run bootkube to start tectonic on all 9 servers

We have deployed bonded interfaces in CoreOS as below: https://coreos.com/ignition/docs/latest/network-configuration.html#bonded-nics

{
  "ignition": { "version": "2.0.0" },
  "networkd": {
    "units": [
      {
        "name": "00-eth.network",
        "contents": "[Match]\nName=eth*\n\n[Network]\nBond=bond0"
      },
      {
        "name": "10-bond0.netdev",
        "contents": "[NetDev]\nName=bond0\nKind=bond"
      },
      {
        "name": "20-bond0.network",
        "contents": "[Match]\nName=bond0\n\n[Network]\nDHCP=true"
      }
    ]
  }
}

Thank you, Alex

[dragoschiplink]

One can change the yaml.tmpl files and add:

networkd:
  units:
   - name: 10-int.network
     contents: |
       [Match]
       Name=enp*
       [Network]
       Bond=bond0
       LinkLocalAddressing=no
       IPv6AcceptRA=no
   - name: 20-bond0.netdev
     contents: |
       [NetDev]
       Name=bond0
       Kind=bond
       [Bond]
       Mode=balance-rr
       MIIMonitorSec=1
   - name: 30-bond0.network
     contents: |
       [Match]
       Name=bond0
       [Network]
       DHCP=true

[mfburnett]

Hey @sorinel, thanks for the question. I've looped in @dghubble here, who can help! He can get back to you in a few days.

[dghubble]

For those using matchbox, you could edit the controller and worker Container Linux Configs in /var/lib/matchbox and add networkd units similar to what @dragoschiplink showed. You can add templating elements if your machines are not homogeneous. Network examples are here and matchbox examples show complete Container Linux Configs for popular clusters.

Writing raw Ignition JSON by hand for each machine isn't recommended, but you could look at the configs created by the Tectonic Installer, edit them and use the ct tool, and provide them to each machine.

[sorinel]

Hello @dghubble , @dragoschiplink , @mfburnett ! Thanks for the help 👍 I've got tectonic running on bond interfaces now :)

Below are the modifications manually made to /var/lib/matchbox:

1. Added to networkd block to the yaml templates below:

   /var/lib/matchbox/tectonic-worker.yaml.tmpl 
   /var/lib/matchbox/tectonic-controller.yaml.tmpl

core@matchbox /var/lib/matchbox/ignition $ cat tectonic-worker.yaml.tmpl
---
networkd:
  units:
   - name: 10-int.network
     contents: |
       [Match]
       Name=enp*
       [Network]
       Bond=bond0
       LinkLocalAddressing=no
       IPv6AcceptRA=no
   - name: 20-bond0.netdev
     contents: |
       [NetDev]
       Name=bond0
       Kind=bond
       [Bond]
       Mode=balance-rr
       MIIMonitorSec=1
   - name: 30-bond0.network
     contents: |
       [Match]
       Name=bond0
       [Network]
       DNS=192.168.1.1
       Address={{.bond0_IP}}/24
       Gateway=192.168.1.1

systemd:
  units:
    - name: etcd-member.service
...............

core@ceph-node01 /var/lib/matchbox/ignition $ cat tectonic-controller.yaml.tmpl
---
networkd:
  units:
   - name: 10-int.network
     contents: |
       [Match]
       Name=enp*
       [Network]
       Bond=bond0
       LinkLocalAddressing=no
       IPv6AcceptRA=no
   - name: 20-bond0.netdev
     contents: |
       [NetDev]
       Name=bond0
       Kind=bond
       [Bond]
       Mode=balance-rr
       MIIMonitorSec=1
   - name: 30-bond0.network
     contents: |
       [Match]
       Name=bond0
       [Network]
       DNS=192.168.1.1
       Address={{.bond0_IP}}/24
       Gateway=192.168.1.1

systemd:

............

2. Because we wanted static IP set on the bond interfaces, we set for each node in: /var/lib/matchbox/groups the IP for bond0:

core@ceph-node01 /var/lib/matchbox/groups $ cat tectonic-node-00-11-22-33-44-55.json  | more
{
        "id": "tectonic-node-00-11-22-33-44-55",
        "profile": "tectonic-worker",
        "selector": {
                "mac": "00:11:22:33:44:55",
                "os": "installed"
        },
        "metadata": {
                "domain_name": "tectonic5.example.com",
                "bond0_IP": "192.168.1.36",
                "etcd_endpoints": "controller1.example.com:2379,controller2.example.com:2379,controller3.example.com:2379",
                "external_etcd": false,
............

[dragoschiplink]

Hello @sorinel,

If you use static IPs, you should modify wait-for-dns.service and change 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done' to something like 'while ! /usr/bin/dig +short quay.io; do sleep 1; done' because /etc/resolv.conf will have something in it before network is ready.

Regards.

Dragos