search

nmstate / kubernetes-nmstate

Declarative node network configuration driven through Kubernetes API.
GNU General Public License v2.0
186 stars 90 forks source link

RFE: Precedence for applying NNCPs #1101

Open rajinator opened 2 years ago

rajinator commented 2 years ago

What happened: When NNCPs are separated by layer of components on the same node/set of nodes, for example:

  1. Base interface/base bond NNCP
  2. NNCPS for subinterfaces to define VLANs and IPs/Routes

There is no order of precedence so upon node reboots, there's a chance that the controller may try to apply sub-interface NNCP before Base interface NNCP

This kind of use-case is essential when having to do interface configuration/modification on the fly without rebooting nodes as much as possible.

What you expected to happen: Base interface configured first followed by subinterfaces: e.g.:

  1. NNCP for bond1 applied first
  2. NNCP forbond1.xyz....bond1.nnn applied after 1 is successfully applied

How to reproduce it (as minimally and precisely as possible): Configure nodes with multiple NNCPs as seen above:

  1. NNCP for bond1
  2. NNCP for bond1.xyz...bond1.nnn with Static IPs and/or routes

Reboot the nodes and observe NNCPs/NNCEs

Anything else we need to know?:

Environment:

phoracek commented 2 years ago

Hello @rajinator. Thanks for bringing this up.

The lack of ordering/merging was a design decision. The motivation was to keep kubernetes-nmstate as a mere shim, without understanding of the contents of the configuration (which is responsibility of nmstate itself). If there is a dependency between two interfaces, they must be defined in a single NNCP.

That being said, this would be clearly a useful feature. The questions are how much complexity it would add to knmstate, how would it affect debugging, and if we could implement this without understanding semantics of the desired state.

qinqon commented 2 years ago

Maybe we can implement a new NodeNetworkConfigurationbTemplate that has some capture placeholders and then have a desiredStateFromTemplate at NNCP to reference it, then the NNCP will fill in the placeholders creating a pair of captures with the IPs.

Something like the following:

piVersion: nmstate.io/v1
kind: NodeNetworkConfigurationTemplate
metadata:
  name: cnf-bond-vlan-bridge
spec:
  snippet:
    interfaces:
    - description: ens1f2
      ipv4:
        enabled: false
      ipv6:
        enabled: false
      mtu: 9000
      name: ens1f2
      state: up
      type: ethernet
    - description: ens1f3
      ipv4:
        enabled: false
      ipv6:
        enabled: false
      mtu: 9000
      name: ens1f3
      state: up
      type: ethernet
    - description: Bond ens1f2 and ens1f3
      ipv4:
        enabled: false
      ipv6:
        enabled: false
      link-aggregation:
        mode: active-backup
        options:
          miimon: "100"
          primary: ens1f2
        slaves:
        - ens1f2
        - ens1f3
      mtu: 9000
      name: bond1
      state: up
      type: bond
    - description: vlan 123 bond1
      ipv4:
        enabled: false
      name: bond1.123
      state: up
      type: vlan
      vlan:
        base-iface: bond1
        id: 123
    - description: vlan 456 bond1
      ipv4:
        enabled: false
      name: bond1.456
      state: up
      type: vlan
      vlan:
        base-iface: bond1
        id: 456
    - description: iscsi bond0
      ipv4:
        address:
        - ip: "{{ capture.ip-bond0 }}"
          prefix-length: 24
        dhcp: false
        enabled: true
      mtu: 9000
      name: bond0.111
      state: up
      type: vlan
      vlan:
        base-iface: bond0
        id: 111
    - description: iscsi bond1
      ipv4:
        address:
        - ip: "{{ capture.ip-bond1 }}"
          prefix-length: 24
        dhcp: false
        enabled: true
      mtu: 9000
      name: bond1.111
      state: up
      type: vlan
      vlan:
        base-iface: bond1
        id: 111
---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: cnf-bond-vlan-bridge-node1
spec:
  capture: 
    ip-bond0: 192.168.1.12
    ip-bond1: 192.168.2.12
  desiredStateFromTemplate: cnf-bond-vlan-bridge
  nodeSelector:
    kubernetes.io/hostname: iscsinode1
qinqon commented 2 years ago

Also note that we are working in a host ip pool proposal that may be of help for you:

qinqon commented 2 years ago

We can go a step beyond and think on a scenario with hundred of nodes with static routes each, the NNCP should just define the specific parts of each nodes and a Template will have to expand per node into a NNCE.