Unable to set DNS on secondary interface

[mavynet]

Hello,

While trying to deploy a Sylva cluster using LeapMicro we are facing an issue setting the DNS server. The baremetal host has multiple interfaces and we use ens3 as provisioning interface and ens4 as primary. the provided configuration is

localhost:~ # jq . /tmp/nmc/desired/management-cluster-server2.yaml
{
  "dns-resolver": {
    "config": {
      "search": [
        "sylva"
      ],
      "server": [
        "192.168.130.1"
      ]
    }
  },
  "interfaces": [
    {
      "ipv4": {
        "address": [
          {
            "ip": "192.168.133.102",
            "prefix-length": 24
          }
        ],
        "auto-dns": true,
        "dhcp": false,
        "enabled": true
      },
      "mac-address": "52:54:00:01:00:02",
      "mtu": 1450,
      "name": "ens3",
      "state": "up",
      "type": "ethernet"
    },
    {
      "ipv4": {
        "address": [
          {
            "ip": "192.168.130.102",
            "prefix-length": 24
          }
        ],
        "auto-dns": true,
        "dhcp": false,
        "enabled": true
      },
      "mac-address": "52:54:00:02:00:02",
      "mtu": 1450,
      "name": "ens4",
      "state": "up",
      "type": "ethernet"
    }
  ],
  "routes": {
    "config": [
      {
        "destination": "0.0.0.0/0",
        "next-hop-address": "192.168.130.1",
        "next-hop-interface": "ens4"
      }
    ]
  }
}

The nmc output sets the default gateway correctly on ens4, but the dns configuration is on ens3:

localhost:~ # cat /tmp/nmc/generated/management-cluster-server2/ens3.nmconnection
[connection]
autoconnect=true
autoconnect-slaves=-1
id=ens3
interface-name=ens3
type=802-3-ethernet
uuid=eb167dc3-77c6-5e5b-94a7-7ca659590917

[ipv4]
address0=192.168.133.102/24
dhcp-timeout=2147483647
dns=192.168.130.1
dns-priority=40
dns-search=sylva
method=manual

[ipv6]
dhcp-timeout=2147483647
method=disabled

[ethernet]
cloned-mac-address=52:54:00:01:00:02
mtu=1450
localhost:~ # cat /tmp/nmc/generated/management-cluster-server2/ens4.nmconnection
[connection]
autoconnect=true
autoconnect-slaves=-1
id=ens4
interface-name=ens4
type=802-3-ethernet
uuid=739065e7-6e8f-5205-923f-530d8fd646f9

[ipv4]
address0=192.168.130.102/24
dhcp-timeout=2147483647
method=manual
route0=0.0.0.0/0,192.168.130.1,0
route0_options=table=254

[ipv6]
dhcp-timeout=2147483647
method=disabled

[ethernet]
cloned-mac-address=52:54:00:02:00:02
mtu=1450

which leads to not having any dns set

localhost:~ # nmcli device show
GENERAL.DEVICE:                         lo
GENERAL.TYPE:                           loopback
GENERAL.HWADDR:                         00:00:00:00:00:00
GENERAL.MTU:                            65536
GENERAL.STATE:                          100 (connected (externally))
GENERAL.CONNECTION:                     lo
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/1
IP4.ADDRESS[1]:                         127.0.0.1/8
IP4.GATEWAY:                            --
IP6.ADDRESS[1]:                         ::1/128
IP6.GATEWAY:                            --

GENERAL.DEVICE:                         ens3
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         52:54:00:01:00:02
GENERAL.MTU:                            1450
GENERAL.STATE:                          100 (connected (externally))
GENERAL.CONNECTION:                     ens3
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/2
WIRED-PROPERTIES.CARRIER:               on
IP4.ADDRESS[1]:                         192.168.133.102/24
IP4.GATEWAY:                            --
IP4.ROUTE[1]:                           dst = 192.168.133.0/24, nh = 0.0.0.0, mt = 100
IP6.GATEWAY:                            --

GENERAL.DEVICE:                         ens4
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         52:54:00:02:00:02
GENERAL.MTU:                            1450
GENERAL.STATE:                          100 (connected (externally))
GENERAL.CONNECTION:                     ens4
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/3
WIRED-PROPERTIES.CARRIER:               on
IP4.ADDRESS[1]:                         192.168.130.102/24
IP4.GATEWAY:                            192.168.130.1
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 192.168.130.1, mt = 0
IP4.ROUTE[2]:                           dst = 192.168.130.0/24, nh = 0.0.0.0, mt = 101
IP6.GATEWAY:                            --

Tried with and without auto-dns: true and the results are the same

[hardys]

I noticed https://github.com/nmstate/nmstate/issues/2538 which may be related - AFAICS there's no option for interface specific DNS via nmstate currently even though NM supports it.

Looking at the nmstate code I think it just sorts the list of interfaces and applies the DNS configuration to the first one?

[hardys]

I also notice that the ens3 state is 100 (connected (externally)) - I could be mistaken but my reading of the nmstate code is that means the interface will be ignored for DNS configuration.

I think the externally managed flag means that there's an existing configuration for this nic, for example a default connection which ref #63 aren't purged until after the static configuration takes place.

[atanasdinov]

Hi, @mavynet. I took a look into the source and @hardys' suggestions but I don't think those are related.

Could you please try appending your routes configuration to also include the ens3 interface? I got the necessary route0 entries in the ens3 connection file by using:

"routes": {
    "config": [
      {
        "destination": "0.0.0.0/0",
        "next-hop-address": "192.168.130.1",
        "next-hop-interface": "ens4"
      },
      {
        "destination": "0.0.0.0/0",
        "next-hop-address": "192.168.130.1",
        "next-hop-interface": "ens3"
      }
    ]
  }