default gw failing at boot, not waiting interface up

[aderumier]

Hi, I have a bug I never seen before, on debian buster, with kernel 5.0 or kernel 5.3

It seem than ifupdown2, at boot, try to setup the default gw before the interface is up. If a do a "ifreload -a" after startup, the gateway is correctly added.

Oct 25 13:21:45  systemd[1]: Starting ifupdown2 networking initialization...
Oct 25 13:21:45  networking[3050]: networking: Configuring network interfaces
Oct 25 13:21:48  networking[3050]: error: bond0.94: cmd '/bin/ip route add default via 10.3.94.1 proto kernel dev bond0.94 onlink' failed: returned 2 (Error: Nexthop device is not up.
Oct 25 13:21:48  networking[3050]: )

auto eth4
iface eth4
        mtu 9000

auto eth5
iface eth5
        mtu 9000

auto bond0
iface bond0
        bond-mode 4
        bond-miimon 100
        bond-lacp_rate fast
        bond-xmit-hash-policy layer3+4
        bond-slaves eth4 eth5
        mtu 9000

auto bond0.94
iface bond0.94
        address 10.3.94.107
        netmask 255.255.255.0
        gateway 10.3.94.1
        mtu 1500

[aderumier]

Here the timing logs:

networking service log

Oct 25 14:01:56  systemd[1]: Starting ifupdown2 networking initialization...
Oct 25 14:01:56  networking[3044]: networking: Configuring network interfaces
Oct 25 14:01:58  networking[3044]: error: bond0.94: cmd '/bin/ip route add default via 10.3.94.1 proto kernel dev bond0.94 onlink' failed: returned 2 (Error: Nexthop device is not up.
Oct 25 14:01:58 m6kv10 networking[3044]: )
Oct 25 14:01:58  systemd[1]: Started ifupdown2 networking initialization.

dmesg -T

[Fri Oct 25 14:01:57 2019] mlx5_core 0000:82:00.1 eth5: Link up
[Fri Oct 25 14:01:57 2019] bond0: Enslaving eth5 as a backup interface with an up link
[Fri Oct 25 14:01:57 2019] mlx5_core 0000:82:00.0 eth4: Link up
[Fri Oct 25 14:01:57 2019] bond0: Enslaving eth4 as a backup interface with an up link
[Fri Oct 25 14:01:58 2019] 8021q: 802.1Q VLAN Support v1.8
[Fri Oct 25 14:01:58 2019] 8021q: adding VLAN 0 to HW filter on device eth4
[Fri Oct 25 14:01:58 2019] 8021q: adding VLAN 0 to HW filter on device eth5
[Fri Oct 25 14:01:58 2019] vmbr1: port 1(bond0) entered blocking state
[Fri Oct 25 14:01:58 2019] vmbr1: port 1(bond0) entered disabled state
[Fri Oct 25 14:01:58 2019] device bond0 entered promiscuous mode
[Fri Oct 25 14:01:58 2019] device eth5 entered promiscuous mode
[Fri Oct 25 14:01:58 2019] device eth4 entered promiscuous mode
[Fri Oct 25 14:01:58 2019] mlx5_core 0000:82:00.1 eth5: S-tagged traffic will be dropped while C-tag vlan stripping is enabled
[Fri Oct 25 14:01:58 2019] 8021q: adding VLAN 0 to HW filter on device bond0
[Fri Oct 25 14:01:58 2019] IPv6: ADDRCONF(NETDEV_CHANGE): bond0.94: link becomes ready

[aderumier]

note that I have the same problem with

"up /bin/ip route add default via 10.3.94.1 proto kernel dev bond0.94 onlink" instead "gateway ...."

same with post-up

This is only at boot.

If I do a ifupdown bond0.94, then ifup bond0.94, it's working.

I wonder if it could be a kernel or driver bug ? (mellanox connect-x5 nic)

[aderumier]

I can produce it too, with

ifdown bond0.94 ifdown bond0 ifup -a

(seem that bond0.94 is ifupped before bond0)

debug: args = Namespace(CLASS=None, all=True, debug=True, excludepats=None, force=False, iflist=[], interfacesfile=None, interfacesfileformat='native', noact=False, noaddons=False, nocache=False, perfmode=False, printdependency=None, quie
t=False, skipupperifaces=False, syntaxcheck=False, syslog=False, type=None, verbose=False, version=None, withdepends=False)
debug: creating ifupdown object ..
debug: {'ifreload_down_changed': '0', 'multiple_vlan_aware_bridge_support': '1', 'ifaceobj_squash': '0', 'ifquery_check_error_str': 'fail', 'state_dir': '/run/network/', 'delay_admin_state_change': '0', 'addon_syntax_check': '1', 'ifquery
_check_unknown_str': '', 'addr_config_squash': '0', 'adjust_logical_dev_mtu': '1', 'default_interfaces_configfile': '/etc/network/interfaces', 'template_lookuppath': '/etc/network/ifupdown2/templates', 'link_master_slave': '1', 'template_
enable': '1', 'ifquery_ifacename_expand_range': '0', 'template_engine': 'mako', 'addon_scripts_support': '1', 'ifquery_check_success_str': 'pass', 'addon_python_modules_support': '1', 'disable_cli_interfacesfile': '0'}
info: loading builtin modules from ['/usr/share/ifupdown2/addons']
info: module ppp not loaded (module init failed: no /usr/bin/pon found)

info: module batman_adv not loaded (module init failed: no /usr/sbin/batctl found)

debug: bridge: using reserved vlan range (0, 0)
debug: bridge: init: warn_on_untagged_bridge_absence=False
debug: bridge: init: vxlan_bridge_default_igmp_snooping=None
debug: bridge: init: arp_nd_suppress_only_on_vxlan=False
info: executing /sbin/sysctl net.bridge.bridge-allow-multiple-vlans
debug: bridge: init: multiple vlans allowed True
info: module mstpctl not loaded (module init failed: no /sbin/mstpctl found)

info: executing /bin/ip rule show
info: executing /bin/ip -6 rule show
info: module ethtool not loaded (module init failed: /sbin/ethtool: not found)

info: address: using default mtu 1500
info: module ethtool not loaded (module init failed: /sbin/ethtool: not found)

info: module ppp not loaded (module init failed: no /usr/bin/pon found)

info: module mstpctl not loaded (module init failed: no /sbin/mstpctl found)

info: module batman_adv not loaded (module init failed: no /usr/sbin/batctl found)

info: looking for user scripts under /etc/network
info: loading scripts under /etc/network/if-pre-up.d ...
info: loading scripts under /etc/network/if-up.d ...
info: loading scripts under /etc/network/if-post-up.d ...
info: loading scripts under /etc/network/if-pre-down.d ...
info: loading scripts under /etc/network/if-down.d ...
info: loading scripts under /etc/network/if-post-down.d ...
info: 'link_master_slave' is set. slave admin state changes will be delayed till the masters admin state change.
info: processing interfaces file /etc/network/interfaces
debug: bond0: evaluating port expr '['eth4', 'eth5']'
debug: vmbr1: evaluating port expr '['bond0']'
debug: vmbr1v5: evaluating port expr '['bond0.5']'
debug: populate_dependency_info: deleting blacklisted interface bond0.1000
debug: scheduling '['pre-up', 'up', 'post-up']' for ['lo', 'eth4', 'eth5', 'bond0', 'bond0.94', 'bond0.5', 'vmbr1', 'vmbr1v5']
debug: dependency graph {
        lo : []
        eth4 : []
        eth5 : []
        bond0 : ['eth5', 'eth4']
        bond0.94 : ['bond0']
        bond0.5 : ['bond0']
        vmbr1 : ['bond0']
        vmbr1v5 : ['bond0.5']
}
debug: graph roots (interfaces that dont have dependents): ['lo', 'bond0.94', 'vmbr1', 'vmbr1v5']
info: lo: running ops ...
debug: lo: pre-up : running module xfrm
debug: lo: pre-up : running module link
debug: lo: pre-up : running module bond
debug: lo: pre-up : running module vlan
debug: lo: pre-up : running module vxlan
debug: lo: pre-up : running module usercmds
debug: lo: pre-up : running module bridge
info: netlink: ip link show
debug: reading '/sys/class/net/bonding_masters'
info: netlink: ip addr show
info: executing /bin/ip addr help
info: address metric support: OK
debug: lo: pre-up : running module bridgevlan
debug: lo: pre-up : running module tunnel
debug: lo: pre-up : running module vrf
debug: _cache_get(['lo', 'master']) : ['master']
info: lo: netlink: ip link set dev lo up
debug: lo: up : running module dhcp
debug: lo: up : running module address
debug: reading '/sys/class/net/lo/ifalias'
info: reading '/proc/sys/net/mpls/conf/lo/input'
info: executing /sbin/sysctl net.mpls.conf.lo.input=0
debug: lo: ipv6 addrgen already on
debug: lo: up : running module addressvirtual
debug: lo: up : running module usercmds
debug: lo: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: lo: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: lo: post-up : running module usercmds
debug: lo: statemanager sync state pre-up
debug: bond0.94: found dependents ['bond0']
debug: bond0: found dependents ['eth5', 'eth4']
info: eth5: running ops ...
debug: eth5: pre-up : running module xfrm
debug: eth5: pre-up : running module link
debug: eth5: pre-up : running module bond
debug: eth5: pre-up : running module vlan
debug: eth5: pre-up : running module vxlan
debug: eth5: pre-up : running module usercmds
debug: eth5: pre-up : running module bridge
debug: eth5: pre-up : running module bridgevlan
debug: eth5: pre-up : running module tunnel
debug: eth5: pre-up : running module vrf
debug: _cache_get(['eth5', 'master']) : ['master']
debug: eth5: up : running module dhcp
debug: eth5: up : running module address
debug: reading '/sys/class/net/eth5/ifalias'
info: reading '/proc/sys/net/mpls/conf/eth5/input'
info: executing /sbin/sysctl net.mpls.conf.eth5.input=0
debug: eth5: ipv6 addrgen already on
debug: _cache_get(['eth5', 'addrs']) : ['addrs']
debug: reading '/sys/class/net/eth5/mtu'
debug: eth5: up : running module addressvirtual
debug: eth5: up : running module usercmds
debug: eth5: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: eth5: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: eth5: post-up : running module usercmds
debug: eth5: statemanager sync state pre-up
info: eth4: running ops ...
debug: eth4: pre-up : running module xfrm
debug: eth4: pre-up : running module link
debug: eth4: pre-up : running module bond
debug: eth4: pre-up : running module vlan
debug: eth4: pre-up : running module vxlan
debug: eth4: pre-up : running module usercmds
debug: eth4: pre-up : running module bridge
debug: eth4: pre-up : running module bridgevlan
debug: eth4: pre-up : running module tunnel
debug: eth4: pre-up : running module vrf
debug: _cache_get(['eth4', 'master']) : ['master']
debug: eth4: up : running module dhcp
debug: eth4: up : running module address
debug: reading '/sys/class/net/eth4/ifalias'
info: reading '/proc/sys/net/mpls/conf/eth4/input'
info: executing /sbin/sysctl net.mpls.conf.eth4.input=0
debug: eth4: ipv6 addrgen already on
debug: _cache_get(['eth4', 'addrs']) : ['addrs']
debug: reading '/sys/class/net/eth4/mtu'
debug: eth4: up : running module addressvirtual
debug: eth4: up : running module usercmds
debug: eth4: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: eth4: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: eth4: post-up : running module usercmds
debug: eth4: statemanager sync state pre-up
info: bond0: running ops ...
debug: bond0: pre-up : running module xfrm
debug: bond0: pre-up : running module link
debug: bond0: pre-up : running module bond
info: bond0: set bond-mode 4
info: bond0: set bond-xmit-hash-policy layer3+4
info: bond0: set bond-miimon 100
info: bond0: set bond-lacp-rate fast
debug: _cache_get(['bond0', 'linkinfo', 18]) : ['bond0']
info: bond0: netlink: ip link add bond0 type bond with attributes
debug: bond0: ifla_info_data OrderedDict([(1, 4), (14, 1), (3, 100), (21, 1)])
debug: _cache_get(['bond0', 'linkinfo', 'slaves']) : ['bond0']
debug: reading '/sys/class/net/bond0/bonding/slaves'
debug: reading '/sys/class/net/eth5/flags'
info: eth5: netlink: ip link set dev eth5 master bond0 
info: eth5: netlink: ip link set dev eth5 up
debug: reading '/sys/class/net/eth4/flags'
info: eth4: netlink: ip link set dev eth4 master bond0 
info: eth4: netlink: ip link set dev eth4 up
debug: bond0: pre-up : running module vlan
debug: bond0: pre-up : running module vxlan
debug: bond0: pre-up : running module usercmds
debug: bond0: pre-up : running module bridge
debug: reading '/sys/class/net/vmbr1/bridge/vlan_filtering'
info: bond0: not enslaved to bridge vmbr1: ignored for now
info: vmbr1: applying bridge port configuration: []
debug: bond0: pre-up : running module bridgevlan
debug: bond0: pre-up : running module tunnel
debug: bond0: pre-up : running module vrf
debug: _cache_get(['bond0', 'master']) : ['bond0']
debug: bond0: up : running module dhcp
debug: bond0: up : running module address
debug: reading '/sys/class/net/bond0/ifalias'
info: reading '/proc/sys/net/mpls/conf/bond0/input'
info: executing /sbin/sysctl net.mpls.conf.bond0.input=0
debug: _cache_get(['bond0', 'af_spec', 10]) : ['bond0']
debug: bond0: ipv6 addrgen probably not supported or disabled on this device
debug: _cache_get(['bond0', 'addrs']) : ['bond0']
debug: _cache_get(['bond0', 'mtu']) : ['bond0']
debug: reading '/sys/class/net/bond0/mtu'
info: executing /bin/ip -force -batch - [link set dev bond0 mtu 9000
]
debug: bond0: up : running module addressvirtual
debug: bond0: up : running module usercmds
debug: bond0: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: bond0: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: bond0: post-up : running module usercmds
debug: bond0: statemanager sync state pre-up
info: bond0.94: running ops ...
debug: bond0.94: pre-up : running module xfrm
debug: bond0.94: pre-up : running module link
debug: bond0.94: pre-up : running module bond
debug: bond0.94: pre-up : running module vlan
debug: _cache_get(['bond0.94', 'linkinfo', 'vlan_protocol']) : ['bond0.94']
info: bond0.94: netlink: ip link add link bond0 name bond0.94 type vlan id 94 protocol 802.1q
debug: bond0.94: pre-up : running module vxlan
debug: bond0.94: pre-up : running module usercmds
debug: bond0.94: pre-up : running module bridge
debug: bond0.94: pre-up : running module bridgevlan
debug: bond0.94: pre-up : running module tunnel
debug: bond0.94: pre-up : running module vrf
debug: _cache_get(['bond0.94', 'master']) : ['bond0.94']
info: bond0.94: netlink: ip link set dev bond0.94 up
debug: bond0.94: up : running module dhcp
debug: bond0.94: up : running module address
debug: reading '/sys/class/net/bond0.94/ifalias'
info: reading '/proc/sys/net/mpls/conf/bond0.94/input'
info: executing /sbin/sysctl net.mpls.conf.bond0/94.input=0
debug: _cache_get(['bond0.94', 'af_spec', 10]) : ['bond0.94']
debug: bond0.94: ipv6 addrgen probably not supported or disabled on this device
debug: _cache_get(['bond0.94', 'addrs']) : ['bond0.94']
debug: reading '/sys/class/net/bond0/mtu'
debug: _cache_get(['bond0.94', 'mtu']) : ['bond0.94']
debug: reading '/sys/class/net/bond0.94/mtu'
info: executing /bin/ip -force -batch - [addr add 10.3.94.107/24 dev bond0.94
link set dev bond0.94 mtu 1500
]
info: executing /bin/ip route add default via 10.3.94.1 proto kernel dev bond0.94 onlink
  File "/usr/sbin/ifup", line 203, in <module>
    sys.exit(main())
  File "/usr/sbin/ifup", line 193, in main
    return ifupdown2_standalone()
  File "/usr/sbin/ifup", line 185, in ifupdown2_standalone
    return ifupdown2.main()
  File "/usr/share/ifupdown2/ifupdown/main.py", line 75, in main
    self.handlers.get(self.op)(self.args)
  File "/usr/share/ifupdown2/ifupdown/main.py", line 177, in run_up
    skipupperifaces=args.skipupperifaces)
  File "/usr/share/ifupdown2/ifupdown/ifupdownmain.py", line 1689, in up
    else False)
  File "/usr/share/ifupdown2/ifupdown/ifupdownmain.py", line 1448, in _sched_ifaces
    sort=True if (sort or ifupdownflags.flags.CLASS) else False)
  File "/usr/share/ifupdown2/ifupdown/scheduler.py", line 579, in sched_ifaces
    followdependents=followdependents)
  File "/usr/share/ifupdown2/ifupdown/scheduler.py", line 308, in run_iface_list
    order, followdependents)
  File "/usr/share/ifupdown2/ifupdown/scheduler.py", line 297, in run_iface_graph
    cls.run_iface_list_ops(ifupdownobj, ifaceobjs, ops)
  File "/usr/share/ifupdown2/ifupdown/scheduler.py", line 181, in run_iface_list_ops
    '0') == '1' else None)
  File "/usr/share/ifupdown2/ifupdown/scheduler.py", line 102, in run_iface_op
    ifaceobj_getfunc=ifupdownobj.get_ifaceobjs)
  File "/usr/share/ifupdown2/addons/address.py", line 1219, in run
    ifaceobj_getfunc=ifaceobj_getfunc)
  File "/usr/share/ifupdown2/addons/address.py", line 878, in _up
    self._add_delete_gateway(ifaceobj, gateways, prev_gw)
  File "/usr/share/ifupdown2/addons/address.py", line 516, in _add_delete_gateway
    self.log_error('%s: %s' % (ifaceobj.name, str(e)))
  File "/usr/share/ifupdown2/ifupdownaddons/modulebase.py", line 107, in log_error
    traceback.print_stack()
Traceback (most recent call last):
  File "/usr/share/ifupdown2/addons/address.py", line 514, in _add_delete_gateway
    self.ipcmd.route_add_gateway(ifaceobj.name, add_gw, vrf, metric, onlink=self.l3_intf_default_gateway_set_onlink)
  File "/usr/share/ifupdown2/ifupdownaddons/LinkUtils.py", line 1156, in route_add_gateway
    utils.exec_command(cmd)
  File "/usr/share/ifupdown2/ifupdown/utils.py", line 435, in exec_command
    stderr=stderr)
  File "/usr/share/ifupdown2/ifupdown/utils.py", line 412, in _execute_subprocess
    stdin))
Exception: cmd '/bin/ip route add default via 10.3.94.1 proto kernel dev bond0.94 onlink' failed: returned 2 (Error: Nexthop device is not up.
)
error: bond0.94: cmd '/bin/ip route add default via 10.3.94.1 proto kernel dev bond0.94 onlink' failed: returned 2 (Error: Nexthop device is not up.
)
debug: bond0.94: up : running module addressvirtual
debug: bond0.94: up : running module usercmds
debug: bond0.94: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: bond0.94: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: bond0.94: post-up : running module usercmds
debug: bond0.94: statemanager sync state pre-up
debug: vmbr1: found dependents ['bond0']
debug: bond0: already processed
info: vmbr1: running ops ...
debug: vmbr1: pre-up : running module xfrm
debug: vmbr1: pre-up : running module link
debug: vmbr1: pre-up : running module bond
debug: vmbr1: pre-up : running module vlan
debug: vmbr1: pre-up : running module vxlan
debug: vmbr1: pre-up : running module usercmds
debug: vmbr1: pre-up : running module bridge
info: vmbr1: bridge already exists
info: vmbr1: apply bridge settings
info: vmbr1: reset bridge-hashel to default: 4
info: reading '/sys/class/net/vmbr1/bridge/stp_state'
info: vmbr1: netlink: ip link set vmbr1 type bridge with attributes
debug: vmbr1: ifla_info_data {26: 4}
debug: vmbr1: evaluating port expr '['bond0']'
debug: _cache_get(['bond0', 'hwaddress']) : ['bond0']
debug: reading '/sys/class/net/bond0/address'
info: writing '1' to file /proc/sys/net/ipv6/conf/bond0/disable_ipv6
info: executing /bin/ip -force -batch - [link set dev bond0 master vmbr1
addr flush dev bond0
]
info: vmbr1: applying bridge port configuration: ['bond0']
info: vmbr1: applying bridge configuration specific to ports
info: vmbr1: processing bridge config for port bond0
info: bond0: netlink: ip link set dev bond0 up
debug: vmbr1: _get_bridge_mac returned (None, None)
debug: vmbr1: pre-up : running module bridgevlan
debug: vmbr1: pre-up : running module tunnel
debug: vmbr1: pre-up : running module vrf
debug: _cache_get(['vmbr1', 'master']) : ['master']
info: vmbr1: netlink: ip link set dev vmbr1 up
debug: vmbr1: up : running module dhcp
debug: vmbr1: up : running module address
debug: reading '/sys/class/net/vmbr1/ifalias'
info: reading '/proc/sys/net/mpls/conf/vmbr1/input'
info: executing /sbin/sysctl net.mpls.conf.vmbr1.input=0
info: reading '/proc/sys/net/ipv4/conf/vmbr1/forwarding'
info: reading '/proc/sys/net/ipv6/conf/vmbr1/forwarding'
debug: vmbr1: ipv6 addrgen already on
info: vmbr1: bridge inherits mtu from its ports. There is no need to assign mtu on a bridge
debug: reading '/sys/class/net/vmbr1/mtu'
debug: reading '/sys/class/net/vmbr1/bridge/vlan_filtering'
debug: vmbr1: up : running module addressvirtual
debug: vmbr1: up : running module usercmds
debug: vmbr1: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: vmbr1: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: vmbr1: post-up : running module usercmds
debug: vmbr1: statemanager sync state pre-up
debug: vmbr1v5: found dependents ['bond0.5']
debug: bond0.5: found dependents ['bond0']
debug: bond0: already processed
info: bond0.5: running ops ...
debug: bond0.5: pre-up : running module xfrm
debug: bond0.5: pre-up : running module link
debug: bond0.5: pre-up : running module bond
debug: bond0.5: pre-up : running module vlan
debug: _cache_get(['bond0.5', 'linkinfo', 'vlan_protocol']) : ['bond0.5']
info: bond0.5: netlink: ip link add link bond0 name bond0.5 type vlan id 5 protocol 802.1q
debug: bond0.5: pre-up : running module vxlan
debug: bond0.5: pre-up : running module usercmds
debug: bond0.5: pre-up : running module bridge
debug: reading '/sys/class/net/vmbr1v5/bridge/vlan_filtering'
info: bond0.5: not enslaved to bridge vmbr1v5: ignored for now
info: vmbr1v5: applying bridge port configuration: []
debug: bond0.5: pre-up : running module bridgevlan
debug: bond0.5: pre-up : running module tunnel
debug: bond0.5: pre-up : running module vrf
debug: _cache_get(['bond0.5', 'master']) : ['bond0.5']
debug: bond0.5: up : running module dhcp
debug: bond0.5: up : running module address
debug: reading '/sys/class/net/bond0.5/ifalias'
info: reading '/proc/sys/net/mpls/conf/bond0.5/input'
info: executing /sbin/sysctl net.mpls.conf.bond0/5.input=0
debug: _cache_get(['bond0.5', 'af_spec', 10]) : ['bond0.5']
debug: bond0.5: ipv6 addrgen probably not supported or disabled on this device
debug: _cache_get(['bond0.5', 'addrs']) : ['bond0.5']
debug: reading '/sys/class/net/bond0/mtu'
debug: _cache_get(['bond0.5', 'mtu']) : ['bond0.5']
debug: reading '/sys/class/net/bond0.5/mtu'
debug: bond0.5: up : running module addressvirtual
debug: bond0.5: up : running module usercmds
debug: bond0.5: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: bond0.5: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: bond0.5: post-up : running module usercmds
debug: bond0.5: statemanager sync state pre-up
info: vmbr1v5: running ops ...
debug: vmbr1v5: pre-up : running module xfrm
debug: vmbr1v5: pre-up : running module link
debug: vmbr1v5: pre-up : running module bond
debug: vmbr1v5: pre-up : running module vlan
debug: vmbr1v5: pre-up : running module vxlan
debug: vmbr1v5: pre-up : running module usercmds
debug: vmbr1v5: pre-up : running module bridge
info: vmbr1v5: bridge already exists
info: vmbr1v5: apply bridge settings
info: vmbr1v5: reset bridge-hashel to default: 4
info: reading '/sys/class/net/vmbr1v5/bridge/stp_state'
info: vmbr1v5: netlink: ip link set vmbr1v5 type bridge with attributes
debug: vmbr1v5: ifla_info_data {26: 4}
debug: vmbr1v5: evaluating port expr '['bond0.5']'
debug: _cache_get(['bond0.5', 'hwaddress']) : ['bond0.5']
debug: reading '/sys/class/net/bond0.5/address'
info: writing '1' to file /proc/sys/net/ipv6/conf/bond0.5/disable_ipv6
info: executing /bin/ip -force -batch - [link set dev bond0.5 master vmbr1v5
addr flush dev bond0.5
]
info: vmbr1v5: applying bridge port configuration: ['bond0.5']
info: vmbr1v5: applying bridge configuration specific to ports
info: vmbr1v5: processing bridge config for port bond0.5
info: bond0.5: netlink: ip link set dev bond0.5 up
debug: vmbr1v5: _get_bridge_mac returned (None, None)
info: executing /sbin/bridge fdb replace 00:00:00:00:00:00 dev vmbr1v5  self 
warning: vmbr1v5: setting bridge mac address: cmd '/sbin/bridge fdb replace 00:00:00:00:00:00 dev vmbr1v5 self' failed: returned 255 (RTNETLINK answers: Invalid argument
)
debug: vmbr1v5: pre-up : running module bridgevlan
debug: vmbr1v5: pre-up : running module tunnel
debug: vmbr1v5: pre-up : running module vrf
debug: _cache_get(['vmbr1v5', 'master']) : ['master']
info: vmbr1v5: netlink: ip link set dev vmbr1v5 up
debug: vmbr1v5: up : running module dhcp
debug: vmbr1v5: up : running module address
debug: reading '/sys/class/net/vmbr1v5/ifalias'
info: reading '/proc/sys/net/mpls/conf/vmbr1v5/input'
info: executing /sbin/sysctl net.mpls.conf.vmbr1v5.input=0
info: reading '/proc/sys/net/ipv4/conf/vmbr1v5/forwarding'
info: reading '/proc/sys/net/ipv6/conf/vmbr1v5/forwarding'
debug: vmbr1v5: ipv6 addrgen already on
info: vmbr1v5: bridge inherits mtu from its ports. There is no need to assign mtu on a bridge
debug: reading '/sys/class/net/vmbr1v5/mtu'
debug: reading '/sys/class/net/vmbr1v5/bridge/vlan_filtering'
info: writing '0' to file /proc/sys/net/ipv4/conf/vmbr1v5/arp_accept
debug: vmbr1v5: up : running module addressvirtual
debug: vmbr1v5: up : running module usercmds
debug: vmbr1v5: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: vmbr1v5: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: vmbr1v5: post-up : running module usercmds
debug: vmbr1v5: statemanager sync state pre-up
debug: saving state ..

[aderumier]

I have found something:

It's not working with

auto bond0
iface bond0
        bond-mode 4
        bond-miimon 100
        bond-lacp_rate fast
        bond-xmit-hash-policy layer3+4
        bond-slaves eth4 eth5
        mtu 9000

auto bond0.94
iface bond0.94
        address 10.3.94.107
        netmask 255.255.255.0
        gateway 10.3.94.1
        mtu 1500

auto vmbr1
iface vmbr1
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        mtu 9000

but working with

auto bond0
iface bond0
        bond-mode 4
        bond-miimon 100
        bond-lacp_rate fast
        bond-xmit-hash-policy layer3+4
        bond-slaves eth4 eth5
        mtu 9000

auto bond0.94
iface bond0.94
        address 10.3.94.107
        netmask 255.255.255.0
        gateway 10.3.94.1
        mtu 1500

not working log with vmbr1 bridge:

debug: args = Namespace(CLASS=None, all=True, debug=True, excludepats=None, force=False, iflist=[], interfacesfile=None, interfacesfileformat='native', noact=False, noaddons=False, nocache=False, perfmode=False, printdependency=None, quie
t=False, skipupperifaces=False, syntaxcheck=False, syslog=False, type=None, verbose=False, version=None, withdepends=False)
debug: creating ifupdown object ..
debug: {'ifreload_down_changed': '0', 'multiple_vlan_aware_bridge_support': '1', 'ifaceobj_squash': '0', 'ifquery_check_error_str': 'fail', 'state_dir': '/run/network/', 'delay_admin_state_change': '0', 'addon_syntax_check': '1', 'ifquery
_check_unknown_str': '', 'addr_config_squash': '0', 'adjust_logical_dev_mtu': '1', 'default_interfaces_configfile': '/etc/network/interfaces', 'template_lookuppath': '/etc/network/ifupdown2/templates', 'link_master_slave': '1', 'template_
enable': '1', 'ifquery_ifacename_expand_range': '0', 'template_engine': 'mako', 'addon_scripts_support': '1', 'ifquery_check_success_str': 'pass', 'addon_python_modules_support': '1', 'disable_cli_interfacesfile': '0'}
info: loading builtin modules from ['/usr/share/ifupdown2/addons']
info: module ppp not loaded (module init failed: no /usr/bin/pon found)

info: module batman_adv not loaded (module init failed: no /usr/sbin/batctl found)

debug: bridge: using reserved vlan range (0, 0)
debug: bridge: init: warn_on_untagged_bridge_absence=False
debug: bridge: init: vxlan_bridge_default_igmp_snooping=None
debug: bridge: init: arp_nd_suppress_only_on_vxlan=False
info: executing /sbin/sysctl net.bridge.bridge-allow-multiple-vlans
debug: bridge: init: multiple vlans allowed True
info: module mstpctl not loaded (module init failed: no /sbin/mstpctl found)

info: executing /bin/ip rule show
info: executing /bin/ip -6 rule show
info: module ethtool not loaded (module init failed: /sbin/ethtool: not found)

info: address: using default mtu 1500
info: module ethtool not loaded (module init failed: /sbin/ethtool: not found)

info: module ppp not loaded (module init failed: no /usr/bin/pon found)

info: module mstpctl not loaded (module init failed: no /sbin/mstpctl found)

info: module batman_adv not loaded (module init failed: no /usr/sbin/batctl found)

info: looking for user scripts under /etc/network
info: loading scripts under /etc/network/if-pre-up.d ...
info: loading scripts under /etc/network/if-up.d ...
info: loading scripts under /etc/network/if-post-up.d ...
info: loading scripts under /etc/network/if-pre-down.d ...
info: loading scripts under /etc/network/if-down.d ...
info: loading scripts under /etc/network/if-post-down.d ...
info: 'link_master_slave' is set. slave admin state changes will be delayed till the masters admin state change.
info: processing interfaces file /etc/network/interfaces
debug: bond0: evaluating port expr '['eth4', 'eth5']'
debug: vmbr1: evaluating port expr '['bond0']'
debug: scheduling '['pre-up', 'up', 'post-up']' for ['lo', 'eth4', 'eth5', 'bond0', 'bond0.94', 'vmbr1']
debug: dependency graph {
        lo : []
        eth4 : []
        eth5 : []
        bond0 : ['eth5', 'eth4']
        bond0.94 : ['bond0']
        vmbr1 : ['bond0']
}
debug: graph roots (interfaces that dont have dependents): ['lo', 'bond0.94', 'vmbr1']
info: lo: running ops ...
debug: lo: pre-up : running module xfrm
debug: lo: pre-up : running module link
debug: lo: pre-up : running module bond
debug: lo: pre-up : running module vlan
debug: lo: pre-up : running module vxlan
debug: lo: pre-up : running module usercmds
debug: lo: pre-up : running module bridge
info: netlink: ip link show
debug: reading '/sys/class/net/bonding_masters'
info: netlink: ip addr show
info: executing /bin/ip addr help
info: address metric support: OK
debug: lo: pre-up : running module bridgevlan
debug: lo: pre-up : running module tunnel
debug: lo: pre-up : running module vrf
debug: _cache_get(['lo', 'master']) : ['master']
info: lo: netlink: ip link set dev lo up
debug: lo: up : running module dhcp
debug: lo: up : running module address
debug: reading '/sys/class/net/lo/ifalias'
info: reading '/proc/sys/net/mpls/conf/lo/input'
info: executing /sbin/sysctl net.mpls.conf.lo.input=0
debug: lo: ipv6 addrgen already on
debug: lo: up : running module addressvirtual
debug: lo: up : running module usercmds
debug: lo: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: lo: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: lo: post-up : running module usercmds
debug: lo: statemanager sync state pre-up
debug: bond0.94: found dependents ['bond0']
debug: bond0: found dependents ['eth5', 'eth4']
info: eth5: running ops ...
debug: eth5: pre-up : running module xfrm
debug: eth5: pre-up : running module link
debug: eth5: pre-up : running module bond
debug: eth5: pre-up : running module vlan
debug: eth5: pre-up : running module vxlan
debug: eth5: pre-up : running module usercmds
debug: eth5: pre-up : running module bridge
debug: eth5: pre-up : running module bridgevlan
debug: eth5: pre-up : running module tunnel
debug: eth5: pre-up : running module vrf
debug: _cache_get(['eth5', 'master']) : ['master']
debug: eth5: up : running module dhcp
debug: eth5: up : running module address
debug: reading '/sys/class/net/eth5/ifalias'
info: reading '/proc/sys/net/mpls/conf/eth5/input'
info: executing /sbin/sysctl net.mpls.conf.eth5.input=0
debug: eth5: ipv6 addrgen already on
debug: _cache_get(['eth5', 'addrs']) : ['addrs']
debug: reading '/sys/class/net/eth5/mtu'
info: executing /bin/ip -force -batch - [link set dev eth5 mtu 9000
]
debug: eth5: up : running module addressvirtual
debug: eth5: up : running module usercmds
debug: eth5: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: eth5: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: eth5: post-up : running module usercmds
debug: eth5: statemanager sync state pre-up
info: eth4: running ops ...
debug: eth4: pre-up : running module xfrm
debug: eth4: pre-up : running module link
debug: eth4: pre-up : running module bond
debug: eth4: pre-up : running module vlan
debug: eth4: pre-up : running module vxlan
debug: eth4: pre-up : running module usercmds
debug: eth4: pre-up : running module bridge
debug: eth4: pre-up : running module bridgevlan
debug: eth4: pre-up : running module tunnel
debug: eth4: pre-up : running module vrf
debug: _cache_get(['eth4', 'master']) : ['master']
debug: eth4: up : running module dhcp
debug: eth4: up : running module address
debug: reading '/sys/class/net/eth4/ifalias'
info: reading '/proc/sys/net/mpls/conf/eth4/input'
info: executing /sbin/sysctl net.mpls.conf.eth4.input=0
debug: eth4: ipv6 addrgen already on
debug: _cache_get(['eth4', 'addrs']) : ['addrs']
debug: reading '/sys/class/net/eth4/mtu'
info: executing /bin/ip -force -batch - [link set dev eth4 mtu 9000
]
debug: eth4: up : running module addressvirtual
debug: eth4: up : running module usercmds
debug: eth4: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: eth4: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: eth4: post-up : running module usercmds
debug: eth4: statemanager sync state pre-up
info: bond0: running ops ...
debug: bond0: pre-up : running module xfrm
debug: bond0: pre-up : running module link
debug: bond0: pre-up : running module bond
info: bond0: set bond-mode 4
info: bond0: set bond-xmit-hash-policy layer3+4
info: bond0: set bond-miimon 100
info: bond0: set bond-lacp-rate fast
debug: _cache_get(['bond0', 'linkinfo', 18]) : ['bond0']
info: bond0: netlink: ip link add bond0 type bond with attributes
debug: bond0: ifla_info_data OrderedDict([(1, 4), (14, 1), (3, 100), (21, 1)])
debug: _cache_get(['bond0', 'linkinfo', 'slaves']) : ['bond0']
debug: reading '/sys/class/net/bond0/bonding/slaves'
debug: reading '/sys/class/net/eth5/flags'
info: eth5: netlink: ip link set dev eth5 master bond0 
info: eth5: netlink: ip link set dev eth5 up
debug: reading '/sys/class/net/eth4/flags'
info: eth4: netlink: ip link set dev eth4 master bond0 
info: eth4: netlink: ip link set dev eth4 up
debug: bond0: pre-up : running module vlan
debug: bond0: pre-up : running module vxlan
debug: bond0: pre-up : running module usercmds
debug: bond0: pre-up : running module bridge
debug: bond0: pre-up : running module bridgevlan
debug: bond0: pre-up : running module tunnel
debug: bond0: pre-up : running module vrf
debug: _cache_get(['bond0', 'master']) : ['bond0']
debug: bond0: up : running module dhcp
debug: bond0: up : running module address
debug: reading '/sys/class/net/bond0/ifalias'
info: reading '/proc/sys/net/mpls/conf/bond0/input'
info: executing /sbin/sysctl net.mpls.conf.bond0.input=0
debug: _cache_get(['bond0', 'af_spec', 10]) : ['bond0']
debug: bond0: ipv6 addrgen probably not supported or disabled on this device
debug: _cache_get(['bond0', 'addrs']) : ['bond0']
debug: _cache_get(['bond0', 'mtu']) : ['bond0']
debug: reading '/sys/class/net/bond0/mtu'
info: executing /bin/ip -force -batch - [link set dev bond0 mtu 9000
]
debug: bond0: up : running module addressvirtual
debug: bond0: up : running module usercmds
debug: bond0: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: bond0: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: bond0: post-up : running module usercmds
debug: bond0: statemanager sync state pre-up
info: bond0.94: running ops ...
debug: bond0.94: pre-up : running module xfrm
debug: bond0.94: pre-up : running module link
debug: bond0.94: pre-up : running module bond
debug: bond0.94: pre-up : running module vlan
debug: _cache_get(['bond0.94', 'linkinfo', 'vlan_protocol']) : ['bond0.94']
info: bond0.94: netlink: ip link add link bond0 name bond0.94 type vlan id 94 protocol 802.1q
debug: bond0.94: pre-up : running module vxlan
debug: bond0.94: pre-up : running module usercmds
debug: bond0.94: pre-up : running module bridge
debug: bond0.94: pre-up : running module bridgevlan
debug: bond0.94: pre-up : running module tunnel
debug: bond0.94: pre-up : running module vrf
debug: _cache_get(['bond0.94', 'master']) : ['bond0.94']
info: bond0.94: netlink: ip link set dev bond0.94 up
debug: bond0.94: up : running module dhcp
debug: bond0.94: up : running module address
debug: reading '/sys/class/net/bond0.94/ifalias'
info: reading '/proc/sys/net/mpls/conf/bond0.94/input'
info: executing /sbin/sysctl net.mpls.conf.bond0/94.input=0
debug: _cache_get(['bond0.94', 'af_spec', 10]) : ['bond0.94']
debug: bond0.94: ipv6 addrgen probably not supported or disabled on this device
debug: _cache_get(['bond0.94', 'addrs']) : ['bond0.94']
debug: reading '/sys/class/net/bond0/mtu'
debug: _cache_get(['bond0.94', 'mtu']) : ['bond0.94']
debug: reading '/sys/class/net/bond0.94/mtu'
info: executing /bin/ip -force -batch - [addr add 10.3.94.107/24 dev bond0.94
link set dev bond0.94 mtu 1500
]
info: executing /bin/ip route add default via 10.3.94.1 proto kernel dev bond0.94 onlink
  File "/usr/sbin/ifup", line 203, in <module>
    sys.exit(main())
  File "/usr/sbin/ifup", line 193, in main
    return ifupdown2_standalone()
  File "/usr/sbin/ifup", line 185, in ifupdown2_standalone
    return ifupdown2.main()
  File "/usr/share/ifupdown2/ifupdown/main.py", line 75, in main
    self.handlers.get(self.op)(self.args)
  File "/usr/share/ifupdown2/ifupdown/main.py", line 177, in run_up
    skipupperifaces=args.skipupperifaces)
  File "/usr/share/ifupdown2/ifupdown/ifupdownmain.py", line 1689, in up
    else False)
  File "/usr/share/ifupdown2/ifupdown/ifupdownmain.py", line 1448, in _sched_ifaces
    sort=True if (sort or ifupdownflags.flags.CLASS) else False)
  File "/usr/share/ifupdown2/ifupdown/scheduler.py", line 579, in sched_ifaces
    followdependents=followdependents)
  File "/usr/share/ifupdown2/ifupdown/scheduler.py", line 308, in run_iface_list
    order, followdependents)
  File "/usr/share/ifupdown2/ifupdown/scheduler.py", line 297, in run_iface_graph
    cls.run_iface_list_ops(ifupdownobj, ifaceobjs, ops)
  File "/usr/share/ifupdown2/ifupdown/scheduler.py", line 181, in run_iface_list_ops
    '0') == '1' else None)
  File "/usr/share/ifupdown2/ifupdown/scheduler.py", line 102, in run_iface_op
    ifaceobj_getfunc=ifupdownobj.get_ifaceobjs)
  File "/usr/share/ifupdown2/addons/address.py", line 1219, in run
    ifaceobj_getfunc=ifaceobj_getfunc)
  File "/usr/share/ifupdown2/addons/address.py", line 878, in _up
    self._add_delete_gateway(ifaceobj, gateways, prev_gw)
  File "/usr/share/ifupdown2/addons/address.py", line 516, in _add_delete_gateway
    self.log_error('%s: %s' % (ifaceobj.name, str(e)))
  File "/usr/share/ifupdown2/ifupdownaddons/modulebase.py", line 107, in log_error
    traceback.print_stack()
Traceback (most recent call last):
  File "/usr/share/ifupdown2/addons/address.py", line 514, in _add_delete_gateway
    self.ipcmd.route_add_gateway(ifaceobj.name, add_gw, vrf, metric, onlink=self.l3_intf_default_gateway_set_onlink)
  File "/usr/share/ifupdown2/ifupdownaddons/LinkUtils.py", line 1156, in route_add_gateway
    utils.exec_command(cmd)
  File "/usr/share/ifupdown2/ifupdown/utils.py", line 435, in exec_command
    stderr=stderr)
  File "/usr/share/ifupdown2/ifupdown/utils.py", line 412, in _execute_subprocess
    stdin))
Exception: cmd '/bin/ip route add default via 10.3.94.1 proto kernel dev bond0.94 onlink' failed: returned 2 (Error: Nexthop device is not up.
)
error: bond0.94: cmd '/bin/ip route add default via 10.3.94.1 proto kernel dev bond0.94 onlink' failed: returned 2 (Error: Nexthop device is not up.
)
debug: bond0.94: up : running module addressvirtual
debug: bond0.94: up : running module usercmds
debug: bond0.94: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: bond0.94: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: bond0.94: post-up : running module usercmds
debug: bond0.94: statemanager sync state pre-up
debug: vmbr1: found dependents ['bond0']
debug: bond0: already processed
info: vmbr1: running ops ...
debug: vmbr1: pre-up : running module xfrm
debug: vmbr1: pre-up : running module link
debug: vmbr1: pre-up : running module bond
debug: vmbr1: pre-up : running module vlan
debug: vmbr1: pre-up : running module vxlan
debug: vmbr1: pre-up : running module usercmds
debug: vmbr1: pre-up : running module bridge
info: vmbr1: set bridge mtu 9000
info: vmbr1: netlink: ip link add vmbr1 type bridge
info: vmbr1: apply bridge settings
info: vmbr1: set bridge-fd 0
info: vmbr1: set bridge-mcsnoop yes
info: reading '/sys/class/net/vmbr1/bridge/stp_state'
info: vmbr1: netlink: ip link set vmbr1 type bridge with attributes
debug: vmbr1: ifla_info_data {1: 0, 23: True}
debug: vmbr1: evaluating port expr '['bond0']'
debug: _cache_get(['bond0', 'hwaddress']) : ['bond0']
debug: reading '/sys/class/net/bond0/address'
info: writing '1' to file /proc/sys/net/ipv6/conf/bond0/disable_ipv6
info: executing /bin/ip -force -batch - [link set dev bond0 master vmbr1
addr flush dev bond0
]
info: vmbr1: applying bridge port configuration: ['bond0']
info: vmbr1: applying bridge configuration specific to ports
info: vmbr1: processing bridge config for port bond0
info: bond0: netlink: ip link set dev bond0 up
debug: vmbr1: _get_bridge_mac returned (None, None)
debug: _cache_get(['vmbr1', 'hwaddress']) : ['vmbr1']
debug: reading '/sys/class/net/vmbr1/address'
debug: vmbr1: pre-up : running module bridgevlan
debug: vmbr1: pre-up : running module tunnel
debug: vmbr1: pre-up : running module vrf
debug: _cache_get(['vmbr1', 'master']) : ['vmbr1']
info: vmbr1: netlink: ip link set dev vmbr1 up
debug: vmbr1: up : running module dhcp
debug: vmbr1: up : running module address
debug: reading '/sys/class/net/vmbr1/ifalias'
info: reading '/proc/sys/net/mpls/conf/vmbr1/input'
info: executing /sbin/sysctl net.mpls.conf.vmbr1.input=0
info: reading '/proc/sys/net/ipv4/conf/vmbr1/forwarding'
info: writing '0
' to file /proc/sys/net/ipv4/conf/vmbr1/forwarding
info: reading '/proc/sys/net/ipv6/conf/vmbr1/forwarding'
debug: _cache_get(['vmbr1', 'af_spec', 10]) : ['vmbr1']
debug: vmbr1: ipv6 addrgen probably not supported or disabled on this device
debug: _cache_get(['vmbr1', 'addrs']) : ['vmbr1']
info: vmbr1: bridge inherits mtu from its ports. There is no need to assign mtu on a bridge
debug: _cache_get(['vmbr1', 'mtu']) : ['vmbr1']
debug: reading '/sys/class/net/vmbr1/mtu'
debug: reading '/sys/class/net/vmbr1/bridge/vlan_filtering'
debug: vmbr1: up : running module addressvirtual
debug: vmbr1: up : running module usercmds
debug: vmbr1: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: vmbr1: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: vmbr1: post-up : running module usercmds
debug: vmbr1: statemanager sync state pre-up
debug: saving state ..

working log without vmbr1 bridge:

debug: args = Namespace(CLASS=None, all=True, debug=True, excludepats=None, force=False, iflist=[], interfacesfile=None, interfacesfileformat='native', noact=False, noaddons=False, nocache=False, perfmode=False, printdependency=None, quie
t=False, skipupperifaces=False, syntaxcheck=False, syslog=False, type=None, verbose=False, version=None, withdepends=False)
debug: creating ifupdown object ..
debug: {'ifreload_down_changed': '0', 'multiple_vlan_aware_bridge_support': '1', 'ifaceobj_squash': '0', 'ifquery_check_error_str': 'fail', 'state_dir': '/run/network/', 'delay_admin_state_change': '0', 'addon_syntax_check': '1', 'ifquery
_check_unknown_str': '', 'addr_config_squash': '0', 'adjust_logical_dev_mtu': '1', 'default_interfaces_configfile': '/etc/network/interfaces', 'template_lookuppath': '/etc/network/ifupdown2/templates', 'link_master_slave': '1', 'template_
enable': '1', 'ifquery_ifacename_expand_range': '0', 'template_engine': 'mako', 'addon_scripts_support': '1', 'ifquery_check_success_str': 'pass', 'addon_python_modules_support': '1', 'disable_cli_interfacesfile': '0'}
info: loading builtin modules from ['/usr/share/ifupdown2/addons']
info: module ppp not loaded (module init failed: no /usr/bin/pon found)

info: module batman_adv not loaded (module init failed: no /usr/sbin/batctl found)

debug: bridge: using reserved vlan range (0, 0)
debug: bridge: init: warn_on_untagged_bridge_absence=False
debug: bridge: init: vxlan_bridge_default_igmp_snooping=None
debug: bridge: init: arp_nd_suppress_only_on_vxlan=False
info: executing /sbin/sysctl net.bridge.bridge-allow-multiple-vlans
debug: bridge: init: multiple vlans allowed True
info: module mstpctl not loaded (module init failed: no /sbin/mstpctl found)

info: executing /bin/ip rule show
info: executing /bin/ip -6 rule show
info: module ethtool not loaded (module init failed: /sbin/ethtool: not found)

info: address: using default mtu 1500
info: module ethtool not loaded (module init failed: /sbin/ethtool: not found)

info: module ppp not loaded (module init failed: no /usr/bin/pon found)

info: module mstpctl not loaded (module init failed: no /sbin/mstpctl found)

info: module batman_adv not loaded (module init failed: no /usr/sbin/batctl found)

info: looking for user scripts under /etc/network
info: loading scripts under /etc/network/if-pre-up.d ...
info: loading scripts under /etc/network/if-up.d ...
info: loading scripts under /etc/network/if-post-up.d ...
info: loading scripts under /etc/network/if-pre-down.d ...
info: loading scripts under /etc/network/if-down.d ...
info: loading scripts under /etc/network/if-post-down.d ...
info: 'link_master_slave' is set. slave admin state changes will be delayed till the masters admin state change.
info: processing interfaces file /etc/network/interfaces
debug: bond0: evaluating port expr '['eth4', 'eth5']'
debug: scheduling '['pre-up', 'up', 'post-up']' for ['lo', 'eth4', 'eth5', 'bond0', 'bond0.94']
debug: dependency graph {
        lo : []
        eth4 : []
        eth5 : []
        bond0 : ['eth5', 'eth4']
        bond0.94 : ['bond0']
}
debug: graph roots (interfaces that dont have dependents): ['lo', 'bond0.94']
info: lo: running ops ...
debug: lo: pre-up : running module xfrm
debug: lo: pre-up : running module link
debug: lo: pre-up : running module bond
debug: lo: pre-up : running module vlan
debug: lo: pre-up : running module vxlan
debug: lo: pre-up : running module usercmds
debug: lo: pre-up : running module bridge
info: netlink: ip link show
debug: reading '/sys/class/net/bonding_masters'
info: netlink: ip addr show
info: executing /bin/ip addr help
info: address metric support: OK
debug: lo: pre-up : running module bridgevlan
debug: lo: pre-up : running module tunnel
debug: lo: pre-up : running module vrf
debug: _cache_get(['lo', 'master']) : ['master']
info: lo: netlink: ip link set dev lo up
debug: lo: up : running module dhcp
debug: lo: up : running module address
debug: reading '/sys/class/net/lo/ifalias'
info: reading '/proc/sys/net/mpls/conf/lo/input'
info: executing /sbin/sysctl net.mpls.conf.lo.input=0
debug: lo: ipv6 addrgen already on
debug: lo: up : running module addressvirtual
debug: lo: up : running module usercmds
debug: lo: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: lo: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: lo: post-up : running module usercmds
debug: lo: statemanager sync state pre-up
debug: bond0.94: found dependents ['bond0']
debug: bond0: found dependents ['eth5', 'eth4']
info: eth5: running ops ...
debug: eth5: pre-up : running module xfrm
debug: eth5: pre-up : running module link
debug: eth5: pre-up : running module bond
debug: eth5: pre-up : running module vlan
debug: eth5: pre-up : running module vxlan
debug: eth5: pre-up : running module usercmds
debug: eth5: pre-up : running module bridge
debug: eth5: pre-up : running module bridgevlan
debug: eth5: pre-up : running module tunnel
debug: eth5: pre-up : running module vrf
debug: _cache_get(['eth5', 'master']) : ['master']
debug: eth5: up : running module dhcp
debug: eth5: up : running module address
debug: reading '/sys/class/net/eth5/ifalias'
info: executing /sbin/sysctl net.mpls.conf.eth5.input=0
debug: eth5: ipv6 addrgen already on
debug: _cache_get(['eth5', 'addrs']) : ['addrs']
debug: reading '/sys/class/net/eth5/mtu'
info: executing /bin/ip -force -batch - [link set dev eth5 mtu 9000
]
debug: eth5: up : running module addressvirtual
debug: eth5: up : running module usercmds
debug: eth5: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: eth5: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: eth5: post-up : running module usercmds
debug: eth5: statemanager sync state pre-up
info: eth4: running ops ...
debug: eth4: pre-up : running module xfrm
debug: eth4: pre-up : running module link
debug: eth4: pre-up : running module bond
debug: eth4: pre-up : running module vlan
debug: eth4: pre-up : running module vxlan
debug: eth4: pre-up : running module usercmds
debug: eth4: pre-up : running module bridge
debug: eth4: pre-up : running module bridgevlan
debug: eth4: pre-up : running module tunnel
debug: eth4: pre-up : running module vrf
debug: _cache_get(['eth4', 'master']) : ['master']
debug: eth4: up : running module dhcp
debug: eth4: up : running module address
debug: reading '/sys/class/net/eth4/ifalias'
info: reading '/proc/sys/net/mpls/conf/eth4/input'
info: executing /sbin/sysctl net.mpls.conf.eth4.input=0
debug: eth4: ipv6 addrgen already on
debug: _cache_get(['eth4', 'addrs']) : ['addrs']
debug: reading '/sys/class/net/eth4/mtu'
info: executing /bin/ip -force -batch - [link set dev eth4 mtu 9000
]
debug: eth4: up : running module addressvirtual
debug: eth4: up : running module usercmds
debug: eth4: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: eth4: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: eth4: post-up : running module usercmds
debug: eth4: statemanager sync state pre-up
info: bond0: running ops ...
debug: bond0: pre-up : running module xfrm
debug: bond0: pre-up : running module link
debug: bond0: pre-up : running module bond
info: bond0: set bond-mode 4
info: bond0: set bond-xmit-hash-policy layer3+4
info: bond0: set bond-miimon 100
info: bond0: set bond-lacp-rate fast
debug: _cache_get(['bond0', 'linkinfo', 18]) : ['bond0']
info: bond0: netlink: ip link add bond0 type bond with attributes
debug: bond0: ifla_info_data OrderedDict([(1, 4), (14, 1), (3, 100), (21, 1)])
debug: _cache_get(['bond0', 'linkinfo', 'slaves']) : ['bond0']
debug: reading '/sys/class/net/bond0/bonding/slaves'
debug: reading '/sys/class/net/eth5/flags'
info: eth5: netlink: ip link set dev eth5 master bond0 
info: eth5: netlink: ip link set dev eth5 up
debug: reading '/sys/class/net/eth4/flags'
info: eth4: netlink: ip link set dev eth4 master bond0 
info: eth4: netlink: ip link set dev eth4 up
debug: bond0: pre-up : running module vlan
debug: bond0: pre-up : running module vxlan
debug: bond0: pre-up : running module usercmds
debug: bond0: pre-up : running module bridge
debug: bond0: pre-up : running module bridgevlan
debug: bond0: pre-up : running module tunnel
debug: bond0: pre-up : running module vrf
debug: _cache_get(['bond0', 'master']) : ['bond0']
info: bond0: netlink: ip link set dev bond0 up
debug: bond0: up : running module dhcp
debug: bond0: up : running module address
debug: reading '/sys/class/net/bond0/ifalias'
info: reading '/proc/sys/net/mpls/conf/bond0/input'
info: executing /sbin/sysctl net.mpls.conf.bond0.input=0
debug: _cache_get(['bond0', 'af_spec', 10]) : ['bond0']
debug: bond0: ipv6 addrgen probably not supported or disabled on this device
debug: _cache_get(['bond0', 'addrs']) : ['bond0']
debug: _cache_get(['bond0', 'mtu']) : ['bond0']
debug: reading '/sys/class/net/bond0/mtu'
info: executing /bin/ip -force -batch - [link set dev bond0 mtu 9000
debug: bond0: up : running module addressvirtual
debug: bond0: up : running module usercmds
debug: bond0: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: bond0: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: bond0: post-up : running module usercmds
debug: bond0: statemanager sync state pre-up
info: bond0.94: running ops ...
debug: bond0.94: pre-up : running module xfrm
debug: bond0.94: pre-up : running module link
debug: bond0.94: pre-up : running module bond
debug: bond0.94: pre-up : running module vlan
debug: _cache_get(['bond0.94', 'linkinfo', 'vlan_protocol']) : ['bond0.94']
info: bond0.94: netlink: ip link add link bond0 name bond0.94 type vlan id 94 protocol 802.1q
debug: bond0.94: pre-up : running module vxlan
debug: bond0.94: pre-up : running module usercmds
debug: bond0.94: pre-up : running module bridge
debug: bond0.94: pre-up : running module bridgevlan
debug: bond0.94: pre-up : running module tunnel
debug: bond0.94: pre-up : running module vrf
debug: _cache_get(['bond0.94', 'master']) : ['bond0.94']
info: bond0.94: netlink: ip link set dev bond0.94 up
debug: bond0.94: up : running module dhcp
debug: bond0.94: up : running module address
debug: reading '/sys/class/net/bond0.94/ifalias'
info: reading '/proc/sys/net/mpls/conf/bond0.94/input'
info: executing /sbin/sysctl net.mpls.conf.bond0/94.input=0
debug: _cache_get(['bond0.94', 'af_spec', 10]) : ['bond0.94']
debug: bond0.94: ipv6 addrgen probably not supported or disabled on this device
debug: _cache_get(['bond0.94', 'addrs']) : ['bond0.94']
debug: reading '/sys/class/net/bond0/mtu'
debug: _cache_get(['bond0.94', 'mtu']) : ['bond0.94']
debug: reading '/sys/class/net/bond0.94/mtu'
info: executing /bin/ip -force -batch - [addr add 10.3.94.107/24 dev bond0.94
link set dev bond0.94 mtu 1500
]
info: executing /bin/ip route add default via 10.3.94.1 proto kernel dev bond0.94 onlink
debug: bond0.94: up : running module addressvirtual
debug: bond0.94: up : running module usercmds
debug: bond0.94: up : running script /etc/network/if-up.d/openntpd
info: executing /etc/network/if-up.d/openntpd
debug: bond0.94: up : running script /etc/network/if-up.d/chrony
info: executing /etc/network/if-up.d/chrony
debug: bond0.94: post-up : running module usercmds
debug: bond0.94: statemanager sync state pre-up
debug: saving state ..

[BarbarossaTM]

Hi @aderumier

When you put bond0 into a bridge, why not configuring br0.94 instead of bond0.94?

Best Max

[aderumier]

Hi @aderumier

When you put bond0 into a bridge, why not configuring br0.94 instead of bond0.94?

Best Max

It's not a vlan-aware bridge, does this work ? Edit: it's working with vmbr1.94 even wihout vlan-aware.

I'll try again with classic ifupdown && bond0.94 + vmbr1

[BarbarossaTM]

I did this before there were vlan-aware bridges, so it should ;-)

I'm wondering what happens, when you specify

vlan-raw-device bond0

on bond0.94

[aderumier]

I did this before there were vlan-aware bridges, so it should ;-)

I'm wondering what happens, when you specify

vlan-raw-device bond0

on bond0.94

no, it's not working.

I have done test with classic ifupdown, it's working fine with bond0.94.

For now, I'm going to use vmbr1.94, as it seem to working fine. But it could be great to have bond0.94 working too.

(I'm looking to push ifupdown2 as default on proxmox hypervisor, but I'll like to avoid breaking users setup)

[julienfortin]

Hi @aderumier,

We officially support kernel 4.19 but haven't experimented with kernel 5. You might be the first one testing it on a kernel 5+, so this is unknown territory. I'll chat with the team and we will try to figure out what is happening.

[aderumier]

Hi @aderumier,

We officially support kernel 4.19 but haven't experimented with kernel 5. You might be the first one testing it on a kernel 5+, so this is unknown territory. I'll chat with the team and we will try to figure out what is happening.

thanks Julien. I'll do test with kernel 4.15 to see if behaviour is the same.

[aderumier]

I have found something, it seem that order in /etc/network/interfaces matters.

working config

auto bond0
iface bond0
        bond-mode 4
        bond-miimon 100
        bond-lacp_rate fast
        bond-xmit-hash-policy layer3+4
        bond-slaves eth4 eth5
        mtu 9000

auto vmbr1
iface vmbr1
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        mtu 9000

auto bond0.94
iface bond0.94
        address 10.3.94.107
        netmask 255.255.255.0
        gateway 10.3.94.1
        mtu 1500

ok.log

not working config

auto bond0
iface bond0
        bond-mode 4
        bond-miimon 100
        bond-lacp_rate fast
        bond-xmit-hash-policy layer3+4
        bond-slaves eth4 eth5
        mtu 9000

auto bond0.94
iface bond0.94
        address 10.3.94.107
        netmask 255.255.255.0
        gateway 10.3.94.1
        mtu 1500

auto vmbr1
iface vmbr1
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        mtu 9000

ko.log

[aderumier]

(tested with kernel 4.15, same behaviour)

[alex-ioma]

After hours of research found this issue. I confirm it is still an issue with Proxmox 8.1.4 and ifupdown2.

What's worst is that if you use the proxmox GUI to prepare /ect/network/interfaces, it does so by nicely arranging device by type - i.e. the bond.vid will take precedence over vmbr bridges. Hence, breaking the configuration upon reboot.

The workaround is to edit the interfaces manually and make sure that they are listed as last devices if they a bond's vlan is the one that carries the default gw.

[aderumier]

please report proxmox bug on bugzilla.proxmox.com.

(That's seem to be a regression, because AFAIK, we had fixed the ordering in proxmox code some year ago)

[alex-ioma]

Thanks, will do.

[lawli3t]

The issue here seems to be that as soon as bond0 is a port of a bridge it only gets set up when the bridge gets configured - not when the bond gets configured. The vlan has a dependency on the bond though, not the bridge, which leads to this issue because the dependency graph is wrong then. It works when you use the bridge as vlan-raw-device, but that is obviously not an equivalent configuration.

Not sure how one would go about solving this though - the cleanest way would probably be that the vlan should have a dependency on the bridge, rather than the vlan-raw-device, but I see no way of getting this information from inside the vlan addon. Maybe in ifupdownmain inside populate_dependency_info?

edit: seems like this can be avoided by setting the config option link_master_slave=0 - nevertheless I think dependency resolution should somehow consider this when link_master_slave=1 is set

[adamcoffee1]

Thank you @lawli3t, I can confirm this fixed my issue after upgrading Proxmox 7->8.