Open k8scoder192 opened 1 year ago
That image has nmstate-2.2.11-1.el9.x86_64
@cathay4t this is the correct version that has the nispor with NETLINK_GET_STRICT_CHK fix ?
Looks like is still not showing routes properly
routes:
config: []
running: []
Only 2.2.13 has the NETLINK_GET_STRICT_CHK
fix. The 2.2.12 and .11 is still using old nispor 1.2.10. :(
I haven't release that to OCP yet.
Only 2.2.13 has the
NETLINK_GET_STRICT_CHK
fix. The 2.2.12 and .11 is still using old nispor 1.2.10. :( I haven't release that to OCP yet.
@cathay4t can we please expedite that? I reported the problem in 1174 back in April and need to get this as soon as possible. Thank you tremendously for your support.
Looks is a matter of doing another release, we did it too early I think Output for image re-creation.
Running transaction
Preparing : 1/1
Installing : nmstate-2.2.13-1.el9.x86_64 1/1
Running scriptlet: nmstate-2.2.13-1.el9.x86_64 1/1
Verifying : nmstate-2.2.13-1.el9.x86_64 1/1
@k8scoder192 can you try "v0.80.0" it contains nmstate-2.2.13-1.el9.x86_64
@qinqon yes in the morning. Please let's not close until validated.
@qinqon
Still seeing the issue; loops multiple times
{"level":"info","ts":"2023-07-25T19:16:17.428Z","logger":"probe","msg":"default gw missing","path":"routes.running.next-hop-address","table-id":254} {"level":"error","ts":"2023-07-25T19:16:17.428Z","logger":"probe","msg":"failed to retrieve default gw","error":"default gw missing","errorVerbose":"default gw missing\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.defaultGw\n\t/opt/app-root/src/pkg/probe/probes.go:167\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.runPing\n\t/opt/app-root/src/pkg/probe/probes.go:184\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.pingCondition.func1\n\t/opt/app-root/src/pkg/probe/probes.go:174\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.WaitForWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:594\nk8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:526\nk8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:512\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.Select\n\t/opt/app-root/src/pkg/probe/probes.go:277\ngithub.com/nmstate/kubernetes-nmstate/pkg/client.ApplyDesiredState\n\t/opt/app-root/src/pkg/client/client.go:159\ngithub.com/nmstate/kubernetes-nmstate/controllers/handler.(*NodeNetworkConfigurationPolicyReconciler).Reconcile\n\t/opt/app-root/src/controllers/handler/nodenetworkconfigurationpolicy_controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1571","stacktrace":"github.com/nmstate/kubernetes-nmstate/pkg/probe.runPing\n\t/opt/app-root/src/pkg/probe/probes.go:186\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.pingCondition.func1\n\t/opt/app-root/src/pkg/probe/probes.go:174\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.WaitForWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:594\nk8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:526\nk8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:512\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.Select\n\t/opt/app-root/src/pkg/probe/probes.go:277\ngithub.com/nmstate/kubernetes-nmstate/pkg/client.ApplyDesiredState\n\t/opt/app-root/src/pkg/client/client.go:159\ngithub.com/nmstate/kubernetes-nmstate/controllers/handler.(*NodeNetworkConfigurationPolicyReconciler).Reconcile\n\t/opt/app-root/src/controllers/handler/nodenetworkconfigurationpolicy_controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
exec into the handler pod; I see new warnings with nispor :-/ and it fails to apply the vlan manifest
dnf list --installed |grep nms
nmstate.x86_64 2.2.13-1.el9 @appstream
[root@sandbox3 /]# nmstatectl show
[2023-07-25T19:22:04Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:22:04Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:22:04Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:22:04Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:22:04Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:22:04Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:22:04Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:22:04Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring
hostname:
running: sandbox3
config: sandbox3
dns-resolver:
running:
server:
- 10.0.80.11
- 10.0.80.12
search: []
config:
server:
- 10.0.80.11
- 10.0.80.12
search: []
route-rules:
config:
- family: ipv6
route-table: 255
- family: ipv6
priority: 32766
route-table: 254
- family: ipv4
priority: 9
route-table: 2004
fwmark: '0x200'
fwmask: '0xf00'
- family: ipv4
priority: 10
route-table: 2005
fwmark: '0xa00'
fwmask: '0xf00'
- family: ipv4
priority: 100
route-table: 255
- family: ipv4
priority: 32766
route-table: 254
- family: ipv4
priority: 32767
route-table: 253
routes:
running: []
config: []
interfaces:
- name: bond0
type: bond
profile-name: netplan-bond0
state: up
wait-ip: any
ipv4:
enabled: true
dhcp: false
ipv6:
enabled: true
dhcp: false
autoconf: false
addr-gen-mode: eui64
lldp:
enabled: false
link-aggregation:
options: {}
- name: cilium_host
type: veth
state: ignore
mac-address: 7E:69:A5:F8:9A:B2
mtu: 1500
ipv4:
enabled: true
address:
- ip: 10.0.2.153
prefix-length: 32
ipv6:
enabled: true
address:
- ip: fe80::7c69:a5ff:fef8:9ab2
prefix-length: 64
mptcp:
address-flags: []
accept-all-mac-addresses: false
veth:
peer: cilium_net
- name: cilium_net
type: veth
state: ignore
mac-address: E6:B9:2D:95:E1:17
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: true
address:
- ip: fe80::e4b9:2dff:fe95:e117
prefix-length: 64
accept-all-mac-addresses: false
veth:
peer: cilium_host
- name: cilium_vxlan
type: vxlan
state: ignore
mac-address: 1A:8F:21:0B:B3:FF
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: true
address:
- ip: fe80::188f:21ff:fe0b:b3ff
prefix-length: 64
accept-all-mac-addresses: false
vxlan:
id: 0
learning: false
destination-port: 8472
- name: docker0
type: linux-bridge
state: ignore
mac-address: 02:42:04:9D:E0:BA
mtu: 1500
ipv4:
enabled: true
address:
- ip: 172.17.0.1
prefix-length: 16
ipv6:
enabled: false
mptcp:
address-flags: []
accept-all-mac-addresses: false
bridge:
options:
gc-timer: 3915
group-addr: 01:80:C2:00:00:00
group-forward-mask: 0
group-fwd-mask: 0
hash-max: 512
hello-timer: 0
mac-ageing-time: 300
multicast-last-member-count: 2
multicast-last-member-interval: 100
multicast-membership-interval: 26000
multicast-querier: false
multicast-querier-interval: 25500
multicast-query-interval: 12500
multicast-query-response-interval: 1000
multicast-query-use-ifaddr: false
multicast-router: auto
multicast-snooping: true
multicast-startup-query-count: 2
multicast-startup-query-interval: 3124
stp:
enabled: false
forward-delay: 15
hello-time: 2
max-age: 20
priority: 32768
vlan-protocol: 802.1q
vlan-default-pvid: 1
port: []
- name: enx3a68dd62794f
type: ethernet
state: down
mac-address: 3A:68:DD:62:79:4F
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: false
accept-all-mac-addresses: false
ethernet: {}
- name: eth0
type: ethernet
profile-name: netplan-eth0
state: up
mac-address: 3C:FD:FE:97:DC:20
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: false
controller: '7'
accept-all-mac-addresses: false
lldp:
enabled: false
ethernet:
sr-iov:
total-vfs: 0
vfs: []
- name: eth1
type: ethernet
state: up
mac-address: 3C:FD:FE:97:DC:21
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: false
accept-all-mac-addresses: false
ethernet:
sr-iov:
total-vfs: 0
vfs: []
- name: eth2
type: ethernet
profile-name: netplan-eth2
state: up
mac-address: 3C:FD:FE:97:DC:20
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: false
controller: '7'
accept-all-mac-addresses: false
lldp:
enabled: false
ethernet:
sr-iov:
total-vfs: 0
vfs: []
- name: eth3
type: ethernet
state: up
mac-address: 3C:FD:FE:97:DC:23
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: false
accept-all-mac-addresses: false
ethernet:
sr-iov:
total-vfs: 0
vfs: []
- name: lo
type: loopback
state: ignore
mac-address: 00:00:00:00:00:00
mtu: 65536
ipv4:
enabled: true
address:
- ip: 127.0.0.1
prefix-length: 8
ipv6:
enabled: true
address:
- ip: ::1
prefix-length: 128
accept-all-mac-addresses: false
- name: lxc723845584691
type: veth
state: ignore
mac-address: 86:30:4E:DD:2E:41
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: true
address:
- ip: fe80::8430:4eff:fedd:2e41
prefix-length: 64
accept-all-mac-addresses: false
- name: lxc_health
type: veth
state: ignore
mac-address: BA:11:23:52:E6:2A
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: true
address:
- ip: fe80::b811:23ff:fe52:e62a
prefix-length: 64
accept-all-mac-addresses: false
- name: lxcc23b7c5337f6
type: veth
state: ignore
mac-address: 6E:49:45:6A:62:D8
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: true
address:
- ip: fe80::6c49:45ff:fe6a:62d8
prefix-length: 64
accept-all-mac-addresses: false
- name: lxce735f0078025
type: veth
state: ignore
mac-address: 82:CB:47:37:C2:EB
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: true
address:
- ip: fe80::80cb:47ff:fe37:c2eb
prefix-length: 64
accept-all-mac-addresses: false
- name: lxcec4a21d034ef
type: veth
state: ignore
mac-address: AA:7E:EA:5A:7A:9C
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: true
address:
- ip: fe80::a87e:eaff:fe5a:7a9c
prefix-length: 64
accept-all-mac-addresses: false
- name: lxcf022b01c3315
type: veth
state: ignore
mac-address: DA:15:6D:71:2A:D7
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: true
address:
- ip: fe80::d815:6dff:fe71:2ad7
prefix-length: 64
accept-all-mac-addresses: false
- name: lxcf88273ffaf67
type: veth
state: ignore
mac-address: 8E:77:2B:B2:0F:1A
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: true
address:
- ip: fe80::8c77:2bff:feb2:f1a
prefix-length: 64
accept-all-mac-addresses: false
- name: lxcfe23c2d8cdb2
type: veth
state: ignore
mac-address: B2:CF:02:ED:B2:DB
mtu: 1500
ipv4:
enabled: false
ipv6:
enabled: true
address:
- ip: fe80::b0cf:2ff:feed:b2db
prefix-length: 64
accept-all-mac-addresses: false
ovs-db: {}
ovn: {}
More info
**kubectl get nodenetworkstate sandbox3 -o yaml|grep -A4 routes**
routes:
config: []
running: []
handlerNmstateVersion: nmstatectl 2.2.13
hostNetworkManagerVersion: 1.22.10
Worker node OS / Kernel info
root@sandbox3:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
root@sandbox3:~# uname -r
4.15.0-041500-generic
FYI this isn't the same exact cluster as the orig I posted previously, that cluster was taken down. This cluster is similar but only worker sandbox3 has networkmanager installed, hence my yaml is targeting that node only (the handler pods for the other nodes errored out due to no networkmanager)
yaml
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: usb-int-v200
spec:
desiredState:
interfaces:
- ipv4:
dhcp: false
enabled: false
mac-address: 02:00:22:18:dd:ab
name: usb-int-v200
state: up
type: vlan
vlan:
base-iface: bond0
id: 200
nodeSelector:
kubernetes.io/hostname: sandbox3
NNCE for failed apply
k describe nnce sandbox3.usb-int-v200
Name: sandbox3.usb-int-v200
Namespace:
Labels: nmstate.io/node=sandbox3
nmstate.io/policy=usb-int-v200
Annotations: <none>
API Version: nmstate.io/v1beta1
Kind: NodeNetworkConfigurationEnactment
Metadata:
Creation Timestamp: 2023-07-25T19:15:12Z
Generation: 1
Managed Fields:
API Version: nmstate.io/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:nmstate.io/node:
f:nmstate.io/policy:
f:ownerReferences:
.:
k:{"uid":"0d77c65b-ba0b-428b-babc-d6662f24ecad"}:
Manager: manager
Operation: Update
Time: 2023-07-25T19:15:12Z
API Version: nmstate.io/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
f:desiredState:
.:
f:interfaces:
f:desiredStateMetaInfo:
f:policyGeneration:
Manager: manager
Operation: Update
Subresource: status
Time: 2023-07-25T19:15:12Z
Owner References:
API Version: v1
Kind: Node
Name: sandbox3
UID: 0d77c65b-ba0b-428b-babc-d6662f24ecad
Resource Version: 56441842
UID: 174093c5-7999-4d29-930e-54803889b3d1
Status:
Conditions:
Last Heartbeat Time: 2023-07-25T19:17:24Z
Last Transition Time: 2023-07-25T19:17:24Z
Reason: FailedToConfigure
Status: False
Type: Progressing
Last Heartbeat Time: 2023-07-25T19:17:24Z
Last Transition Time: 2023-07-25T19:17:24Z
Message: error reconciling NodeNetworkConfigurationPolicy on node sandbox3 at desired state apply: "",
failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' '' 'Using 'set' is deprecated, use 'apply' instead.
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring
[2023-07-25T19:17:12Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxcf88273ffaf67 type ethernet
[2023-07-25T19:17:12Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxc723845584691 type ethernet
[2023-07-25T19:17:12Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lo type loopback
[2023-07-25T19:17:12Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxcc23b7c5337f6 type ethernet
[2023-07-25T19:17:12Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface cilium_vxlan type vxlan
[2023-07-25T19:17:12Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxcfe23c2d8cdb2 type ethernet
[2023-07-25T19:17:12Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxc_health type ethernet
[2023-07-25T19:17:12Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxcf022b01c3315 type ethernet
[2023-07-25T19:17:12Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface cilium_host type ethernet
[2023-07-25T19:17:12Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface cilium_net type ethernet
[2023-07-25T19:17:12Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxce735f0078025 type ethernet
[2023-07-25T19:17:12Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface docker0 type linux-bridge
[2023-07-25T19:17:12Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxcec4a21d034ef type ethernet
[2023-07-25T19:17:12Z INFO nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/3
[2023-07-25T19:17:12Z INFO nmstate::nm::query_apply::profile] Creating connection UUID Some("ae6598e5-eca0-4af7-82c6-c4c6922922ef"), ID Some("usb-int-v200"), type Some("vlan") name Some("usb-int-v200")
[2023-07-25T19:17:12Z INFO nmstate::nm::query_apply::profile] Activating connection ae6598e5-eca0-4af7-82c6-c4c6922922ef: usb-int-v200/vlan
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:12Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring
[2023-07-25T19:17:12Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-07-25T19:17:13Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:13Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:13Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:13Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:13Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:13Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:13Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:14Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring
[2023-07-25T19:17:14Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-07-25T19:17:15Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:15Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:15Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:15Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:15Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:15Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:15Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:15Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring
[2023-07-25T19:17:15Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-07-25T19:17:16Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:16Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:16Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:16Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:16Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:16Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:16Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:16Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring
[2023-07-25T19:17:16Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-07-25T19:17:17Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:17Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:17Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:17Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:17Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:17Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:17Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:17Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring
[2023-07-25T19:17:17Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-07-25T19:17:19Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("ae6598e5-eca0-4af7-82c6-c4c6922922ef"), ID Some("usb-int-v200"), type Some("vlan") name Some("usb-int-v200")
[2023-07-25T19:17:19Z INFO nmstate::nm::query_apply::profile] Reapplying connection ae6598e5-eca0-4af7-82c6-c4c6922922ef: usb-int-v200/vlan
[2023-07-25T19:17:19Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:19Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:19Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:19Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:19Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:19Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:19Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:19Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring
[2023-07-25T19:17:19Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-07-25T19:17:20Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:20Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:20Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:20Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:20Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:20Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:20Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:20Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring
[2023-07-25T19:17:20Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-07-25T19:17:21Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:21Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:21Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:21Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:21Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:21Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:21Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:21Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring
[2023-07-25T19:17:21Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-07-25T19:17:22Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:22Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:22Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:22Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:22Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:22Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:22Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:22Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring
[2023-07-25T19:17:22Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-07-25T19:17:23Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:23Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:23Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:23Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:23Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:23Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:23Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-07-25T19:17:24Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring
[2023-07-25T19:17:24Z INFO nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/3
NmstateError: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '"bond0"', current '"7"'
'
Reason: FailedToConfigure
Status: True
Type: Failing
Last Heartbeat Time: 2023-07-25T19:17:24Z
Last Transition Time: 2023-07-25T19:17:24Z
Reason: FailedToConfigure
Status: False
Type: Available
Last Heartbeat Time: 2023-07-25T19:17:24Z
Last Transition Time: 2023-07-25T19:15:12Z
Reason: FailedToConfigure
Status: False
Type: Pending
Last Heartbeat Time: 2023-07-25T19:17:24Z
Last Transition Time: 2023-07-25T19:15:12Z
Reason: SuccessfullyConfigured
Status: False
Type: Aborted
Desired State:
Interfaces:
ipv4:
Dhcp: false
Enabled: false
Mac - Address: 02:00:22:18:dd:ab
Name: usb-int-v200
State: up
Type: vlan
Vlan:
Base - Iface: bond0
Id: 200
Desired State Meta Info:
Policy Generation: 1
Events: <none>
Target Worker Node Network Config
nmcli con show
NAME UUID TYPE DEVICE
netplan-bond0 ed99d67c-a858-3e46-8bab-6a5caa421e47 bond bond0
netplan-eth0 626dd384-8b3d-3690-9511-192b2c79b3fd ethernet eth0
netplan-eth2 e4f315ef-c9bc-3458-827e-dbff93a2bac6 ethernet eth2
ip link show type bond
7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 3c:fd:fe:97:dc:20 brd ff:ff:ff:ff:ff:ff
FYI all resources in the nmstate namespace
k get all -n nmstate -o wide
Warning: kubevirt.io/v1 VirtualMachineInstancePresets is now deprecated and will be removed in v2.
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nmstate-cert-manager-c878c5fc6-kkmqp 1/1 Running 0 32m 10.0.0.248 sandbox <none> <none>
pod/nmstate-handler-8v6jg 0/1 CrashLoopBackOff 11 (37s ago) 32m 10.220.51.34 sandbox <none> <none>
pod/nmstate-handler-9lq27 0/1 CrashLoopBackOff 11 (23s ago) 32m 10.220.51.14 sandbox2 <none> <none>
pod/nmstate-handler-pmdqm 1/1 Running 0 32m 10.220.51.25 sandbox3 <none> <none>
pod/nmstate-handler-rq4zp 0/1 CrashLoopBackOff 11 (14s ago) 32m 10.220.51.10 sandbox4 <none> <none>
pod/nmstate-operator-64bcd868d9-nrr6p 1/1 Running 0 32m 10.0.0.63 sandbox <none> <none>
pod/nmstate-webhook-5b5cb9cc95-4vzwj 1/1 Running 0 32m 10.0.0.86 sandbox <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/nmstate-webhook ClusterIP 172.16.96.63 <none> 443/TCP 32m name=nmstate-webhook
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR
daemonset.apps/nmstate-handler 4 4 1 4 1 kubernetes.io/arch=amd64,kubernetes.io/os=linux 32m nmstate-handler quay.io/nmstate/kubernetes-nmstate-handler:v0.80.0 name=nmstate-handler
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/nmstate-cert-manager 1/1 1 1 32m nmstate-cert-manager quay.io/nmstate/kubernetes-nmstate-handler:v0.80.0 name=nmstate-cert-manager
deployment.apps/nmstate-operator 1/1 1 1 32m nmstate-operator quay.io/nmstate/kubernetes-nmstate-operator:v0.80.0 name=kubernetes-nmstate-operator
deployment.apps/nmstate-webhook 1/1 1 1 32m nmstate-webhook quay.io/nmstate/kubernetes-nmstate-handler:v0.80.0 name=nmstate-webhook
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/nmstate-cert-manager-c878c5fc6 1 1 1 32m nmstate-cert-manager quay.io/nmstate/kubernetes-nmstate-handler:v0.80.0 name=nmstate-cert-manager,pod-template-hash=c878c5fc6
replicaset.apps/nmstate-operator-64bcd868d9 1 1 1 32m nmstate-operator quay.io/nmstate/kubernetes-nmstate-operator:v0.80.0 name=kubernetes-nmstate-operator,pod-template-hash=64bcd868d9
replicaset.apps/nmstate-webhook-5b5cb9cc95 1 1 1 32m nmstate-webhook quay.io/nmstate/kubernetes-nmstate-handler:v0.80.0 name=nmstate-webhook,pod-template-hash=5b5cb9cc95
(again only sandbox3 (worker node) had networkmanger hence the rest of the handler pods failing; all testing only targeted sandbox3)
EDIT
Upgrading to kernel 5.4.0-155-generic resolved the issues; everything applied and quickly.
However, 5.x is not certified for use in production so the Nispor
"fix" doesn't appear to work or is still not in nmstate-2.2.13-1.el9.x86_64
root@sandbox:~# kubectl get nodenetworkstate sandbox3 -o yaml|grep -A4 routes
routes:
config:
- destination: 0.0.0.0/0
metric: 300
next-hop-address: 10.220.51.1
dnf list of packages in handler pod (exec -it) 1.2.10-1.el9 :-(
[root@ /]# dnf list |grep -i nispor
nispor.i686 1.2.10-1.el9 appstream
nispor.x86_64 1.2.10-1.el9 appstream
python3-nispor.noarch 1.2.10-1.el9 appstream
Hey @cathay4t we will have to take a look on it,
Thanks @k8scoder192
@cathay4t can you please look into this? It's still failing with the kernel version listed
The rpm of nispor does not matters any more as nmstate is bundle it up.
@k8scoder192 Can you try to upgrade rpm from https://people.redhat.com/fge/tmp/nmstate-2.2.15-1.el9_2.x86_64.rpm in the nmstate operater?
Sure but the problem still exists in the latest nmstate release. Please have someone look into this
On Wed, Aug 23, 2023, 8:50 AM Gris Ge @.***> wrote:
The rpm of nispor does not matters any more as nmstate is bundle it up.
@k8scoder192 https://github.com/k8scoder192 Can you try to upgrade rpm from https://people.redhat.com/fge/tmp/nmstate-2.2.15-1.el9_2.x86_64.rpm in the nmstate operater?
— Reply to this email directly, view it on GitHub https://github.com/nmstate/kubernetes-nmstate/issues/1196#issuecomment-1690003000, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKYPL6YZWFUUALLWJQFTXRLXWYDCBANCNFSM6AAAAAA2K262SM . You are receiving this because you were mentioned.Message ID: @.***>
@k8scoder192 Can you share a way to setup a cluster on Ubuntu kernel?
Never mind. The fix in nispor is incorrect(I was guessing. My bad).
@k8scoder192 Please try https://people.redhat.com/fge/tmp/nmstate-2.2.15-2.ubuntu_fix.el9.x86_64.rpm
I am also installing Ubuntu 20.04 in my VM to reproduce this problem.
@cathay4t please ensure the kernel version is 4.15.x (example I had 4.15.0-041500-generic), the problem went away with 5.4.0-155-generic). Thank you
The Ubuntu 20.04 ships with 5.4+ kernel. Why should I spend my PlayStation time on a old unsupported kernel shipped by Ubuntu 18.04?
Anyway, above nmstate-2.2.15-2.ubuntu_fix.el9.x86_64.rpm should fixed the problem on old kernel. I will try Ubuntu 18.04 tomorrow.
@cathay4t because the original issue https://github.com/nmstate/kubernetes-nmstate/issues/1174 which was closed prematurely was with 18.04 and 4.15...
That cluster got torn down and was given a new cluster based on 20.04. To replicate the issue, I simply downgraded the Kernel to 4.15.0-041500-generic. This proved that there is something not working well with 4.x kernels. Anyway, ideally just install 18.04 and give it a try (less work than downgrading since it already ships with 4.15.x)
I will try 18.04 tomorrow. Please try above rpm in the meantime.
Will do. Thanks
On Wed, Aug 23, 2023, 9:59 AM Gris Ge @.***> wrote:
I will try 18.04 tomorrow. Please try above rpm in the meantime.
— Reply to this email directly, view it on GitHub https://github.com/nmstate/kubernetes-nmstate/issues/1196#issuecomment-1690122847, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKYPL67S5AQSUHGDUK4FCRDXWYLD3ANCNFSM6AAAAAA2K262SM . You are receiving this because you were mentioned.Message ID: @.***>
Issue reproduced on Ubuntu 18.04 and PR https://github.com/nispor/nispor/pull/235 tested.
Will do official build to openshift and CentOS Stream 9.
@cathay4t @qinqon
The latest version of nmstate is 2.2.14-1.el9 per dnf (queried in the the handler). When is the fixed version going to be upstream? A new build of kubernetes-nmstate will also be needed to pull that in.
dnf --showduplicates list nmstate
Installed Packages
nmstate.x86_64 2.2.13-1.el9 @appstream
Available Packages
nmstate.noarch 2.0.0-0.4.alpha3.el9 appstream
nmstate.x86_64 2.2.10-3.el9 appstream
nmstate.x86_64 2.2.11-1.el9 appstream
nmstate.x86_64 2.2.12-2.el9 appstream
nmstate.x86_64 2.2.13-1.el9 appstream
nmstate.x86_64 2.2.14-1.el9 appstream
Also did you verify the fix worked? v0.80.0 It doesn't seem to on my end at all, seems like a regression. I used to be able to successfully apply it would just poll for the gw forever. I can't even apply now:
nnce shows FailedToConfigure
Here is handler log
{"level":"info","ts":"2023-09-01T16:51:49.300Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"vmaas-sandbox3.usb-int-v200"}
{"level":"error","ts":"2023-09-01T16:51:49.300Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Rolling back network configuration, manual intervention needed: ","nodenetworkconfigurationpolicy":"/usb-int-v200","error":"error reconciling NodeNetworkConfigurationPolicy on node vmaas-sandbox3 at desired state apply: \"\",\n failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' '' 'Using 'set' is deprecated, use 'apply' instead.\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-01T16:51:38Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lo type loopback\n[2023-09-01T16:51:38Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface cilium_host type ethernet\n[2023-09-01T16:51:38Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxcadf733877932 type ethernet\n[2023-09-01T16:51:38Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface cilium_net type ethernet\n[2023-09-01T16:51:38Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxc_health type ethernet\n[2023-09-01T16:51:38Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface docker0 type linux-bridge\n[2023-09-01T16:51:38Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxce36324736136 type ethernet\n[2023-09-01T16:51:38Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface cilium_vxlan type vxlan\n[2023-09-01T16:51:38Z INFO nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/2\n[2023-09-01T16:51:38Z INFO nmstate::nm::query_apply::profile] Creating connection UUID Some(\"06cae4ed-b226-4f09-b460-6dc2ad32f5bf\"), ID Some(\"usb-int-v200\"), type Some(\"vlan\") name Some(\"usb-int-v200\")\n[2023-09-01T16:51:38Z INFO nmstate::nm::query_apply::profile] Activating connection 06cae4ed-b226-4f09-b460-6dc2ad32f5bf: usb-int-v200/vlan\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:38Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-01T16:51:38Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T16:51:39Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:39Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:39Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:39Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:39Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:39Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:39Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:39Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-01T16:51:39Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T16:51:40Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:40Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:40Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:40Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:40Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:40Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:40Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:40Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-01T16:51:40Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T16:51:41Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:41Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:41Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:41Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:41Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:41Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:41Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:41Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-01T16:51:41Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T16:51:42Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:42Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:42Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:42Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:42Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:42Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:42Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:42Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-01T16:51:42Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T16:51:44Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some(\"06cae4ed-b226-4f09-b460-6dc2ad32f5bf\"), ID Some(\"usb-int-v200\"), type Some(\"vlan\") name Some(\"usb-int-v200\")\n[2023-09-01T16:51:44Z INFO nmstate::nm::query_apply::profile] Reapplying connection 06cae4ed-b226-4f09-b460-6dc2ad32f5bf: usb-int-v200/vlan\n[2023-09-01T16:51:44Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:44Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:44Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:44Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:44Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:44Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:44Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:44Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-01T16:51:44Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T16:51:45Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:45Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:45Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:45Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:45Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:45Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:45Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:45Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-01T16:51:45Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T16:51:46Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:46Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:46Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:46Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:46Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:46Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:46Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:47Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-01T16:51:47Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T16:51:48Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:48Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:48Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:48Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:48Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:48Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:48Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:48Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-01T16:51:48Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T16:51:49Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:49Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:49Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:49Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:49Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:49Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:49Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-01T16:51:49Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-01T16:51:49Z INFO nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/2\nNmstateError: VerificationError: Verification failure: usb-int-v200.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n'","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
{"level":"info","ts":"2023-09-01T16:51:49.383Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 1, false: 0, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 0, false: 1, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-v200"}
{"level":"info","ts":"2023-09-01T16:51:49.383Z","logger":"policyconditions","msg":"SetPolicyFailedToConfigure"}
{"level":"info","ts":"2023-09-01T16:51:49.394Z","logger":"policyconditions","msg":"conflict updating policy conditions, retrying","policy":"usb-int-v200"}
{"level":"info","ts":"2023-09-01T16:51:49.441Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 1, false: 0, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 0, false: 1, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-v200"}
{"level":"info","ts":"2023-09-01T16:51:49.441Z","logger":"policyconditions","msg":"SetPolicyFailedToConfigure"}
exec into handler and run nmstatectl show
[root@vmaas-sandbox3 /]# nmstatectl show|grep "\- name:"
[2023-09-01T16:54:16Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-09-01T16:54:16Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-09-01T16:54:16Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-09-01T16:54:16Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-09-01T16:54:16Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-09-01T16:54:16Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-09-01T16:54:16Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)
[2023-09-01T16:54:16Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring
- name: bond0
- name: cilium_host
- name: cilium_net
- name: cilium_vxlan
- name: docker0
- name: enx3a68dd62794f
- name: eth0
- name: eth1
- name: eth2
- name: eth3
- name: lo
- name: lxc_health
- name: lxcadf733877932
- name: lxce36324736136
dnf shows nispor is NOT installed and nmstate is at 2.2.13-1.el9
[root@vmaas-sandbox3 /]# dnf list --installed |grep "nmst\|nisp"
nmstate.x86_64 2.2.13-1.el9 @appstream
wget https://people.redhat.com/fge/tmp/nmstate-2.2.15-2.ubuntu_fix.el9.x86_64.rpm
dnf install nmstate-2.2.15-2.ubuntu_fix.el9.x86_64.rpm -y
dnf list --installed |grep "nmst\|nisp"
nmstate.x86_64 2.2.15-2.ubuntu_fix.el9 @@commandline
And it still fails to apply
NAME STATUS REASON
usb-int-v600 Degraded FailedToConfigure
NAME STATUS STATUS AGE REASON
vmaas-sandbox3.usb-int-v600 Failing 29s FailedToConfigure
nnce output
Message: error reconciling NodeNetworkConfigurationPolicy on node vmaas-sandbox3 at desired state apply: "",
failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' '' 'Using 'set' is deprecated, use 'apply' instead.
[2023-09-01T17:03:41Z INFO nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/3
[2023-09-01T17:03:42Z INFO nmstate::nm::query_apply::profile] Creating connection UUID Some("d72d3e9c-1be8-4f5c-b162-ccbcdb28ca4a"), ID Some("usb-int-v600"), type Some("vlan") name Some("usb-int-v600")
[2023-09-01T17:03:42Z INFO nmstate::nm::query_apply::profile] Activating connection d72d3e9c-1be8-4f5c-b162-ccbcdb28ca4a: usb-int-v600/vlan
[2023-09-01T17:03:42Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-09-01T17:03:43Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-09-01T17:03:44Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-09-01T17:03:45Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-09-01T17:03:46Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-09-01T17:03:48Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("d72d3e9c-1be8-4f5c-b162-ccbcdb28ca4a"), ID Some("usb-int-v600"), type Some("vlan") name Some("usb-int-v600")
[2023-09-01T17:03:48Z INFO nmstate::nm::query_apply::profile] Reapplying connection d72d3e9c-1be8-4f5c-b162-ccbcdb28ca4a: usb-int-v600/vlan
[2023-09-01T17:03:48Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-09-01T17:03:49Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-09-01T17:03:51Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-09-01T17:03:52Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '"bond0"', current '"7"'
[2023-09-01T17:03:53Z INFO nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/3
NmstateError: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '"bond0"', current '"7"'
'
Reason: FailedToConfigure
Status: True
Type: Failing
Last Heartbeat Time: 2023-09-01T17:03:53Z
Last Transition Time: 2023-09-01T17:03:53Z
Reason: FailedToConfigure
Status: False
Type: Available
Last Heartbeat Time: 2023-09-01T17:03:53Z
Last Transition Time: 2023-09-01T17:01:41Z
Reason: FailedToConfigure
Status: False
Type: Pending
Last Heartbeat Time: 2023-09-01T17:03:53Z
Last Transition Time: 2023-09-01T17:01:41Z
Reason: SuccessfullyConfigured
Status: False
Type: Aborted
Desired State:
Interfaces:
ipv4:
Dhcp: false
Enabled: false
Mac - Address: 02:00:34:99:dd:ae
Name: usb-int-v600
State: up
Type: vlan
Vlan:
Base - Iface: bond0
Id: 600
Desired State Meta Info:
Policy Generation: 1
Events: <none>
handler pod log
{"level":"error","ts":"2023-09-01T17:03:41.582Z","logger":"probe","msg":"error pinging default gateway -> output: ''","error":"failed running ping probe: cmd output: 'ping: SO_BINDTODEVICE 7: No such device\n': exit status 2","errorVerbose":"exit status 2\nfailed running ping probe: cmd output: 'ping: SO_BINDTODEVICE 7: No such device\n'\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.ping\n\t/opt/app-root/src/pkg/probe/probes.go:212\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.runPing\n\t/opt/app-root/src/pkg/probe/probes.go:190\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.pingCondition.func1\n\t/opt/app-root/src/pkg/probe/probes.go:174\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.WaitForWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:594\nk8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:526\nk8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:512\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.Select\n\t/opt/app-root/src/pkg/probe/probes.go:277\ngithub.com/nmstate/kubernetes-nmstate/pkg/client.ApplyDesiredState\n\t/opt/app-root/src/pkg/client/client.go:159\ngithub.com/nmstate/kubernetes-nmstate/controllers/handler.(*NodeNetworkConfigurationPolicyReconciler).Reconcile\n\t/opt/app-root/src/controllers/handler/nodenetworkconfigurationpolicy_controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1571","stacktrace":"github.com/nmstate/kubernetes-nmstate/pkg/probe.runPing\n\t/opt/app-root/src/pkg/probe/probes.go:192\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.pingCondition.func1\n\t/opt/app-root/src/pkg/probe/probes.go:174\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.WaitForWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:594\nk8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:526\nk8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:512\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.Select\n\t/opt/app-root/src/pkg/probe/probes.go:277\ngithub.com/nmstate/kubernetes-nmstate/pkg/client.ApplyDesiredState\n\t/opt/app-root/src/pkg/client/client.go:159\ngithub.com/nmstate/kubernetes-nmstate/controllers/handler.(*NodeNetworkConfigurationPolicyReconciler).Reconcile\n\t/opt/app-root/src/controllers/handler/nodenetworkconfigurationpolicy_controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":"2023-09-01T17:03:41.672Z","logger":"probe","msg":"error pinging default gateway -> output: ''","error":"failed running ping probe: cmd output: 'ping: SO_BINDTODEVICE 7: No such device\n': exit status 2","errorVerbose":"exit status 2\nfailed running ping probe: cmd output: 'ping: SO_BINDTODEVICE 7: No such device\n'\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.ping\n\t/opt/app-root/src/pkg/probe/probes.go:212\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.runPing\n\t/opt/app-root/src/pkg/probe/probes.go:190\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.pingCondition.func1\n\t/opt/app-root/src/pkg/probe/probes.go:174\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.WaitForWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:594\nk8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:526\nk8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:512\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.Select\n\t/opt/app-root/src/pkg/probe/probes.go:277\ngithub.com/nmstate/kubernetes-nmstate/pkg/client.ApplyDesiredState\n\t/opt/app-root/src/pkg/client/client.go:159\ngithub.com/nmstate/kubernetes-nmstate/controllers/handler.(*NodeNetworkConfigurationPolicyReconciler).Reconcile\n\t/opt/app-root/src/controllers/handler/nodenetworkconfigurationpolicy_controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1571","stacktrace":"github.com/nmstate/kubernetes-nmstate/pkg/probe.runPing\n\t/opt/app-root/src/pkg/probe/probes.go:192\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.pingCondition.func1\n\t/opt/app-root/src/pkg/probe/probes.go:174\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.WaitForWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:594\nk8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:526\nk8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:512\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.Select\n\t/opt/app-root/src/pkg/probe/probes.go:277\ngithub.com/nmstate/kubernetes-nmstate/pkg/client.ApplyDesiredState\n\t/opt/app-root/src/pkg/client/client.go:159\ngithub.com/nmstate/kubernetes-nmstate/controllers/handler.(*NodeNetworkConfigurationPolicyReconciler).Reconcile\n\t/opt/app-root/src/controllers/handler/nodenetworkconfigurationpolicy_controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
{"level":"info","ts":"2023-09-01T17:03:41.672Z","logger":"probe","msg":"WARNING not selecting ping probe"}
{"level":"info","ts":"2023-09-01T17:03:53.313Z","logger":"enactmentconditions","msg":"NotifyFailedToConfigure","enactment":"vmaas-sandbox3.usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:03:53.319Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- ipv4:\n dhcp: false\n enabled: false\n mac-address: 02:00:34:99:dd:ae\n name: usb-int-v600\n state: up\n type: vlan\n vlan:\n base-iface: bond0\n id: 600\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[{Type:Progressing Status:False Reason:FailedToConfigure Message: LastHeartbeatTime:2023-09-01 17:03:53.319846408 +0000 UTC m=+887.308635838 LastTransitionTime:2023-09-01 17:03:53.319846408 +0000 UTC m=+887.308635838} {Type:Failing Status:True Reason:FailedToConfigure Message:error reconciling NodeNetworkConfigurationPolicy on node vmaas-sandbox3 at desired state apply: \"\",\n failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' '' 'Using 'set' is deprecated, use 'apply' instead.\n[2023-09-01T17:03:41Z INFO nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/3\n[2023-09-01T17:03:42Z INFO nmstate::nm::query_apply::profile] Creating connection UUID Some(\"d72d3e9c-1be8-4f5c-b162-ccbcdb28ca4a\"), ID Some(\"usb-int-v600\"), type Some(\"vlan\") name Some(\"usb-int-v600\")\n[2023-09-01T17:03:42Z INFO nmstate::nm::query_apply::profile] Activating connection d72d3e9c-1be8-4f5c-b162-ccbcdb28ca4a: usb-int-v600/vlan\n[2023-09-01T17:03:42Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:43Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:44Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:45Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:46Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:48Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some(\"d72d3e9c-1be8-4f5c-b162-ccbcdb28ca4a\"), ID Some(\"usb-int-v600\"), type Some(\"vlan\") name Some(\"usb-int-v600\")\n[2023-09-01T17:03:48Z INFO nmstate::nm::query_apply::profile] Reapplying connection d72d3e9c-1be8-4f5c-b162-ccbcdb28ca4a: usb-int-v600/vlan\n[2023-09-01T17:03:48Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:49Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:51Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:52Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:53Z INFO nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/3\nNmstateError: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n' LastHeartbeatTime:2023-09-01 17:03:53.319845085 +0000 UTC m=+887.308634533 LastTransitionTime:2023-09-01 17:03:53.319845085 +0000 UTC m=+887.308634533} {Type:Available Status:False Reason:FailedToConfigure Message: LastHeartbeatTime:2023-09-01 17:03:53.319846023 +0000 UTC m=+887.308635455 LastTransitionTime:2023-09-01 17:03:53.319846023 +0000 UTC m=+887.308635455} {Type:Pending Status:False Reason:FailedToConfigure Message: LastHeartbeatTime:2023-09-01 17:03:53.319846994 +0000 UTC m=+887.308636423 LastTransitionTime:2023-09-01 17:01:41 +0000 UTC} {Type:Aborted Status:False Reason:SuccessfullyConfigured Message: LastHeartbeatTime:2023-09-01 17:03:53.319847577 +0000 UTC m=+887.308637007 LastTransitionTime:2023-09-01 17:01:41 +0000 UTC}]}","enactment":"vmaas-sandbox3.usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:03:53.344Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"vmaas-sandbox3.usb-int-v600"}
{"level":"error","ts":"2023-09-01T17:03:53.344Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Rolling back network configuration, manual intervention needed: ","nodenetworkconfigurationpolicy":"/usb-int-v600","error":"error reconciling NodeNetworkConfigurationPolicy on node vmaas-sandbox3 at desired state apply: \"\",\n failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' '' 'Using 'set' is deprecated, use 'apply' instead.\n[2023-09-01T17:03:41Z INFO nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/3\n[2023-09-01T17:03:42Z INFO nmstate::nm::query_apply::profile] Creating connection UUID Some(\"d72d3e9c-1be8-4f5c-b162-ccbcdb28ca4a\"), ID Some(\"usb-int-v600\"), type Some(\"vlan\") name Some(\"usb-int-v600\")\n[2023-09-01T17:03:42Z INFO nmstate::nm::query_apply::profile] Activating connection d72d3e9c-1be8-4f5c-b162-ccbcdb28ca4a: usb-int-v600/vlan\n[2023-09-01T17:03:42Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:43Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:44Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:45Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:46Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:48Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some(\"d72d3e9c-1be8-4f5c-b162-ccbcdb28ca4a\"), ID Some(\"usb-int-v600\"), type Some(\"vlan\") name Some(\"usb-int-v600\")\n[2023-09-01T17:03:48Z INFO nmstate::nm::query_apply::profile] Reapplying connection d72d3e9c-1be8-4f5c-b162-ccbcdb28ca4a: usb-int-v600/vlan\n[2023-09-01T17:03:48Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:49Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:51Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:52Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-01T17:03:53Z INFO nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/3\nNmstateError: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n'","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
{"level":"info","ts":"2023-09-01T17:03:53.413Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 1, false: 0, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 0, false: 1, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:03:53.414Z","logger":"policyconditions","msg":"SetPolicyFailedToConfigure"}
{"level":"info","ts":"2023-09-01T17:03:53.427Z","logger":"policyconditions","msg":"conflict updating policy conditions, retrying","policy":"usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:03:53.466Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 1, false: 0, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 0, false: 1, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:03:53.466Z","logger":"policyconditions","msg":"SetPolicyFailedToConfigure"}
yaml I am applying
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: usb-int-v600
spec:
desiredState:
interfaces:
- ipv4:
dhcp: false
enabled: false
mac-address: 02:00:34:99:dd:ae
name: usb-int-v600
state: up
type: vlan
vlan:
base-iface: bond0
id: 600
nodeSelector:
kubernetes.io/hostname: vmaas-sandbox3
So I then uninstalled v0.80.0 and tried v0.74.0 It worked, successfully applied that yaml and I saw the vlan interface on the target host So there seems to be a regression with later versions
just to be thorough here is the handler pod output for v0.74.0
{"level":"info","ts":"2023-09-01T17:13:23.069Z","logger":"setup","msg":"Try to take exclusive lock on file: /var/k8s_nmstate/handler_lock"}
{"level":"info","ts":"2023-09-01T17:13:23.069Z","logger":"setup","msg":"Successfully took nmstate exclusive lock"}
I0901 17:13:24.120398 1 request.go:601] Waited for 1.029380638s due to client-side throttling, not priority and fairness, request: GET:https://172.16.0.1:443/apis/objectbucket.io/v1alpha1?timeout=32s
{"level":"info","ts":"2023-09-01T17:13:26.027Z","logger":"setup","msg":"Marking handler as healthy touching healthy file","healthyFile":"/tmp/healthy"}
{"level":"info","ts":"2023-09-01T17:13:26.027Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2023-09-01T17:13:26.028Z","msg":"Starting EventSource","controller":"NodeNetworkState","source":"kind source: *v1.Node"}
{"level":"info","ts":"2023-09-01T17:13:26.028Z","msg":"Starting EventSource","controller":"NodeNetworkState","source":"kind source: *v1beta1.NodeNetworkState"}
{"level":"info","ts":"2023-09-01T17:13:26.028Z","msg":"Starting Controller","controller":"NodeNetworkState"}
{"level":"info","ts":"2023-09-01T17:13:26.028Z","msg":"Starting EventSource","controller":"nodenetworkconfigurationenactment","controllerGroup":"nmstate.io","controllerKind":"NodeNetworkConfigurationEnactment","source":"kind source: *v1beta1.NodeNetworkConfigurationEnactment"}
{"level":"info","ts":"2023-09-01T17:13:26.028Z","msg":"Starting Controller","controller":"nodenetworkconfigurationenactment","controllerGroup":"nmstate.io","controllerKind":"NodeNetworkConfigurationEnactment"}
{"level":"info","ts":"2023-09-01T17:13:26.028Z","msg":"Starting EventSource","controller":"NodeNetworkConfigurationPolicy","source":"kind source: *v1.NodeNetworkConfigurationPolicy"}
{"level":"info","ts":"2023-09-01T17:13:26.028Z","msg":"Starting EventSource","controller":"NodeNetworkConfigurationPolicy","source":"kind source: *v1.Node"}
{"level":"info","ts":"2023-09-01T17:13:26.028Z","msg":"Starting Controller","controller":"NodeNetworkConfigurationPolicy"}
{"level":"info","ts":"2023-09-01T17:13:26.129Z","msg":"Starting workers","controller":"NodeNetworkState","worker count":1}
{"level":"info","ts":"2023-09-01T17:13:26.129Z","msg":"Starting workers","controller":"nodenetworkconfigurationenactment","controllerGroup":"nmstate.io","controllerKind":"NodeNetworkConfigurationEnactment","worker count":1}
{"level":"info","ts":"2023-09-01T17:13:26.129Z","msg":"Starting workers","controller":"NodeNetworkConfigurationPolicy","worker count":1}
{"level":"info","ts":"2023-09-01T17:13:26.732Z","logger":"controllers.Node","msg":"Creating/updating NodeNetworkState"}
{"level":"info","ts":"2023-09-01T17:13:27.145Z","logger":"client","msg":"Skipping NodeNetworkState update, node network configuration not changed"}
{"level":"info","ts":"2023-09-01T17:14:16.435Z","logger":"controllers.NodeNetworkConfigurationPolicy.initializeEnactment","msg":"creating enactment","policy":"usb-int-v600","enactment":"vmaas-sandbox3.usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:14:17.001Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- ipv4:\n dhcp: false\n enabled: false\n mac-address: 02:00:34:99:dd:ae\n name: usb-int-v600\n state: up\n type: vlan\n vlan:\n base-iface: bond0\n id: 600\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[]}","enactment":"vmaas-sandbox3.usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:14:17.015Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"vmaas-sandbox3.usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:14:17.044Z","logger":"enactmentconditions","msg":"NotifyProgressing","enactment":"vmaas-sandbox3.usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:14:17.049Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- ipv4:\n dhcp: false\n enabled: false\n mac-address: 02:00:34:99:dd:ae\n name: usb-int-v600\n state: up\n type: vlan\n vlan:\n base-iface: bond0\n id: 600\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[{Type:Progressing Status:True Reason:ConfigurationProgressing Message:Applying desired state MessageEncoded: LastHeartbeatTime:2023-09-01 17:14:17.049452793 +0000 UTC m=+54.038457031 LastTransitionTime:2023-09-01 17:14:17.049452793 +0000 UTC m=+54.038457031} {Type:Failing Status:Unknown Reason:ConfigurationProgressing Message: MessageEncoded: LastHeartbeatTime:2023-09-01 17:14:17.04945397 +0000 UTC m=+54.038458194 LastTransitionTime:2023-09-01 17:14:17.04945397 +0000 UTC m=+54.038458194} {Type:Available Status:Unknown Reason:ConfigurationProgressing Message: MessageEncoded: LastHeartbeatTime:2023-09-01 17:14:17.049455042 +0000 UTC m=+54.038459220 LastTransitionTime:2023-09-01 17:14:17.049455042 +0000 UTC m=+54.038459220} {Type:Pending Status:False Reason:ConfigurationProgressing Message: MessageEncoded: LastHeartbeatTime:2023-09-01 17:14:17.049456069 +0000 UTC m=+54.038460233 LastTransitionTime:2023-09-01 17:14:17.049456069 +0000 UTC m=+54.038460233} {Type:Aborted Status:False Reason:ConfigurationProgressing Message: MessageEncoded: LastHeartbeatTime:2023-09-01 17:14:17.049456498 +0000 UTC m=+54.038460660 LastTransitionTime:2023-09-01 17:14:17.049456498 +0000 UTC m=+54.038460660}]}","enactment":"vmaas-sandbox3.usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:14:17.063Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"vmaas-sandbox3.usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:14:17.092Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 0, unknown: 1}, progressing: {true: 1, false: 0, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 0, false: 0, unknown: 1}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:14:17.092Z","logger":"policyconditions","msg":"SetPolicyProgressing"}
{"level":"info","ts":"2023-09-01T17:14:19.070Z","logger":"unavailable_link_workaround","msg":"Ethernet interface in 'unavailable' state was found, setting explicitly UP","iface":"eth1"}
{"level":"info","ts":"2023-09-01T17:14:19.074Z","logger":"unavailable_link_workaround","msg":"Ethernet interface in 'unavailable' state was found, setting explicitly UP","iface":"eth3"}
{"level":"info","ts":"2023-09-01T17:14:20.326Z","logger":"probe","msg":"Running 'ping' probe"}
{"level":"info","ts":"2023-09-01T17:14:20.884Z","logger":"probe","msg":"Running 'dns' probe"}
{"level":"info","ts":"2023-09-01T17:14:21.744Z","logger":"probe","msg":"Running 'api-server' probe"}
I0901 17:14:22.795122 1 request.go:601] Waited for 1.042586318s due to client-side throttling, not priority and fairness, request: GET:https://172.16.0.1:443/apis/tekton.dev/v1beta1?timeout=32s
{"level":"info","ts":"2023-09-01T17:14:24.153Z","logger":"probe","msg":"Running 'node-readiness' probe"}
{"level":"info","ts":"2023-09-01T17:14:24.657Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"nmstate","nodenetworkconfigurationpolicy":"/usb-int-v600","output":"setOutput: Desired state applied: \n---\ninterfaces:\n- name: usb-int-v600\n type: vlan\n state: up\n ipv4:\n enabled: false\n dhcp: false\n mac-address: 02:00:34:99:dd:ae\n vlan:\n base-iface: bond0\n id: 600\nCheckpoint: NetworkManager|/org/freedesktop/NetworkManager/Checkpoint/4\n \n"}
{"level":"info","ts":"2023-09-01T17:14:24.658Z","logger":"enactmentconditions","msg":"NotifySuccess","enactment":"vmaas-sandbox3.usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:14:24.665Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- ipv4:\n dhcp: false\n enabled: false\n mac-address: 02:00:34:99:dd:ae\n name: usb-int-v600\n state: up\n type: vlan\n vlan:\n base-iface: bond0\n id: 600\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[{Type:Progressing Status:False Reason:SuccessfullyConfigured Message: MessageEncoded: LastHeartbeatTime:2023-09-01 17:14:24.665384758 +0000 UTC m=+61.654388924 LastTransitionTime:2023-09-01 17:14:24.665384758 +0000 UTC m=+61.654388924} {Type:Failing Status:False Reason:SuccessfullyConfigured Message: MessageEncoded: LastHeartbeatTime:2023-09-01 17:14:24.665384351 +0000 UTC m=+61.654388529 LastTransitionTime:2023-09-01 17:14:24.665384351 +0000 UTC m=+61.654388529} {Type:Available Status:True Reason:SuccessfullyConfigured Message:successfully reconciled MessageEncoded: LastHeartbeatTime:2023-09-01 17:14:24.665383745 +0000 UTC m=+61.654387992 LastTransitionTime:2023-09-01 17:14:24.665383745 +0000 UTC m=+61.654387992} {Type:Pending Status:False Reason:SuccessfullyConfigured Message: MessageEncoded: LastHeartbeatTime:2023-09-01 17:14:24.665385205 +0000 UTC m=+61.654389371 LastTransitionTime:2023-09-01 17:14:17 +0000 UTC} {Type:Aborted Status:False Reason:SuccessfullyConfigured Message: MessageEncoded: LastHeartbeatTime:2023-09-01 17:14:24.665385877 +0000 UTC m=+61.654390041 LastTransitionTime:2023-09-01 17:14:17 +0000 UTC}]}","enactment":"vmaas-sandbox3.usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:14:24.685Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"vmaas-sandbox3.usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:14:24.685Z","logger":"controllers.NodeNetworkConfigurationPolicy.forceNNSRefresh","msg":"forcing NodeNetworkState refresh after NNCP applied","node":"vmaas-sandbox3"}
{"level":"info","ts":"2023-09-01T17:14:24.773Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:14:24.773Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2023-09-01T17:14:24.788Z","logger":"policyconditions","msg":"conflict updating policy conditions, retrying","policy":"usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:14:24.823Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-v600"}
{"level":"info","ts":"2023-09-01T17:14:24.823Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2023-09-01T17:14:25.313Z","logger":"controllers.Node","msg":"Creating/updating NodeNetworkState"}
The error message usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n
indicate the nispor
failed to convert interface index 7 to interface name bond0
when querying the kernel.
Let me try the same in Ubuntu 18.04.
Cannot reproduce this usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n
error in Ubuntu 18.04. (the NM in Ubuntu 18.04 is too old, so I manually created a VLAN over bond, the nmstatectl show
works well, so not a kernel problem here).
@k8scoder192 Can you share the steps for me to reproduce this problem locally?
@cathay4t I just confirmed with the team we are migrating to Ubuntu 22.04 with 5.x Kernel so hopefully we don't need to worry about this issue.
I did some prelim testing on 22.04 and I still ran into an issue with nmstate not picking up dns-resolver info so it would loop on the prob for very long time. I need to do more testing and ensure I'm on v.0.80 before I comment further
If still interested in the issue I reported previously, I upped the version of Ubuntu on the worker node to 20.04 with kernel 4.15.0-041500-generic since that's the kernel I saw the issue with; master is still on 18.04 / 4.15.0-208-generic but this shouldn't matter
version of nmcli on worker node (1.22.10)
root@vmaas-sandbox3:~# nmcli --version
nmcli tool, version 1.22.10
k-nmstate
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nmstate-cert-manager-c878c5fc6-hbfhh 1/1 Running 0 16s 10.0.0.35 vmaas-sandbox <none> <none>
pod/nmstate-handler-hr69v 1/1 Running 0 15s 10.220.51.25 vmaas-sandbox3 <none> <none>
pod/nmstate-operator-64bcd868d9-x97d8 1/1 Running 0 30s 10.0.0.98 vmaas-sandbox <none> <none>
pod/nmstate-webhook-5b5cb9cc95-z5rcr 1/1 Running 0 16s 10.0.0.36 vmaas-sandbox <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/nmstate-webhook ClusterIP 172.16.231.78 <none> 443/TCP 16s name=nmstate-webhook
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR
daemonset.apps/nmstate-handler 4 4 1 4 1 kubernetes.io/arch=amd64,kubernetes.io/os=linux 17s nmstate-handler quay.io/nmstate/kubernetes-nmstate-handler:v0.80.0 name=nmstate-handler
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/nmstate-cert-manager 1/1 1 1 17s nmstate-cert-manager quay.io/nmstate/kubernetes-nmstate-handler:v0.80.0 name=nmstate-cert-manager
deployment.apps/nmstate-operator 1/1 1 1 31s nmstate-operator quay.io/nmstate/kubernetes-nmstate-operator:v0.80.0 name=kubernetes-nmstate-operator
deployment.apps/nmstate-webhook 1/1 1 1 18s nmstate-webhook quay.io/nmstate/kubernetes-nmstate-handler:v0.80.0 name=nmstate-webhook
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/nmstate-cert-manager-c878c5fc6 1 1 1 17s nmstate-cert-manager quay.io/nmstate/kubernetes-nmstate-handler:v0.80.0 name=nmstate-cert-manager,pod-template-hash=c878c5fc6
replicaset.apps/nmstate-operator-64bcd868d9 1 1 1 31s nmstate-operator quay.io/nmstate/kubernetes-nmstate-operator:v0.80.0 name=kubernetes-nmstate-operator,pod-template-hash=64bcd868d9
replicaset.apps/nmstate-webhook-5b5cb9cc95 1 1 0 18s nmstate-webhook quay.io/nmstate/kubernetes-nmstate-handler:v0.80.0 name=nmstate-webhook,pod-template-hash=5b5cb9cc95
yaml for vlan
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: usb-int-v600
spec:
desiredState:
interfaces:
- ipv4:
dhcp: false
enabled: false
mac-address: 02:00:34:99:dd:ae
name: usb-int-v600
state: up
type: vlan
vlan:
base-iface: bond0
id: 600
nodeSelector:
kubernetes.io/hostname: vmaas-sandbox3
start fresh, delete vlan600 on worker node
root@vmaas-sandbox3:~# nmcli con del usb-int-v600
Connection 'usb-int-v600' (8a99ca85-8373-4ab7-87f2-2f7a686202f8) successfully deleted.
root@vmaas-sandbox3:~# ip link show type vlan <-- empty response so it's gone
delete all nncp/nnce
root@vmaas-sandbox:~/nmstate# k delete nncp --all -A; k delete nnce --all -A
No resources found
No resources found
Start a log tail on the handler
k logs -f pod/nmstate-handler-hr69v -n nmstate
Apply the vlan600 yaml
k apply -f usb-v600.yaml
nodenetworkconfigurationpolicy.nmstate.io/usb-int-v600 created
Handler shows very long looping on probe default gw
{"level":"error","ts":"2023-09-05T16:14:13.663Z","logger":"probe","msg":"failed to retrieve default gw","error":"default gw missing","errorVerbose":"default gw missing\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.defaultGw\n\t/opt/app-root/src/pkg/probe/probes.go:167\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.runPing\n\t/opt/app-root/src/pkg/probe/probes.go:184\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.pingCondition.func1\n\t/opt/app-root/src/pkg/probe/probes.go:174\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.WaitForWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:594\nk8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:526\nk8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:512\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.Select\n\t/opt/app-root/src/pkg/probe/probes.go:277\ngithub.com/nmstate/kubernetes-nmstate/pkg/client.ApplyDesiredState\n\t/opt/app-root/src/pkg/client/client.go:159\ngithub.com/nmstate/kubernetes-nmstate/controllers/handler.(*NodeNetworkConfigurationPolicyReconciler).Reconcile\n\t/opt/app-root/src/controllers/handler/nodenetworkconfigurationpolicy_controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1571","stacktrace":"github.com/nmstate/kubernetes-nmstate/pkg/probe.runPing\n\t/opt/app-root/src/pkg/probe/probes.go:186\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.pingCondition.func1\n\t/opt/app-root/src/pkg/probe/probes.go:174\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.WaitForWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:594\nk8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:526\nk8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:512\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.Select\n\t/opt/app-root/src/pkg/probe/probes.go:277\ngithub.com/nmstate/kubernetes-nmstate/pkg/client.ApplyDesiredState\n\t/opt/app-root/src/pkg/client/client.go:159\ngithub.com/nmstate/kubernetes-nmstate/controllers/handler.(*NodeNetworkConfigurationPolicyReconciler).Reconcile\n\t/opt/app-root/src/controllers/handler/nodenetworkconfigurationpolicy_controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
Then
"msg":"failed to retrieve default gw","error":"default gw missing","errorVerbose":"default gw .....
Then
{"level":"info","ts":"2023-09-05T16:15:45.129Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- ipv4:\n dhcp: false\n enabled: false\n mac-address: 02:00:34:99:dd:ae\n name: usb-int-v600\n state: up\n type: vlan\n vlan:\n base-iface: bond0\n id: 600\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[{Type:Progressing Status:False Reason:FailedToConfigure Message: LastHeartbeatTime:2023-09-05 16:15:45.129479106 +0000 UTC m=+805.320197004 LastTransitionTime:2023-09-05 16:15:45.129479106 +0000 UTC m=+805.320197004} {Type:Failing Status:True Reason:FailedToConfigure Message:error reconciling NodeNetworkConfigurationPolicy on node vmaas-sandbox3 at desired state apply: \"\",\n failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' '' 'Using 'set' is deprecated, use 'apply' instead.\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxcadf733877932 type ethernet\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxce36324736136 type ethernet\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface cilium_vxlan type vxlan\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface docker0 type linux-bridge\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lo type loopback\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface cilium_host type ethernet\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface cilium_net type ethernet\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxc_health type ethernet\n[2023-09-05T16:15:33Z INFO nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/6\n[2023-09-05T16:15:34Z INFO nmstate::nm::query_apply::profile] Creating connection UUID Some(\"9e2a9aca-ba01-403f-ae3b-ec64c525a9a4\"), ID Some(\"usb-int-v600\"), type Some(\"vlan\") name Some(\"usb-int-v600\")\n[2023-09-05T16:15:34Z INFO nmstate::nm::query_apply::profile] Activating connection 9e2a9aca-ba01-403f-ae3b-ec64c525a9a4: usb-int-v600/vlan\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:34Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:35Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:36Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:37Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:38Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:40Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some(\"9e2a9aca-ba01-403f-ae3b-ec64c525a9a4\"), ID Some(\"usb-int-v600\"), type Some(\"vlan\") name Some(\"usb-int-v600\")\n[2023-09-05T16:15:40Z INFO nmstate::nm::query_apply::profile] Reapplying connection 9e2a9aca-ba01-403f-ae3b-ec64c525a9a4: usb-int-v600/vlan\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:40Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:41Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:42Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:43Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:45Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:45Z INFO nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/6\nNmstateError: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n' LastHeartbeatTime:2023-09-05 16:15:45.129477871 +0000 UTC m=+805.320195851 LastTransitionTime:2023-09-05 16:15:45.129477871 +0000 UTC m=+805.320195851} {Type:Available Status:False Reason:FailedToConfigure Message: LastHeartbeatTime:2023-09-05 16:15:45.129478727 +0000 UTC m=+805.320196621 LastTransitionTime:2023-09-05 16:15:45.129478727 +0000 UTC m=+805.320196621} {Type:Pending Status:False Reason:FailedToConfigure Message: LastHeartbeatTime:2023-09-05 16:15:45.129479555 +0000 UTC m=+805.320197451 LastTransitionTime:2023-09-05 16:13:33 +0000 UTC} {Type:Aborted Status:False Reason:SuccessfullyConfigured Message: LastHeartbeatTime:2023-09-05 16:15:45.12948019 +0000 UTC m=+805.320198087 LastTransitionTime:2023-09-05 16:13:33 +0000 UTC}]}","enactment":"vmaas-sandbox3.usb-int-v600"} {"level":"info","ts":"2023-09-05T16:15:45.155Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"vmaas-sandbox3.usb-int-v600"}
AND
{"level":"error","ts":"2023-09-05T16:15:45.155Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Rolling back network configuration, manual intervention needed: ","nodenetworkconfigurationpolicy":"/usb-int-v600","error":"error reconciling NodeNetworkConfigurationPolicy on node vmaas-sandbox3 at desired state apply: \"\",\n failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' '' 'Using 'set' is deprecated, use 'apply' instead.\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:33Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxcadf733877932 type ethernet\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxce36324736136 type ethernet\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface cilium_vxlan type vxlan\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface docker0 type linux-bridge\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lo type loopback\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface cilium_host type ethernet\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface cilium_net type ethernet\n[2023-09-05T16:15:33Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface lxc_health type ethernet\n[2023-09-05T16:15:33Z INFO nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/6\n[2023-09-05T16:15:34Z INFO nmstate::nm::query_apply::profile] Creating connection UUID Some(\"9e2a9aca-ba01-403f-ae3b-ec64c525a9a4\"), ID Some(\"usb-int-v600\"), type Some(\"vlan\") name Some(\"usb-int-v600\")\n[2023-09-05T16:15:34Z INFO nmstate::nm::query_apply::profile] Activating connection 9e2a9aca-ba01-403f-ae3b-ec64c525a9a4: usb-int-v600/vlan\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:34Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:34Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:35Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:35Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:36Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:36Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:37Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:37Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:38Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:38Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:40Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some(\"9e2a9aca-ba01-403f-ae3b-ec64c525a9a4\"), ID Some(\"usb-int-v600\"), type Some(\"vlan\") name Some(\"usb-int-v600\")\n[2023-09-05T16:15:40Z INFO nmstate::nm::query_apply::profile] Reapplying connection 9e2a9aca-ba01-403f-ae3b-ec64c525a9a4: usb-int-v600/vlan\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:40Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:40Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:41Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:41Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:42Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:42Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:43Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:43Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve Boot route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve Static route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve Ra route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve Dhcp route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve Mrouted route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve KeepAlived route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:44Z WARN nmstate::nispor::route] Failed to retrieve Babel route via nispor: Failed to set socket option NETLINK_GET_STRICT_CHK: error Protocol not available (os error 92)\n[2023-09-05T16:15:45Z INFO nmstate::nm::show] Got unsupported interface type generic: lo, ignoring\n[2023-09-05T16:15:45Z INFO nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/6\nNmstateError: VerificationError: Verification failure: usb-int-v600.interface.vlan.base-iface desire '\"bond0\"', current '\"7\"'\n'","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"} {"level":"info","ts":"2023-09-05T16:15:45.220Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 1, false: 0, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 0, false: 1, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-v600"} {"level":"info","ts":"2023-09-05T16:15:45.220Z","logger":"policyconditions","msg":"SetPolicyFailedToConfigure"} {"level":"info","ts":"2023-09-05T16:15:45.234Z","logger":"policyconditions","msg":"conflict updating policy conditions, retrying","policy":"usb-int-v600"} {"level":"info","ts":"2023-09-05T16:15:45.281Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 1, false: 0, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 0, false: 1, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-v600"} {"level":"info","ts":"2023-09-05T16:15:45.281Z","logger":"policyconditions","msg":"SetPolicyFailedToConfigure"}
What happened: Please reference Issue 1174; I'm running into the same issue
Using the latest version quay.io/nmstate/kubernetes-nmstate-handler:v0.79.0
As such it keeps looping on
{"level":"info","ts":"2023-07-14T21:47:42.674Z","logger":"probe","msg":"default gw missing","path":"routes.running.next-hop-address","table-id":254} {"level":"error","ts":"2023-07-14T21:47:42.674Z","logger":"probe","msg":"failed to retrieve default gw","error":"default gw missing","errorVerbose":"default gw missing\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.defaultGw\n\t/opt/app-root/src/pkg/probe/probes.go:160\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.runPing\n\t/opt/app-root/src/pkg/probe/probes.go:177\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.pingCondition.func1\n\t/opt/app-root/src/pkg/probe/probes.go:167\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.WaitForWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:594\nk8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:526\nk8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:512\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.Select\n\t/opt/app-root/src/pkg/probe/probes.go:264\ngithub.com/nmstate/kubernetes-nmstate/pkg/client.ApplyDesiredState\n\t/opt/app-root/src/pkg/client/client.go:159\ngithub.com/nmstate/kubernetes-nmstate/controllers/handler.(*NodeNetworkConfigurationPolicyReconciler).Reconcile\n\t/opt/app-root/src/controllers/handler/nodenetworkconfigurationpolicy_controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1571","stacktrace":"github.com/nmstate/kubernetes-nmstate/pkg/probe.runPing\n\t/opt/app-root/src/pkg/probe/probes.go:179\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.pingCondition.func1\n\t/opt/app-root/src/pkg/probe/probes.go:167\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.WaitForWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:594\nk8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:526\nk8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/opt/app-root/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:512\ngithub.com/nmstate/kubernetes-nmstate/pkg/probe.Select\n\t/opt/app-root/src/pkg/probe/probes.go:264\ngithub.com/nmstate/kubernetes-nmstate/pkg/client.ApplyDesiredState\n\t/opt/app-root/src/pkg/client/client.go:159\ngithub.com/nmstate/kubernetes-nmstate/controllers/handler.(*NodeNetworkConfigurationPolicyReconciler).Reconcile\n\t/opt/app-root/src/controllers/handler/nodenetworkconfigurationpolicy_controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
What you expected to happen:Not to loop for over 4 min (for 1 interface on 1 node) before moving on to next probe. Configuring multiple nodes takes even longer
How to reproduce it (as minimally and precisely as possible):
Use the OS and Kernel version listed below and this manifest yaml
Anything else we need to know?:
OS and Kernel Version
Environment:
NodeNetworkState
on affected nodes (usekubectl get nodenetworkstate <node_name> -o yaml
):Problematic
NodeNetworkConfigurationPolicy
: kubectl get nncp usb-int-v300 -o yam1kubernetes-nmstate image (use
kubectl get pods --all-namespaces -l app=kubernetes-nmstate -o jsonpath='{.items[0].spec.containers[0].image}'
): quay.io/nmstate/kubernetes-nmstate-handler:v0.79.0NetworkManager version (use
nmcli --version
) nmcli tool, version 1.22.10Kubernetes version (use
kubectl version
): v1.26.0OS (e.g. from /etc/os-release): worker1 = VERSION="20.04.6 LTS (Focal Fossa)"
Others:
Routing table
nmstate namespace (NOTE: handler does not run on master node since I do not have networmanager installed there)