NNCP apply takes *very long* time to apply; nmstatectl does not

k8scoder192 commented 1 month ago

What happened: Apply of a nncp takes over 60 seconds. If I exec into the node's nmstate-handler pod and use nmstatectl to create an interface (vlan or bridge) it takes seconds.

What you expected to happen: Apply of nncp to apply quickly

How to reproduce it (as minimally and precisely as possible): Apply a yaml such as this

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: svc-uc-v108-dal4-qz4-sr7-rk047-s04
spec:
  desiredState:
    interfaces:
    - ipv4:
        dhcp: false
        enabled: false
      mac-address: 02:10:2c:2c:7e:52
      name: svc-uc-v108
      state: up
      type: vlan
      vlan:
        base-iface: bond1
        id: 108
  nodeSelector:
    kubernetes.io/hostname: dal4-qz4-sr7-rk047-s04

Anything else we need to know?: nmstate version --> kubernetes-nmstate-handler:v0.80.0

Environment:

NodeNetworkState on affected nodes (use kubectl get nodenetworkstate <node_name> -o yaml):

apiVersion: nmstate.io/v1beta1
kind: NodeNetworkState
metadata:
creationTimestamp: "2023-10-20T21:26:46Z"
generation: 1
name: dal4-qz4-sr7-rk048-s47
ownerReferences:
- apiVersion: v1
kind: Node
name: dal4-qz4-sr7-rk048-s47
uid: 0e2aece7-b94c-4d1e-94d8-cfa8d55c08a3
resourceVersion: "428891868"
uid: cdd70d0c-0b6d-482c-a8e0-25c017e4e0c7
status:
currentState:
dns-resolver:
  config:
    search: null
    server: null
  running:
    search: []
    server: []
interfaces:
- accept-all-mac-addresses: false
  ethtool:
    feature:
      esp-hw-offload: false
      esp-tx-csum-hw-offload: false
      highdma: true
      rx-gro: true
      rx-gro-list: false
      rx-lro: false
      rx-udp-gro-forwarding: false
      rx-vlan-filter: true
      rx-vlan-hw-parse: true
      tx-checksum-ip-generic: true
      tx-esp-segmentation: false
      tx-generic-segmentation: true
      tx-gre-csum-segmentation: true
      tx-gre-segmentation: true
      tx-gso-list: false
      tx-ipxip4-segmentation: true
      tx-ipxip6-segmentation: true
      tx-nocache-copy: false
      tx-scatter-gather-fraglist: false
      tx-sctp-segmentation: false
      tx-tcp-ecn-segmentation: true
      tx-tcp-mangleid-segmentation: true
      tx-tcp-segmentation: true
      tx-tcp6-segmentation: true
      tx-udp-segmentation: true
      tx-udp_tnl-csum-segmentation: true
      tx-udp_tnl-segmentation: true
  ipv4:
    address:
    - ip: 198.19.0.17
      prefix-length: 22
    enabled: true
  ipv6:
    address:
    - ip: fe80::849c:86ff:feed:378b
      prefix-length: 64
    enabled: true
  link-aggregation:
    mode: 802.3ad
    options:
      ad_actor_sys_prio: 65535
      ad_actor_system: "00:00:00:00:00:00"
      ad_select: stable
      ad_user_port_key: 0
      all_slaves_active: dropped
      arp_all_targets: any
      arp_interval: 0
      arp_validate: none
      downdelay: 0
      lacp_rate: fast
      lp_interval: 1
      miimon: 1000
      min_links: 0
      primary_reselect: always
      updelay: 0
      use_carrier: true
      xmit_hash_policy: layer3+4
    port:
    - enp175s0f0np0
    - enp175s0f1np1
  mac-address: 86:9C:86:ED:37:8B
  max-mtu: 65535
  min-mtu: 68
  mptcp:
    address-flags: []
  mtu: 1500
  name: bond1
  state: ignore
  type: bond
- accept-all-mac-addresses: false
  ethtool:
    feature:
      rx-checksum: true
      rx-gro: true
      rx-gro-list: false
      rx-udp-gro-forwarding: false
      tx-checksum-ip-generic: true
      tx-generic-segmentation: true
      tx-gso-list: true
      tx-nocache-copy: false
      tx-scatter-gather-fraglist: true
      tx-sctp-segmentation: true
      tx-tcp-ecn-segmentation: true
      tx-tcp-mangleid-segmentation: true
      tx-tcp-segmentation: true
      tx-tcp6-segmentation: true
      tx-udp-segmentation: true
  ipv4:
    enabled: false
  ipv6:
    address:
    - ip: fe80::2c9c:bff:fe94:9164
      prefix-length: 64
    enabled: true
  mac-address: 2E:9C:0B:94:91:64
  max-mtu: 65535
  min-mtu: 68
  mtu: 1500
  name: cilium_vxlan
  state: ignore
  type: vxlan
  vxlan:
    destination-port: 8472
    id: 0
    learning: false
- accept-all-mac-addresses: false
  bridge:
    options:
      group-addr: 01:80:C2:00:00:00
      group-forward-mask: 0
      group-fwd-mask: 0
      hash-max: 4096
      mac-ageing-time: 300
      multicast-last-member-count: 2
      multicast-last-member-interval: 100
      multicast-membership-interval: 26000
      multicast-querier: false
      multicast-querier-interval: 25500
      multicast-query-interval: 12500
      multicast-query-response-interval: 1000
      multicast-query-use-ifaddr: false
      multicast-router: auto
      multicast-snooping: true
      multicast-startup-query-count: 2
      multicast-startup-query-interval: 3124
      stp:
        enabled: false
        forward-delay: 15
        hello-time: 2
        max-age: 20
        priority: 32768
      vlan-default-pvid: 1
      vlan-protocol: 802.1q
    port: []
  ethtool:
    feature:
      highdma: true
      rx-gro: true
      rx-gro-list: false
      rx-udp-gro-forwarding: false
      tx-checksum-ip-generic: true
      tx-esp-segmentation: true
      tx-fcoe-segmentation: true
      tx-generic-segmentation: true
      tx-gre-csum-segmentation: true
      tx-gre-segmentation: true
      tx-gso-list: true
      tx-gso-partial: true
      tx-gso-robust: true
      tx-ipxip4-segmentation: true
      tx-ipxip6-segmentation: true
      tx-nocache-copy: false
      tx-scatter-gather-fraglist: true
      tx-sctp-segmentation: true
      tx-tcp-ecn-segmentation: true
      tx-tcp-mangleid-segmentation: true
      tx-tcp-segmentation: true
      tx-tcp6-segmentation: true
      tx-tunnel-remcsum-segmentation: true
      tx-udp-segmentation: true
      tx-udp_tnl-csum-segmentation: true
      tx-udp_tnl-segmentation: true
      tx-vlan-hw-insert: true
      tx-vlan-stag-hw-insert: true
  ipv4:
    address:
    - ip: 172.17.0.1
      prefix-length: 16
    enabled: true
  ipv6:
    enabled: false
  mac-address: 02:42:F3:B6:47:0A
  max-mtu: 65535
  min-mtu: 68
  mptcp:
    address-flags: []
  mtu: 1500
  name: docker0
  state: ignore
  type: linux-bridge
- accept-all-mac-addresses: false
  ethernet:
    auto-negotiation: true
    duplex: full
    speed: 10000
    sr-iov:
      total-vfs: 0
      vfs: []
  ethtool:
    coalesce:
      rx-usecs: 1
      tx-usecs: 0
    feature:
      esp-hw-offload: true
      esp-tx-csum-hw-offload: true
      hw-tc-offload: false
      l2-fwd-offload: false
      rx-all: false
      rx-checksum: true
      rx-gro: true
      rx-gro-list: false
      rx-hashing: true
      rx-lro: false
      rx-ntuple-filter: false
      rx-udp-gro-forwarding: false
      rx-vlan-filter: true
      rx-vlan-hw-parse: true
      tx-checksum-ip-generic: true
      tx-checksum-sctp: true
      tx-esp-segmentation: true
      tx-generic-segmentation: true
      tx-gre-csum-segmentation: true
      tx-gre-segmentation: true
      tx-gso-partial: true
      tx-ipxip4-segmentation: true
      tx-ipxip6-segmentation: true
      tx-nocache-copy: false
      tx-tcp-mangleid-segmentation: false
      tx-tcp-segmentation: true
      tx-tcp6-segmentation: true
      tx-udp-segmentation: true
      tx-udp_tnl-csum-segmentation: true
      tx-udp_tnl-segmentation: true
      tx-vlan-hw-insert: true
    pause:
      autoneg: true
      rx: true
      tx: true
    ring:
      rx: 512
      rx-max: 4096
      tx: 512
      tx-max: 4096
  ipv4:
    address:
    - ip: 198.18.0.17
      prefix-length: 25
    enabled: true
  ipv6:
    address:
    - ip: fe80::3eec:efff:fe60:532e
      prefix-length: 64
    enabled: true
  mac-address: 3C:EC:EF:60:53:2E
  max-mtu: 9710
  min-mtu: 68
  mptcp:
    address-flags: []
  mtu: 1500
  name: eno1
  state: ignore
  type: ethernet
- accept-all-mac-addresses: false
  ethernet:
    auto-negotiation: true
    sr-iov:
      total-vfs: 0
      vfs: []
  ethtool:
    coalesce:
      rx-usecs: 1
      tx-usecs: 0
    feature:
      esp-hw-offload: true
      esp-tx-csum-hw-offload: true
      hw-tc-offload: false
      l2-fwd-offload: false
      rx-all: false
      rx-checksum: true
      rx-gro: true
      rx-gro-list: false
      rx-hashing: true
      rx-lro: false
      rx-ntuple-filter: false
      rx-udp-gro-forwarding: false
      rx-vlan-filter: true
      rx-vlan-hw-parse: true
      tx-checksum-ip-generic: true
      tx-checksum-sctp: true
      tx-esp-segmentation: true
      tx-generic-segmentation: true
      tx-gre-csum-segmentation: true
      tx-gre-segmentation: true
      tx-gso-partial: true
      tx-ipxip4-segmentation: true
      tx-ipxip6-segmentation: true
      tx-nocache-copy: false
      tx-tcp-mangleid-segmentation: false
      tx-tcp-segmentation: true
      tx-tcp6-segmentation: true
      tx-udp-segmentation: true
      tx-udp_tnl-csum-segmentation: true
      tx-udp_tnl-segmentation: true
      tx-vlan-hw-insert: true
    pause:
      autoneg: true
      rx: true
      tx: true
    ring:
      rx: 512
      rx-max: 4096
      tx: 512
      tx-max: 4096
  ipv4:
    enabled: false
  ipv6:
    enabled: false
  mac-address: 3C:EC:EF:60:53:2F
  max-mtu: 9710
  min-mtu: 68
  mtu: 1500
  name: eno2
  state: ignore
  type: ethernet
- accept-all-mac-addresses: false
  controller: bond1
  ethernet:
    auto-negotiation: true
    duplex: full
    speed: 100000
    sr-iov:
      total-vfs: 0
      vfs: []
  ethtool:
    coalesce:
      adaptive-rx: true
      adaptive-tx: true
      rx-frames: 128
      rx-usecs: 8
      tx-frames: 128
      tx-usecs: 8
    feature:
      hw-tc-offload: true
      rx-all: false
      rx-checksum: true
      rx-fcs: false
      rx-gro: true
      rx-gro-list: false
      rx-hashing: true
      rx-lro: false
      rx-ntuple-filter: false
      rx-udp-gro-forwarding: false
      rx-udp_tunnel-port-offload: true
      rx-vlan-filter: true
      rx-vlan-hw-parse: true
      tx-checksum-ip-generic: true
      tx-generic-segmentation: true
      tx-gre-csum-segmentation: true
      tx-gre-segmentation: true
      tx-gso-partial: true
      tx-ipxip4-segmentation: true
      tx-ipxip6-segmentation: true
      tx-nocache-copy: false
      tx-tcp-mangleid-segmentation: false
      tx-tcp-segmentation: true
      tx-tcp6-segmentation: true
      tx-udp-segmentation: true
      tx-udp_tnl-csum-segmentation: true
      tx-udp_tnl-segmentation: true
      tx-vlan-hw-insert: true
      tx-vlan-stag-hw-insert: true
    pause:
      autoneg: false
      rx: true
      tx: true
    ring:
      rx: 1024
      rx-max: 8192
      tx: 1024
      tx-max: 8192
  ipv4:
    enabled: false
  ipv6:
    enabled: false
  mac-address: 86:9C:86:ED:37:8B
  max-mtu: 9978
  min-mtu: 68
  mtu: 1500
  name: enp175s0f0np0
  state: ignore
  type: ethernet
- accept-all-mac-addresses: false
  controller: bond1
  ethernet:
    auto-negotiation: true
    duplex: full
    speed: 100000
    sr-iov:
      total-vfs: 0
      vfs: []
  ethtool:
    coalesce:
      adaptive-rx: true
      adaptive-tx: true
      rx-frames: 128
      rx-usecs: 8
      tx-frames: 128
      tx-usecs: 8
    feature:
      hw-tc-offload: true
      rx-all: false
      rx-checksum: true
      rx-fcs: false
      rx-gro: true
      rx-gro-list: false
      rx-hashing: true
      rx-lro: false
      rx-ntuple-filter: false
      rx-udp-gro-forwarding: false
      rx-udp_tunnel-port-offload: true
      rx-vlan-filter: true
      rx-vlan-hw-parse: true
      tx-checksum-ip-generic: true
      tx-generic-segmentation: true
      tx-gre-csum-segmentation: true
      tx-gre-segmentation: true
      tx-gso-partial: true
      tx-ipxip4-segmentation: true
      tx-ipxip6-segmentation: true
      tx-nocache-copy: false
      tx-tcp-mangleid-segmentation: false
      tx-tcp-segmentation: true
      tx-tcp6-segmentation: true
      tx-udp-segmentation: true
      tx-udp_tnl-csum-segmentation: true
      tx-udp_tnl-segmentation: true
      tx-vlan-hw-insert: true
      tx-vlan-stag-hw-insert: true
    pause:
      autoneg: false
      rx: true
      tx: true
    ring:
      rx: 1024
      rx-max: 8192
      tx: 1024
      tx-max: 8192
  ipv4:
    enabled: false
  ipv6:
    enabled: false
  mac-address: 86:9C:86:ED:37:8B
  max-mtu: 9978
  min-mtu: 68
  mtu: 1500
  name: enp175s0f1np1
  state: ignore
  type: ethernet
- accept-all-mac-addresses: false
  ethtool:
    feature:
      rx-gro: true
      rx-gro-list: false
      rx-udp-gro-forwarding: false
      tx-generic-segmentation: true
      tx-gso-list: true
      tx-sctp-segmentation: true
      tx-tcp-ecn-segmentation: true
      tx-tcp-mangleid-segmentation: true
      tx-tcp-segmentation: true
      tx-tcp6-segmentation: true
      tx-udp-segmentation: true
  ipv4:
    address:
    - ip: 127.0.0.1
      prefix-length: 8
    enabled: true
  ipv6:
    address:
    - ip: ::1
      prefix-length: 128
    enabled: true
  mac-address: "00:00:00:00:00:00"
  mtu: 65536
  name: lo
  state: ignore
  type: loopback
routes:
  config:
  - destination: 10.203.92.0/23
    next-hop-address: 198.19.0.30
    next-hop-interface: bond1
    table-id: 254
  - destination: 0.0.0.0/0
    next-hop-address: 198.18.0.1
    next-hop-interface: eno1
    table-id: 254
  running:
  - destination: 10.203.92.0/23
    next-hop-address: 198.19.0.30
    next-hop-interface: bond1
    table-id: 254
  - destination: 0.0.0.0/0
    next-hop-address: 198.18.0.1
    next-hop-interface: eno1
    table-id: 254
handlerNmstateVersion: nmstatectl 2.2.13
hostNetworkManagerVersion: 1.36.6
lastSuccessfulUpdateTime: "2024-05-02T16:12:49Z"

Problematic NodeNetworkConfigurationPolicy: Any but see above yaml, here is the actual output from the nncp

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
annotations:
nmstate.io/webhook-mutating-timestamp: "1716582405477407436"
creationTimestamp: "2024-05-24T20:26:45Z"
generation: 1
name: svc-uc-v108-dal4-qz4-sr7-rk047-s04
resourceVersion: "525584546"
uid: dae7d032-d339-4bbd-b7d9-ac1d6d42e2a6
spec:
desiredState:
interfaces:
- ipv4:
    dhcp: false
    enabled: false
  mac-address: 02:10:2c:2c:7e:52
  name: svc-uc-v108
  state: up
  type: vlan
  vlan:
    base-iface: bond1
    id: 108
nodeSelector:
kubernetes.io/hostname: dal4-qz4-sr7-rk047-s04
status:
conditions:
- lastHeartbeatTime: "2024-05-28T16:03:32Z"
lastTransitionTime: "2024-05-28T16:03:32Z"
message: 1/1 nodes successfully configured
reason: SuccessfullyConfigured
status: "True"
type: Available
- lastHeartbeatTime: "2024-05-28T16:03:32Z"
lastTransitionTime: "2024-05-28T16:03:32Z"
reason: SuccessfullyConfigured
status: "False"
type: Degraded
- lastHeartbeatTime: "2024-05-28T16:03:32Z"
lastTransitionTime: "2024-05-28T16:03:32Z"
reason: ConfigurationProgressing
status: "False"
type: Progressing
lastUnavailableNodeCountUpdate: "2024-05-28T16:03:32Z"

kubernetes-nmstate image (use kubectl get pods --all-namespaces -l app=kubernetes-nmstate -o jsonpath='{.items[0].spec.containers[0].image}'):
```
kubernetes-nmstate-handler:v0.80.0
```
NetworkManager version (use nmcli --version)
```
nmcli tool, version 1.36.6
```

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.10", GitCommit:"7e54d50d3012cf3389e43b096ba35300f36e0817", GitTreeState:"clean", BuildDate:"2022-08-17T18:32:54Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.10", GitCommit:"7e54d50d3012cf3389e43b096ba35300f36e0817", GitTreeState:"clean", BuildDate:"2022-08-17T18:26:59Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

OS (e.g. from /etc/os-release):

PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Kernel

5.15.0-1020

Others: nmstate pod log:

kubectl logs nmstate-handler-9zkz6 -n nmstate

{"level":"info","ts":"2024-05-28T15:48:59.214Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- bridge:\n    options:\n      stp:\n        enabled: false\n    port:\n    - name: usb-int-v53\n      vlan:\n        mode: trunk\n        trunk-tags:\n        - id-range:\n            max: 4094\n            min: 2\n  ipv4:\n    dhcp: false\n    enabled: false\n  mac-address: 02:20:34:a8:07:57\n  name: usb-int-br53\n  state: up\n  type: linux-bridge\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[{Type:Progressing Status:True Reason:ConfigurationProgressing Message:Applying desired state LastHeartbeatTime:2024-05-28 15:48:59.21462073 +0000 UTC m=+3231.903715825 LastTransitionTime:2024-05-28 15:48:59.21462073 +0000 UTC m=+3231.903715825} {Type:Failing Status:Unknown Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:48:59.214621417 +0000 UTC m=+3231.903716506 LastTransitionTime:2024-05-28 15:48:59.214621417 +0000 UTC m=+3231.903716506} {Type:Available Status:Unknown Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:48:59.214623185 +0000 UTC m=+3231.903718275 LastTransitionTime:2024-05-28 15:48:59.214623185 +0000 UTC m=+3231.903718275} {Type:Pending Status:False Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:48:59.214623966 +0000 UTC m=+3231.903719056 LastTransitionTime:2024-05-28 15:48:59.214623966 +0000 UTC m=+3231.903719056} {Type:Aborted Status:False Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:48:59.214624079 +0000 UTC m=+3231.903719168 LastTransitionTime:2024-05-28 15:48:59.214624079 +0000 UTC m=+3231.903719168}]}","enactment":"dal4-qz4-sr7-rk047-s04.usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:48:59.222Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"dal4-qz4-sr7-rk047-s04.usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:48:59.306Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 0, unknown: 1}, progressing: {true: 1, false: 0, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 0, false: 0, unknown: 1}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:48:59.306Z","logger":"policyconditions","msg":"SetPolicyProgressing"}
{"level":"info","ts":"2024-05-28T15:49:03.879Z","logger":"probe","msg":"Running 'ping' probe"}
{"level":"info","ts":"2024-05-28T15:49:04.581Z","logger":"probe","msg":"Running 'dns' probe"}
{"level":"info","ts":"2024-05-28T15:49:05.185Z","logger":"probe","msg":"Running 'api-server' probe"}
I0528 15:49:06.236434       1 request.go:601] Waited for 1.046322626s due to client-side throttling, not priority and fairness, request: GET:https://172.16.0.1:443/apis/objectbucket.io/v1alpha1?timeout=32s
{"level":"info","ts":"2024-05-28T15:49:07.048Z","logger":"probe","msg":"Running 'node-readiness' probe"}
{"level":"info","ts":"2024-05-28T15:49:07.063Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"nmstate","nodenetworkconfigurationpolicy":"/usb-int-br53-dal4-qz4-sr7-rk047-s04","output":"setOutput: dns-resolver: {}\nroute-rules: {}\nroutes: {}\ninterfaces:\n- name: usb-int-br53\n  type: linux-bridge\n  state: up\n  mac-address: 02:20:34:a8:07:57\n  ipv4:\n    enabled: false\n    dhcp: false\n  bridge:\n    options:\n      stp:\n        enabled: false\n    port:\n    - name: usb-int-v53\n      vlan:\n        mode: trunk\n        trunk-tags:\n        - id-range:\n            min: 2\n            max: 4094\novs-db: {}\novn: {}\n\n \n"}
{"level":"info","ts":"2024-05-28T15:49:07.063Z","logger":"enactmentconditions","msg":"NotifySuccess","enactment":"dal4-qz4-sr7-rk047-s04.usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:07.067Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- bridge:\n    options:\n      stp:\n        enabled: false\n    port:\n    - name: usb-int-v53\n      vlan:\n        mode: trunk\n        trunk-tags:\n        - id-range:\n            max: 4094\n            min: 2\n  ipv4:\n    dhcp: false\n    enabled: false\n  mac-address: 02:20:34:a8:07:57\n  name: usb-int-br53\n  state: up\n  type: linux-bridge\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[{Type:Progressing Status:False Reason:SuccessfullyConfigured Message: LastHeartbeatTime:2024-05-28 15:49:07.067145399 +0000 UTC m=+3239.756240528 LastTransitionTime:2024-05-28 15:49:07.067145399 +0000 UTC m=+3239.756240528} {Type:Failing Status:False Reason:SuccessfullyConfigured Message: LastHeartbeatTime:2024-05-28 15:49:07.067145008 +0000 UTC m=+3239.756240137 LastTransitionTime:2024-05-28 15:49:07.067145008 +0000 UTC m=+3239.756240137} {Type:Available Status:True Reason:SuccessfullyConfigured Message:successfully reconciled LastHeartbeatTime:2024-05-28 15:49:07.067144298 +0000 UTC m=+3239.756239442 LastTransitionTime:2024-05-28 15:49:07.067144298 +0000 UTC m=+3239.756239442} {Type:Pending Status:False Reason:SuccessfullyConfigured Message: LastHeartbeatTime:2024-05-28 15:49:07.06714591 +0000 UTC m=+3239.756241040 LastTransitionTime:2024-05-28 15:48:59 +0000 UTC} {Type:Aborted Status:False Reason:SuccessfullyConfigured Message: LastHeartbeatTime:2024-05-28 15:49:07.067146601 +0000 UTC m=+3239.756241728 LastTransitionTime:2024-05-28 15:48:59 +0000 UTC}]}","enactment":"dal4-qz4-sr7-rk047-s04.usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:07.082Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"dal4-qz4-sr7-rk047-s04.usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:07.082Z","logger":"controllers.NodeNetworkConfigurationPolicy.forceNNSRefresh","msg":"forcing NodeNetworkState refresh after NNCP applied","node":"dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:07.319Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-br53-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:07.320Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.335Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/cos-dp-br53-dal4-qz4-sr7-rk047-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.421Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"cos-dp-br53-dal4-qz4-sr7-rk047-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.421Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.435Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-uc-v108-dal4-qz4-sr7-rk046-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.518Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-v108-dal4-qz4-sr7-rk046-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.518Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.533Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/default-br90-dal4-qz4-sr7-rk049-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.620Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"default-br90-dal4-qz4-sr7-rk049-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.620Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.635Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-uc-v39-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.721Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-v39-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.721Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.738Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/usb-int-br106-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.819Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"usb-int-br106-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.819Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.835Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-uc-br37-dal4-qz4-sr7-rk047-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.920Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-br37-dal4-qz4-sr7-rk047-s06"}
{"level":"info","ts":"2024-05-28T15:49:07.920Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:07.935Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-uc-v37-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.050Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-v37-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.050Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:08.069Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-vsi-br107-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.157Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-vsi-br107-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.157Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:08.173Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-uc-v108-dal4-qz4-sr7-rk049-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.263Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-v108-dal4-qz4-sr7-rk049-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.263Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:08.279Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/default-br90-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.366Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"default-br90-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.366Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:08.382Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-uc-br43-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.466Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-br43-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.466Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:08.482Z","logger":"controllers.NodeNetworkConfigurationPolicy","msg":"Policy node selectors does not match node, removing previous enactment if any","nodenetworkconfigurationpolicy":"/svc-vsi-v107-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.567Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 1, unknown: 0}, progressing: {true: 0, false: 1, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 1, false: 0, unknown: 0}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-vsi-v107-dal4-qz4-sr7-rk048-s06"}
{"level":"info","ts":"2024-05-28T15:49:08.567Z","logger":"policyconditions","msg":"SetPolicySuccess"}
{"level":"info","ts":"2024-05-28T15:49:08.587Z","logger":"enactmentconditions","msg":"Reset","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:08.590Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- ipv4:\n    dhcp: false\n    enabled: false\n  mac-address: 02:10:2c:2c:7e:52\n  name: svc-uc-v108\n  state: up\n  type: vlan\n  vlan:\n    base-iface: bond1\n    id: 108\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[]}","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:08.598Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.358Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- ipv4:\n    dhcp: false\n    enabled: false\n  mac-address: 02:10:2c:2c:7e:52\n  name: svc-uc-v108\n  state: up\n  type: vlan\n  vlan:\n    base-iface: bond1\n    id: 108\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[]}","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.367Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.474Z","logger":"enactmentconditions","msg":"NotifyProgressing","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.477Z","logger":"enactmentstatus","msg":"status: {DesiredState:interfaces:\n- ipv4:\n    dhcp: false\n    enabled: false\n  mac-address: 02:10:2c:2c:7e:52\n  name: svc-uc-v108\n  state: up\n  type: vlan\n  vlan:\n    base-iface: bond1\n    id: 108\n DesiredStateMetaInfo:{Version: TimeStamp:0001-01-01 00:00:00 +0000 UTC} CapturedStates:map[] PolicyGeneration:1 Conditions:[{Type:Progressing Status:True Reason:ConfigurationProgressing Message:Applying desired state LastHeartbeatTime:2024-05-28 15:49:09.477075435 +0000 UTC m=+3242.166170578 LastTransitionTime:2024-05-28 15:49:09.477075435 +0000 UTC m=+3242.166170578} {Type:Failing Status:Unknown Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:49:09.477076702 +0000 UTC m=+3242.166171846 LastTransitionTime:2024-05-28 15:49:09.477076702 +0000 UTC m=+3242.166171846} {Type:Available Status:Unknown Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:49:09.477077817 +0000 UTC m=+3242.166172945 LastTransitionTime:2024-05-28 15:49:09.477077817 +0000 UTC m=+3242.166172945} {Type:Pending Status:False Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:49:09.477079183 +0000 UTC m=+3242.166174311 LastTransitionTime:2024-05-28 15:49:09.477079183 +0000 UTC m=+3242.166174311} {Type:Aborted Status:False Reason:ConfigurationProgressing Message: LastHeartbeatTime:2024-05-28 15:49:09.477079649 +0000 UTC m=+3242.166174778 LastTransitionTime:2024-05-28 15:49:09.477079649 +0000 UTC m=+3242.166174778}]}","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.485Z","logger":"enactmentstatus","msg":"enactment updated at the node: true","enactment":"dal4-qz4-sr7-rk047-s04.svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.565Z","logger":"policyconditions","msg":"numberOfNmstateMatchingNodes: 1, enactments count: {failed: {true: 0, false: 0, unknown: 1}, progressing: {true: 1, false: 0, unknown: 0}, pending: {true: 0, false: 1, unknown: 0}, available: {true: 0, false: 0, unknown: 1}, aborted: {true: 0, false: 1, unknown: 0}}","policy":"svc-uc-v108-dal4-qz4-sr7-rk047-s04"}
{"level":"info","ts":"2024-05-28T15:49:09.565Z","logger":"policyconditions","msg":"SetPolicyProgressing"}

I killed the pod and a new one properly started up. Tried the apply for another vlan interface and it still took > 60s to apply

I attached the full logs of nmstate-handler-9zkz6 logs--svc-uc-v108-apply.txt

qinqon commented 1 month ago

kubernetes-nmstate do some probes before commit the changes to ensure that it do not break the cluster, that's where the time goes on.

k8scoder192 commented 1 month ago

@qinqon did you look at the full logs? It's NOT due to the probes. The probes come back extremely quick. Something else is wrong.

Edit I also noticed this is only happening on 1 worker node. If I look at the logs of the other nmstate-handler pods, they aren't constantly reporting "enactment updated at the node: true". Something is triggering this pod to constantly perform state changes like below for all the interfaces

journalctl -S today -u NetworkManager.service -f (output on the node I see the problem on)

May 31 18:12:37 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179157.9313] device (usb-int-v87): state change: deactivating -> disconnected (reason 'new-activation', sys-iface-state: 'managed')
May 31 18:12:37 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179157.9321] device (usb-int-v87): Activation: starting connection 'usb-int-v87' (74ec521d-7a67-4ade-82ae-19f6892e645c)
May 31 18:12:37 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179157.9326] device (usb-int-v87): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
May 31 18:12:37 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179157.9342] device (usb-int-v87): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
May 31 18:12:37 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179157.9371] device (usb-int-v87): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
May 31 18:12:38 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179158.0251] device (usb-int-br87): attached bridge port usb-int-v87
May 31 18:12:38 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179158.0251] device (usb-int-v87): Activation: connection 'usb-int-v87' enslaved, continuing activation
May 31 18:12:38 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179158.0256] device (usb-int-v87): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
May 31 18:12:38 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179158.0281] device (usb-int-v87): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
May 31 18:12:38 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179158.0283] device (usb-int-v87): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
May 31 18:12:38 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179158.0287] device (usb-int-v87): Activation: successful, device activated.
May 31 18:12:42 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179162.9356] checkpoint[0x55bcd4c8d450]: destroy /org/freedesktop/NetworkManager/Checkpoint/292
May 31 18:12:42 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179162.9390] audit: op="checkpoint-destroy" arg="/org/freedesktop/NetworkManager/Checkpoint/292" pid=2292492 uid=0 result="success"
May 31 18:12:46 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179166.5941] audit: op="checkpoint-create" arg="/org/freedesktop/NetworkManager/Checkpoint/293" pid=2292893 uid=0 result="success"
May 31 18:12:46 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179166.5968] audit: op="checkpoint-adjust-rollback-timeout" arg="/org/freedesktop/NetworkManager/Checkpoint/293" pid=2292893 uid=0 result="success"
May 31 18:12:47 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179167.3864] audit: op="connection-update" uuid="c504a0fa-730b-4182-8b98-f404ab44c196" name="usb-int-br67" args="connection.timestamp" pid=2292893 uid=0 result="success"
May 31 18:12:47 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179167.5521] audit: op="device-reapply" interface="usb-int-br67" ifindex=177998 pid=2292893 uid=0 result="success"
May 31 18:12:47 dal4-qz4-sr7-rk047-s04 NetworkManager[2025380]: <info>  [1717179167.5563] audit: op="checkpoint-adjust-rollback-timeout" arg="/org/freedesktop/NetworkManager/Checkpoint/293" pid=2292893 uid=0 result="succe

When I look at NNCP for that node, they are constantly going from "Available SuccessfullyConfigured" to blank status then back to "Available SuccessfullyConfigured". When that happens I see the above journalctl output for that particular interface.

I really need to figure out why k8s nmstate is behaving this way. This node's nmstate-handler pod restarted this week. The other node's pods have been up for 200+ days. I'm afraid to bound the other node's pods because of this odd behavior.

nmstate / kubernetes-nmstate

NNCP apply takes very long time to apply; nmstatectl does not #1242

nmstate / kubernetes-nmstate

NNCP apply takes *very long* time to apply; nmstatectl does not #1242

NNCP apply takes very long time to apply; nmstatectl does not #1242