nmstate / kubernetes-nmstate

Declarative node network configuration driven through Kubernetes API.
GNU General Public License v2.0
180 stars 89 forks source link

nmstate-handler crashed #1249

Closed fanzy618 closed 2 weeks ago

fanzy618 commented 3 months ago

What happened: nmstate-handler's Pods went into CrashLoopBackOff status. The logs of pod is

{"level":"info","ts":"2024-06-28T09:02:25.172Z","logger":"setup","msg":"Try to take exclusive lock on file: /var/k8s_nmstate/handler_lock"} {"level":"info","ts":"2024-06-28T09:02:25.172Z","logger":"setup","msg":"Successfully took nmstate exclusive lock"} {"level":"info","ts":"2024-06-28T09:02:25.172Z","logger":"setup","msg":"Creating manager"} {"level":"info","ts":"2024-06-28T09:02:25.225Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8089"} {"level":"info","ts":"2024-06-28T09:02:25.225Z","logger":"setup","msg":"Creating Node controller"} {"level":"info","ts":"2024-06-28T09:02:25.225Z","logger":"setup","msg":"Creating non cached client"} {"level":"info","ts":"2024-06-28T09:02:25.225Z","logger":"setup","msg":"Creating NodeNetworkConfigurationPolicy controller"} {"level":"info","ts":"2024-06-28T09:02:25.225Z","logger":"setup","msg":"Creating NodeNetworkConfigurationEnactment controller"} {"level":"info","ts":"2024-06-28T09:02:25.225Z","logger":"setup","msg":"Checking availability of nmstatectl"} {"level":"error","ts":"2024-06-28T09:02:25.271Z","logger":"setup","msg":"failed checking nmstatectl health","error":"failed to execute nmstatectl show: 'exit status 1' '' '[2024-06-28T09:02:25Z INFO nmstatectl] Nmstate version: 2.2.33\n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type geneve\n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type geneve\n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type geneve\n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::ovsdb::show] Unknown OVS interface type \n[2024-06-28T09:02:25Z WARN nmstate::nm::nm_dbus::error] Unknown DBUS error MethodError(\"org.freedesktop.DBus.Error.ServiceUnknown\", Some(\"The name org.freedesktop.NetworkManager was not provided by any .service files\"), Msg { type: Error, sender: \"org.freedesktop.DBus\", reply-serial: 2, body: Signature: [\n \ts (115),\n ] })\nNmstateError: Bug: DbusConnectionError: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.NetworkManager was not provided by any .service files\n'","stacktrace":"main.checkNmstateIsWorking\n\t/cmd/handler/main.go:267\nmain.mainHandler\n\t/cmd/handler/main.go:167\nmain.main\n\t/cmd/handler/main.go:89\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}

What you expected to happen: Is there any way that can let nmstate ignore the devices which it doesn't know? I don't want to any ops on those OVS device.

How to reproduce it (as minimally and precisely as possible): Run kuberenetes-nmstate on a cluster that using kube-ovn cni

Anything else we need to know?: Execute command "nmstatectl show" on the node gets the same error.

Environment:

qinqon commented 3 months ago

This failure is related to NetworkManager not running at nodes or not exposing the proper dbus socket

You need to have networkmanager running at nodes also with NetworkManger-ovs extension if you want to configure ovs with k-nmstate.

Bug: DbusConnectionError: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.NetworkManager was not provided by any .service