kmesh-net / kmesh

High Performance ServiceMesh Data Plane Based on Programmable Kernel
https://kmesh.net
Apache License 2.0
362 stars 46 forks source link

kmesh panic when change bpfloglevel #477

Closed lec-bit closed 9 hours ago

lec-bit commented 2 days ago

What happened: When kmesh is started in ADS mode, the EBPF log level cannot be changed. panic

kubectl exec -it -n kmesh-system kmesh-q4f5d -- curl -s localhost:15200/debug/bpfLogLevel/3
command terminated with exit code 52

kubectl logs -f -n kmesh-system kmesh-q4f5d

2024-07-03T02:36:33.168057Z info    http: panic serving 127.0.0.1:52122: runtime error: invalid memory address or nil pointer dereference
goroutine 78 [running]:
net/http.(*conn).serve.func1()
    /usr/local/go/src/net/http/server.go:1898 +0xbe
panic({0x28de820?, 0x4e09b80?})
    /usr/local/go/src/runtime/panic.go:770 +0x132
kmesh.net/kmesh/pkg/status.(*Server).bpfLogLevel(0xc0012aa030, {0x32e5f30, 0xc000d96000}, 0xc00074bb00)
    /kmesh/pkg/status/status_server.go:298 +0xe6
net/http.HandlerFunc.ServeHTTP(0xc00129b520?, {0x32e5f30?, 0xc000d96000?}, 0x78c0fa?)
    /usr/local/go/src/net/http/server.go:2166 +0x29
net/http.(*ServeMux).ServeHTTP(0x474b79?, {0x32e5f30, 0xc000d96000}, 0xc00074bb00)
    /usr/local/go/src/net/http/server.go:2683 +0x1ad
net/http.serverHandler.ServeHTTP({0xc0009fe420?}, {0x32e5f30?, 0xc000d96000?}, 0x6?)
    /usr/local/go/src/net/http/server.go:3137 +0x8e
net/http.(*conn).serve(0xc00088c000, {0x32f72c0, 0xc0013062d0})
    /usr/local/go/src/net/http/server.go:2039 +0x5e8
created by net/http.(*Server).Serve in goroutine 120
    /usr/local/go/src/net/http/server.go:3285 +0x4b4
time="2024-07-03T02:54:58Z" level=info msg="grpc reconnect succeed" subsys=controller

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

hzxuzhonghu commented 2 days ago

cc @bfforever

bfforever commented 2 days ago

https://github.com/kmesh-net/kmesh/blob/bb1990048f1a58fd1a253b2bcbdf1ad5df287a1b/pkg/status/status_server.go#L298 In ads mode, bpfWorkloadObj not load, cause to nil pointer, I would fix recently.

hzxuzhonghu commented 2 days ago

This is urgent