kmesh-net / kmesh

High Performance ServiceMesh Data Plane Based on Programmable Kernel
https://kmesh.net
Apache License 2.0
424 stars 59 forks source link

Add support for profiling kmesh controlplane #199

Closed hzxuzhonghu closed 2 months ago

hzxuzhonghu commented 6 months ago

What would you like to be added:

Recently we have figured out some issues on perf, stability, i have a rough though, we can add some metrics to help profiling kmesh-daemon. Like

  1. Basic CPU/Memory usage
  2. xDS related metrics like xds response handling time., bpf map operations

Why is this needed:

Currently, kmesh-daemon is like a black box, we can see some xds configurations by dumping the bpf map. But we couldn't know whether it is consistent with xds response, which flow is slow, which object is memory costing, etc. This is super critical for production use.

hzxuzhonghu commented 4 months ago

we can use net/http/pprof package

Okabe-Rintarou-0 commented 4 months ago

xDS related metrics like xds response handling time., bpf map operations

Shall we use Prometheus to expose custom metrics?

hzxuzhonghu commented 4 months ago

we can split some tasks, metrics can be one

Okabe-Rintarou-0 commented 4 months ago

pprof will start a http server, shall we make the port configurable or use a default one?

Okabe-Rintarou-0 commented 4 months ago

seems that pprof also supports custom profile

Okabe-Rintarou-0 commented 4 months ago

/assign

hzxuzhonghu commented 2 months ago

done