search

performancecopilot / grafana-pcp

Performance Co-Pilot Grafana Plugin
https://grafana-pcp.readthedocs.io
Apache License 2.0
115 stars 19 forks source link

hotproc metrics displayed in grafana without pid & process name #203

Open mallilinux1980 opened 1 month ago

mallilinux1980 commented 1 month ago

i've enabled PCP hotproc metrics using hotproc.conf (cpuburn > 0.10) & also seup sasl based authentication and installed redis as well as grafana everything on a test server.

the real issue I've observed for most metrics example "hotproc.predicate.cpuburn" shows up instances (40digit alpha-numeric number) who're more than 10% CPU usage as per predicate but no PID and PROC name associated with it. Sometime takes its own time and sometimes it doesn't at all.

my env:

so am I missing anything here?

natoscott commented 1 month ago

I've setup hotproc metrics going into Valkey now (Fedora40, pcp-6.3.1) and everything seems to be fine from the PCP end. There may be some effort needed on the Grafana end, not sure, but this is what I see using just PCP tools (i.e. pmseries command line, which talks directly to Valkey/Redis):

nathans@fedora:~/git/pcp$ pmseries hotproc.psinfo.rss
014f2aaf4d97f588a00c52ea3fcc672fbd423ecb
nathans@fedora:~/git/pcp$ pmseries -i `pmseries hotproc.psinfo.rss`| head

014f2aaf4d97f588a00c52ea3fcc672fbd423ecb
    inst [1350 or "001350 /usr/libexec/pcp/bin/pmcd"] series e720ad93a6039f435c0ce497129091b7a59be7c7
    inst [1354 or "001354 /var/lib/pcp/pmdas/root/pmdaroot"] series 782aeda842aac7a4a64a103639976f654d418148
    inst [1360 or "001360 /var/lib/pcp/pmdas/proc/pmdaproc"] series e1ae808971df1715ae8ba61e3135db85cf53e299
    inst [1361 or "001361 /var/lib/pcp/pmdas/xfs/pmdaxfs"] series 2cce339576c439f6f792df35eaf75ba339465892
    inst [1362 or "001362 /var/lib/pcp/pmdas/sample/pmdasample"] series 7e1f58318cc56efa0eeca1de410995ab1229c5de
    inst [1363 or "001363 /var/lib/pcp/pmdas/statsd/pmdastatsd"] series eff572062f652a3f888239be08bf7156f1c76892
    inst [1367 or "001367 /var/lib/pcp/pmdas/linux/pmdalinux"] series 20017f275086c90adfaeb8cdb8d76d4df712f6eb
    inst [1371 or "001371 /usr/bin/python3"] series e0632ae864826d8addcb01a8e8718df6d68d0ae2
nathans@fedora:~/git/pcp$

so, the process identifiers and command names (i.e. PCP instances for hotproc metrics), is there - there may be some magic required to display these in Grafana rather than the 40-digit SID (series identifier). @sfeifer or @andreasgerstmayr may be able to help us further there.