search

performancecopilot / pcp

Performance Co-Pilot
https://pcp.io
Other
974 stars 237 forks source link

process level metrics in grafana + redis #2077

Closed mallilinux1980 closed 2 months ago

mallilinux1980 commented 2 months ago

how do i configure pcp and metrics to be collected by pmlogger and visualize cpu/memory/network/disk metrics in grafana + redis by individual processes?

natoscott commented 2 months ago

@mallilinux1980 sending metrics for all processes has been found to have a crippling effect on many Redis installations due to memory required to hold such high cardinality data. One way to tackle this is to use the "hotproc" functionality in pmdaproc(1) to produce a reduced set of interesting processes, once recorded those metrics will be automatically indexed into Redis.

mallilinux1980 commented 2 months ago

are there few examples hotproc.conf ?

another question I've is can I store or how do I store derived merics into redis and visualize them in grafana?

natoscott commented 2 months ago

are there few examples hotproc.conf ?

There's a samplehotproc.conf shipped in PCP and the pmdaproc(1) man page has multiple examples.

another question I've is can I store or how do I store derived merics into redis and visualize them in grafana?

The grafana-pcp Redis data source uses the pmseries(1) query language which is similar but different to derived metrics. Derived metrics only exist for PMAPI contexts (in grafana-pcp, this means the Vector datasource).

mallilinux1980 commented 2 months ago

@mallilinux1980 sending metrics for all processes has been found to have a crippling effect on many Redis installations due to memory required to hold such high cardinality data. One way to tackle this is to use the "hotproc" functionality in pmdaproc(1) to produce a reduced set of interesting processes, once recorded those metrics will be automatically indexed into Redis.

able to see proc level metrics in grafana after hotproc is configured (cpuburn > 0.10) and setting up sasl based basic authentication. But since yesterday I'm seeing same single process and its value all time so what needs to be done for real-time display in the grafana?

natoscott commented 2 months ago

@mallilinux1980 check the values via "pminfo -f hotproc" - if Vector is displaying something out-of-date compared to that, please open a grafana-pcp github issue with details.

mallilinux1980 commented 2 months ago

Interestingly pminfo -f hotproc most metrics says no value found.

On Tue, 24 Sep 2024 at 4:02 AM, Nathan Scott @.***> wrote:

@mallilinux1980 https://github.com/mallilinux1980 check the values via "pminfo -f hotproc" - if Vector is displaying something out-of-date compared to that, please open a grafana-pcp github issue with details.

— Reply to this email directly, view it on GitHub https://github.com/performancecopilot/pcp/issues/2077#issuecomment-2369645213, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANKM3QJZVT2YS5K2R2SC4V3ZYCJIJAVCNFSM6AAAAABOP7PH5CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRZGY2DKMRRGM . You are receiving this because you were mentioned.Message ID: @.***>