Add support for PCP connection count on puppetserver

bastelfreak commented 2 years ago

Use Case

I'm currently debugging pcp-agent<>puppetserver connection problems with Puppet Engineers for a customer. I noticed that there is a metric for the amount of open PCP connections, but no related dashboard. Having that would help a lot to identify overloaded puppetservers / uneven loadbalancing of PCP connections. I used:

curl --cert $(puppet config print hostcert) --key $(puppet config print hostprivkey) --cacert $(puppet config print localcacert) https://$(hostname -f):8140/metrics/v2/read/default:name=puppetlabs.pcp.connections --silent | jq '.value[]'

Describe the Solution You Would Like

A dashboard that tracks a connection counter over time

Describe Alternatives You've Considered

Customer has other monitoring soltutions where we could implement this, but I think such a panel would be really helpful for other users as well.

Additional Context

I'm not 100% sure if default:name=puppetlabs.pcp.connections gives the amount of current open connections, but I think so. I didn't find any good documentation about the JMX/Jolokia metrics

m0dular commented 2 years ago

Thanks, we're currently working on adding more meaningful Orchestrator metrics in PE, so we'll consider this when we build out a dashboard for those. It may be worth making a dashboard with what is already available, such as this one and JVM metrics.

SimonHoenscheid commented 1 year ago

This may be part of #148

m0dular commented 1 year ago

This particular mbean, puppetlabs.pcp.connections, looks to be the only one not exposed by the /status api, so it's not currently available. We would have to add another set of config to do it, but since it would be a useful metric I'll leave this issue open.

SimonHoenscheid commented 1 year ago

Yes, and the metric seems to be only available on compilers. Did not check a standard installation.

m0dular commented 1 year ago

I think it's available on the primary via Orchestrator's api, so if you replaced 8140 with 8143 in the example it should work. That reminds me that we missed collecting PCP metrics on compilers in #148 because they come from Puppet server, but we do have a ticket for that.

tuxmea commented 1 year ago

Is this feature already implemented in the orchestration dashboard?

m0dular commented 1 year ago

We don't currently have puppetlabs.pcp.connections since it's not accessible via the /status api. We'd need to add a new puppet_operational_dashboards::telegraf::config type and have it POST to the /metrics/v2 api similar to how the PDB metrics do here.

For compiler PCP metrics, we need to do something like

Add the broker service metrics (and any others, for compilers I think it's only the broker service) from here to the Puppet server config.
Update the Orchestrator panels that use them to select from both the orchestrator and puppetserver measurements

One concern might be if the broker service metrics don't exist on OSP, I'm not sure if they do. If you define a path on a json_v2.object that doesn't exist, the whole collection will fail. We could work around that by adding the fields to included_keys instead, which will just skip them if they aren't available.