replacement for Prometheus JMX exporter for use with puppetserver

[corporate-gadfly]

Thanks in advance for entertaining my request.

For exposing metrics from puppetserver, all searches were pointing towards JMX Exporter. I tried it out, (naively in my production puppetserver) and ended up with /metrics endpoint going silent after a few minutes of successfully returning metrics.

Since existing grafana dashboards are available for those metrics, would you be able to guide me to translate JMX exporter rules to scriptable JMX exporter _(here is an example)_?

Cheers.

[corporate-gadfly]

Some more details.

Currently using a very default-style config:

server:
  bind_address: 0.0.0.0:9639
options:
  include_timestamp: false
rules:
- pattern:
  - 'com\\.sun\\.management:type=HotSpotDiagnostic:DiagnosticOptions'
  - 'java\\.lang:type=Threading:AllThreadIds'
  - 'jdk\\.management\\.jfr'
  skip: true
- pattern: 'java\\.lang|java\\.nio|jboss\\.threads|net\\.thisptr\\.jmx\\.exporter\\.agent.*'
  transform: |
    !java
    V1.transform(in, out, "type");
- transform: |
    !java
    V1.transform(in, out, "type");

[corporate-gadfly]

I had little bit of time to debug more.

For the following MBean:

{
  "domain": "puppetserver",
  "key_properties": {
    "name": "puppetlabs.localhost.jruby.num-free-jrubies"
  },
  "bean_description": "Information on the management interface of the MBean",
  "bean_class": "com.codahale.metrics.JmxReporter$JmxGauge",
  "attribute_name": "Value",
  "attribute_description": "Attribute exposed for management",
  "attribute_type": "java.lang.Object",
  "timestamp": 1624749874255,
  "value": "1"
}

the prometheus output needs to look like:

puppetserver_jruby_num_jrubies 1.0

[corporate-gadfly]

With the following declarations:

declarations: |
  public static V1.Builder puppet(AttributeValue in) {
    String truncatedName = ((String) in.keyProperties.get("name")).replace("puppetlabs.localhost", "");
    return V1.name(in.domain + truncatedName)
      .addLabelsExcluding(in.keyProperties)
      .timestamp(in.timestamp)
      .help(in.attributeDescription);
  }

and pattern:

- pattern: 
  - puppetserver:name=puppetlabs.localhost.jruby.num-free-jrubies
  transform: |
    puppet(in)
      .type("counter")
      .transform(in.value, in.attributeType, out).done();

I can now get one metric as follows:

puppetserver_jruby_num_free_jrubies{name="puppetlabs.localhost.jruby.num-free-jrubies",} 1

Ideally, this would look like:

puppetserver_jruby_num_jrubies 1.0

Clearly, I have no idea what I'm doing, but making progress, I suppose.

[corporate-gadfly]

Some more progress. Latest YAML:

declarations: |
  public static V1.Builder puppet(AttributeValue in) {
    String truncatedName = ((String) in.keyProperties.get("name")).replace("puppetlabs.localhost", "");
    return V1.name(in.domain + truncatedName)
      .timestamp(in.timestamp);
  }

server:
  bind_address: 0.0.0.0:9639
options:
  include_timestamp: false
rules:
- pattern:
  - 'com\\.sun\\.management:type=HotSpotDiagnostic:DiagnosticOptions'
  - 'java\\.lang:type=Threading:AllThreadIds'
  - 'java.lang:type=(ClassLoading|Compilation|GarbageCollector|Memory.*|OperatingSystem)'
  - 'jdk\\.management\\.jfr'
  - 'org.eclipse.jetty.*'
  - 'com.puppetlabs.trapperkeeper.services.webserver.jetty9.utils'
  - 'jboss.threads'
  - 'jmx4perl'
  - '::.*(98thPercentile|999thPercentile|FifteenMinuteRate|FiveMinuteRate|MeanRate|OneMinuteRate|SnapshotSize)'
  - 'puppetserver:name=.*http\\.puppet.*-requests'
  - 'puppetserver:name=.*http\\.puppet.*catalog-/\\*/-percentage'
  - 'puppetserver:name=.*http\\.puppet.*(environment_|facts|file|module-name|node|report).*'
  - 'puppetserver:name=.*http\\.(active-histo|other-requests|other-percentage)'
  - 'puppetserver:name=.*num-cpus'
  skip: true
- pattern: 'java\\.lang|java\\.nio'
  transform: |
    !java
    V1.transform(in, out, "type");
## JRuby metrics
- pattern:
  - puppetserver:name=.*\\.jruby\\.(?<stat>borrow-timer|free-jrubies-histo|lock-held-timer|lock-wait-timer|requested-jrubies-histo|wait-timer)
  transform: |
    V1.name("puppetserver_jruby_" + (String) match.get("stat"))
      .type("gauge")
      .addLabel("measurement", (String) in.attributeName)
      .timestamp(in.timestamp)
      .transform(in.value, in.attributeType, out).done();
# Counters only
## HTTP metrics
- pattern:
  - puppetserver:name=.*\\.http\\.total-requests
  transform: |
    V1.name("puppetserver_http_requests_duration_milliseconds")
      .type("gauge")
      .addLabel("endpoint", "total")
      .addLabel("measurement", (String) in.attributeName)
      .timestamp(in.timestamp)
      .transform(in.value, in.attributeType, out).done();
- pattern:
  - puppetserver:name=.*\\.http\\.(?<endpoint>puppet.+percentage)
  transform: |
    V1.name("puppetserver_http_requests_percentage")
      .type("gauge")
      .addLabel("endpoint", (String) match.get("endpoint"))
      .timestamp(in.timestamp)
      .transform(in.value, in.attributeType, out).done();
## JRuby metrics
- pattern:
  - puppetserver:name=.*\\.jruby\\..*
  transform: |
    puppet(in)
      .type("counter")
      .transform(in.value, in.attributeType, out).done();
## Other metrics
- pattern:
  - puppetserver:name=puppetlabs.localhost.uptime
  transform: |
    puppet(in)
      .type("counter")
      .help("Puppet Server process’s uptime.")
      .transform(in.value, in.attributeType, out).done();
- pattern:
  - puppetserver:name=.*\\.memory\\.(?<area>.*)\\.(?<stat>.*)
  transform: |
    V1.name("puppetserver_memory_bytes_" + (String) match.get("stat"))
      .type("gauge")
      .addLabel("area", (String) match.get("area"))
      .transform(in.value, in.attributeType, out).done();
- transform: |
    !java
    V1.transform(in, out, "type");

[corporate-gadfly]

Based on my needs and the grafana dashboard that I wanted to used, this is the YAML that I ended up with:

declarations: |
  public static V1.Builder puppet(AttributeValue in) {
    String truncatedName = ((String) in.keyProperties.get("name")).replace("puppetlabs.localhost", "");
    return V1.name(in.domain + truncatedName)
      .timestamp(in.timestamp);
  }

server:
  bind_address: 0.0.0.0:9101
options:
  include_timestamp: false
rules:
## HTTP metrics with JmxReporter$JmxCounter
- pattern:
  - puppetserver:name=.*http.active-requests
  transform: |
    puppet(in)
      .type("gauge")
      .transform(in.value, in.attributeType, out).done();
- pattern:
  - 'com\\.sun\\.management:type=HotSpotDiagnostic:DiagnosticOptions'
  - 'java\\.lang:type=Threading:AllThreadIds'
  - 'java.lang:type=(ClassLoading|Compilation|GarbageCollector|Memory.*|OperatingSystem|Runtime|Threading)'
  - 'jdk\\.management\\.jfr'
  - 'java.nio'
  - 'org.eclipse.jetty.*'
  - 'com.puppetlabs.trapperkeeper.services.webserver.jetty9.utils'
  - 'jboss.threads'
  - 'jmx4perl'
  - 'jolokia'
  - 'puppetlabs.trapperkeeper.*'
  - 'net.thisptr.jmx.exporter.*'
  - '::.*(50thPercentile|75thPercentile|95thPercentile|98thPercentile|99thPercentile|999thPercentile|Count|DurationUnit|FifteenMinuteRate|FiveMinuteRate|MeanRate|Min|Max|OneMinuteRate|RateUnit|SnapshotSize|StdDev)'
  - 'puppetserver:name=.*http\\.puppet-v4.*'
  - 'puppetserver:name=puppetlabs.*http.other-percentage'
  - 'puppetserver:name=puppetlabs.*compiler'
  - 'puppetserver:name=puppetlabs.*compiler.compile.production.*'
  - 'puppetserver:name=puppetlabs.*compiler.(create_settings_scope|evaluate_|finish|init|iterate|set|validate).*'
  - 'puppetserver:name=puppetlabs.*functions\\..*'
  - 'puppetserver:name=.*http\\.puppet.*(compile|environment|facts|file_bucket_file|static_file_content|plans|tasks).*'
  - 'puppetserver:name=puppetlabs.*puppetdb.(aliases|catalog|edges|events|facts|keys|logs|name|meta|metrics|parameters|payload|report|resource|titles).*'
  - 'puppetserver:name=puppetlabs.*indirector.*'
  - 'puppetserver:name=puppetlabs.*puppetdb$'
  - 'puppetserver:name=puppetlabs.*puppetdb.command$'
  - 'puppetserver:name=puppetlabs.*puppetdb.command.submit.*'
  skip: true
## HTTP/Compiler/PuppetDB metrics
- pattern:
  - puppetserver:name=.*\\.http\\.(.*)-percentage
  transform: |
    puppet(in)
      .type("gauge")
      .transform(in.value, in.attributeType, out).done();
- pattern:
  - puppetserver:name=.*\\.((compiler|http|puppetdb|http.*puppet)\\..*)
  transform: |
    puppet(in)
      .type("gauge")
      .addLabel("measurement", (String) in.attributeName)
      .transform(in.value, in.attributeType, out).done();
## JRuby metrics
- pattern:
  - puppetserver:name=.*\\.jruby\\.(borrow-timer|free-jrubies-histo|lock-held-timer|lock-wait-timer|requested-jrubies-histo|wait-timer)
  transform: |
    puppet(in)
      .type("gauge")
      .addLabel("measurement", (String) in.attributeName)
      .transform(in.value, in.attributeType, out).done();
- pattern:
  - puppetserver:name=.*\\.jruby\\..*
  transform: |
    puppet(in)
      .type("gauge")
      .transform(in.value, in.attributeType, out).done();
## Other metrics
- pattern:
  - puppetserver:name=.*\\.functions
  transform: |
    puppet(in)
      .type("gauge")
      .addLabel("measurement", (String) in.attributeName)
      .transform(in.value, in.attributeType, out).done();
- pattern:
  - puppetserver:name=puppetlabs.localhost.uptime
  transform: |
    puppet(in)
      .type("counter")
      .transform(in.value, in.attributeType, out).done();
- pattern:
  - puppetserver:name=.*num-cpus
  transform: |
    puppet(in)
      .type("gauge")
      .transform(in.value, in.attributeType, out).done();
- pattern:
  - puppetserver:name=.*\\.memory\\.(?<area>.*)\\.(?<stat>.*)
  transform: |
    V1.name("puppetserver_memory_bytes_" + (String) match.get("stat"))
      .type("gauge")
      .addLabel("area", (String) match.get("area"))
      .transform(in.value, in.attributeType, out).done();
- transform: |
    !java
    V1.transform(in, out, "type");