Expose JVM metrics with Prometheus inline exporter

[piotr-yuxuan]

Hello, thanks for this nice project! I've been stumbling upon one exception, and one unexpected behaviour while trying to expose JVM metrics to Prometheus scrapper:

(ns piotr-yuxuan.mulog-expose-kafka-publisher
  (:require [com.brunobonacci.mulog :as mulog]
            [com.brunobonacci.mulog.publishers.prometheus :as prom]))

(def pub
  (prom/prometheus-publisher {:type :prometheus}))

(def px
  (mulog/start-publisher! {:type :inline :publisher pub}))

(prom/registry pub)
;; Execution error (ClassCastException) at com.brunobonacci.mulog.publishers.prometheus.PrometheusPublisher/registry (prometheus.clj:226).
;; class io.prometheus.client.CollectorRegistry cannot be cast to class clojure.lang.IFn (io.prometheus.client.CollectorRegistry and clojure.lang.IFn are in unnamed module of loader 'app')

(prom/write-str pub)
;; => ""

(mulog/log "event-name" :a "a" :b "b")

(prom/write-str pub)
;; => "# HELP piotr_yuxuan_mulog_expose_kafka_publisher_event_name Counter of piotr-yuxuan.mulog-expose-kafka-publisher/\"event-name\" events.
;;     # TYPE piotr_yuxuan_mulog_expose_kafka_publisher_event_name counter
;;     piotr_yuxuan_mulog_expose_kafka_publisher_event_name{a=\"a\",b=\"b\",} 1.0
;;     "

(def jvm-pub
  (mulog/start-publisher! {:type :jvm-metrics
                           ;; the interval in millis between two samples (default: 60s)
                           :sampling-interval 100
                           :jvm-metrics {:memory true
                                         :gc true
                                         :threads true
                                         :jvm-attrs true}}))

(prom/write-str pub)
;; => "# HELP clojure_core_mulog_jvm_metrics_sampled Counter of clojure.core/:mulog/jvm-metrics-sampled events.
;;     # TYPE clojure_core_mulog_jvm_metrics_sampled counter
;;     clojure_core_mulog_jvm_metrics_sampled 113.0
;;     # HELP piotr_yuxuan_mulog_expose_kafka_publisher_event_name Counter of piotr-yuxuan.mulog-expose-kafka-publisher/\"event-name\" events.
;;     # TYPE piotr_yuxuan_mulog_expose_kafka_publisher_event_name counter
;;     piotr_yuxuan_mulog_expose_kafka_publisher_event_name{a=\"a\",b=\"b\",} 1.0
;;     "

So far the counter clojure_core_mulog_jvm_metrics_sampled increases, but no further metrics are exposed. I think this snippet is pretty close to the doc so I'm a bit puzzled.

On my machine this behaviour appeared consistent accross versions 0.5.0, 0.6.0, 0.7.0, and 0.7.1.

[BrunoBonacci]

This issue if fixed (thanks @piotr-yuxuan) it will be released with the next release (v0.7.2 or v0.8.0)

[BrunoBonacci]

Reopening as this issue needs more work.

[BrunoBonacci]

Currently the μ/log Prometheus publisher converts events into metrics by creating a new metrics for every numerical value. It uses string values as labels. However, it only looks at numerical values on the top-level event map, it doesn't nest.

See here: https://github.com/brunobonacci/mulog/blob/master/mulog-prometheus/src/com/brunobonacci/mulog/publishers/prometheus/metrics.clj#L70

One option/workaround is to add a :transform function on your Prometheus publisher to copy the metrics you are interested into publishing on the top level.

If you want all metrics you will need to "flatten" a nested map while preserving the names.

I will think whether this could be something that it is done automatically by μ/log publisher. My worry is that this might cause an explosion of metrics to be published.