Need to publish KSQLDB pull metrics using prometheus

[rajeshrawool]

Hi All,

I am trying to pull the KSQLDB Pull queries and User-defined functions in prometheus using JMX exporter. For this I have tried enabling ksql.query.pull.metrics.enabled=true in ksql-server.properties. Also, added few entries in ksql_rule.yml file as below:

io.confluent.ksql.metrics:type=_confluent-ksql-pull-query

• pattern: io.confluent.ksql.metrics<>([^:]+) name: "ksql_ksql_pull_querystats$2" labels: ksql_query: $1 But still, unable to fetch the metrics in prometheus.

Please help me with this to pull the required data in prometheus.

[rajeshrawool]

I am able to fetch the ksql pull metrics using JMX exporter as below.

io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-total (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-remote-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-latency-max (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-response-size (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-error-total (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-remote-count (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-error-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-local-count (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-distribution-90 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-distribution-50 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-distribution-75 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-request-size (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-local-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-latency-min (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-distribution-99 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-latency-avg (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-total (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-remote-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-latency-max (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-response-size (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-error-total (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-remote-count (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-error-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-local-count (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-distribution-90 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-distribution-50 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-distribution-75 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-request-size (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-local-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-latency-min (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-distribution-99 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-latency-avg (Double) = nan

Now, I think I am not able to parse this values. Not sure, but please suggest if any changes in rule files is required.

Below is the rule defined in ksql_rule.yml

 # io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=
  # io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-total
  # _confluent-pull-query,ksql_service_id=poc_/
  # pattern: io.confluent.ksql.metrics:type=_confluent-pull-query
  - pattern: io.confluent.ksql.metrics<type=_confluent-pull-query,ksql_service_id=poc/*>
    name: "ksql_confluent-pull-query_query_stats_$2"
    labels:
      ksql_confluent-pull_query: $1

[agavra]

closing this issue since this seems to be specific to prometheus - ksqlDB only supports exporting metrics via JMX, any other integration is separate. Feel free to reopen this issue if it seems specific to ksqlDB