RedHatInsights / expandjsonsmt

Kafka Connect SMT to expand JSON field
Apache License 2.0
17 stars 18 forks source link

Expand Prometheus api result #15

Open alexanghel23 opened 2 years ago

alexanghel23 commented 2 years ago

Hello,

Please assist me with an issue i'm having using this smt.

I'm using it alongside kafka-connect-http connector which calls the Prometheus api. The string data before the expand json transformation looks like this:

value:"{"metric":{"__name__":"ALERTS","alertname":"CPUThrottlingHigh","alertstate":"firing","container":"container-watcher","namespace":"kube-system","pod":"container-watcher-2q6nc","severity":"info"},"value":[1.643298906724E9,"1"]}"
key:"85ef1e42-ae32-3e07-a25f-dfbdd02da2b4"
timestamp:1643298906726

After applying the exapand json transformation, i get the json without the value part ("value":[1.643298906724E9,"1"]):

{"value":{"metric":{"__name__":"ALERTS","alertname":"CPUThrottlingHigh","alertstate":"firing","container":"container-watcher","namespace":"kube-system","pod":"container-watcher-2q6nc","severity":"info"}},"key":"7caf8f55-9293-35d2-9d04-70cda2e10094","timestamp":1643299372003}

Kafka connector definition:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: source-alerts
  namespace: message-bus
  labels:
    strimzi.io/cluster: kafka-connect-cluster
spec:
  class: com.github.castorm.kafka.connect.http.HttpSourceConnector
  tasksMax: 1
  config:
    http.request.url: "http://localhost:9090/api/v1/query?query=ALERTS"
    http.timer.interval.millis: 10000
    topic.creation.enable: true
    topic.creation.default.replication.factor: 1
    topic.creation.default.partitions: 1
    kafka.topic: alerts
    http.request.headers: "Content-Type: application/json"
    ###JSON (without Schema Registry)
    key.converter: org.apache.kafka.connect.json.JsonConverter #ok
    value.converter: org.apache.kafka.connect.json.JsonConverter #ok
    key.converter.schemas.enable: false
    value.converter.schemas.enable: false
    http.response.list.pointer: "/data/result"
    http.response.record.pointer: "/"
    transforms: "expandjson"
    transforms.expandjson.type: com.redhat.insights.expandjsonsmt.ExpandJSON$Value
    transforms.expandjson.sourceFields: 'value'

Is there a way to expand the Prometheus result? Or the format is not supported by the smt?

Thank you!