Allow url in prometheus query variables

[bartoszcisek]

Describe the enhancement:

Currently prometheus integration query config variable accept variables, but escape them before creating query.

  - id: prometheus/metrics-prometheus-${kubernetes.hints.container_id}
    name: prometheus-1
    type: prometheus/metrics
    use_output: default
    meta:
      package:
        name: prometheus
        version: 1.12.0
    data_stream:
      namespace: default
    processors:
      - drop_fields:
          fields: ["/^kubernetes.node./", "/^kubernetes.namespace_labels/", "kubernetes.namespace_uid"]
      - add_fields:
          fields:
            name: default
          target: orchestrator.cluster
    streams:
      - condition: ${kubernetes.hints.prometheus.collector.enabled} == true or ${kubernetes.hints.prometheus.enabled} == true
        data_stream:
          dataset: prometheus.collector
          type: metrics
        hosts:
          - ${kubernetes.hints.prometheus.collector.host|kubernetes.hints.prometheus.host|'localhost:9090'}
        metrics_filters.exclude: null
        metrics_filters.include: null
        metrics_path: '/probe'
        metricsets:
          - collector
        period: ${kubernetes.hints.prometheus.collector.period|kubernetes.hints.prometheus.period|'60s'}
        rate_counters: true
        timeout: ${kubernetes.hints.prometheus.collector.timeout|kubernetes.hints.prometheus.timeout|'3s'}
        use_types: true
        query:
          target: "http://127.0.0.1:3000/json/endpoint"

Results in query to json-exporter:

http://10.1.1.1:9090/probe?target=http%3A%2F%2F127.0.0.1%3A3000%2Fjson%2Fendpoint

Describe a specific use case for the enhancement or feature:

json_exporter, blackbox_exporter allow unified configuration where specific endpoint url is passed via target variable.

What is the definition of done?

Query variables allow URL type values, which are not escaped.

Notes Cross posting https://github.com/elastic/elastic-agent/issues/3421

[belimawr]

Looking at the documentation/example provided in the integration page (in Kibana go to Integrations -> Prometheus -> Configs) (https://<your kibana URL>/app/integrations/detail/prometheus-1.13.1/configs), that seems to be the correct behaviour to me.

There is a hosts field that can be used to configure Prometheus host, any query string parameters go into the query map.

If I understood it correctly you would like to set the host field the integration is using to query Prometheus, is that correct?

So in the example you posted we can see (I'm quoting only the interesting parts here):

        hosts:
          - ${kubernetes.hints.prometheus.collector.host|kubernetes.hints.prometheus.host|'localhost:9090'}
        query:
          target: "http://127.0.0.1:3000/json/endpoint"

And looking at the final URL you mentioned:

http://10.1.1.1:9090/probe?target=http%3A%2F%2F127.0.0.1%3A3000%2Fjson%2Fendpoint

The query is correctly URL escaped, the host (http://10.1.1.1:9090/) looks to be coming from the Kubernetes hints from the hosts field, so either kubernetes.hints.prometheus.collector.host or kubernetes.hints.prometheus.host are resolving to http://10.1.1.1:9090

[belimawr]

I did some testing on a Fleet managed Elastic-Agent and it works as I described. For the Prometheus host I used https://httpdump.app/ that will give you a URL dumping all requests made against it.

My input YAML configuration (I added a few extra query parameters just for the sake of testing):

  - id: prometheus/metrics-prometheus-fd262db7-f731-430f-bfb1-c741e15d8f3d
    name: prometheus-1
    revision: 2
    type: prometheus/metrics
    use_output: default
    meta:
      package:
        name: prometheus
        version: 1.13.1
    data_stream:
      namespace: default
    package_policy_id: fd262db7-f731-430f-bfb1-c741e15d8f3d
    streams:
      - id: >-
          prometheus/metrics-prometheus.collector-fd262db7-f731-430f-bfb1-c741e15d8f3d
        data_stream:
          dataset: prometheus.collector
        metricsets:
          - collector
        hosts:
          - 'https://httpdump.app/dumps/2ecc5393-3ed4-483a-ac45-48defce229a3'
        metrics_filters.exclude: null
        metrics_filters.include: null
        metrics_path: /metrics
        period: 10s
        rate_counters: true
        use_types: true
        username: user
        password: secret
        query:
          target: 'http://127.0.0.1:3000/json/endpoint'
          foo: bar
          answer: 42

Note that hosts is set to the httpdump.app URL The final requests are as expected:

https://httpdump.app/dumps/2ecc5393-3ed4-483a-ac45-48defce229a3?answer=42&foo=bar&target=http%3A%2F%2F127.0.0.1%3A3000%2Fjson%2Fendpoint

A screenshot showing the full request:

[belimawr]

The details of how I setup the integration in Fleet UI. Sorry for the bad quality image, I just got a random screenshot plugin.

[ishleenk17]

@bartoszcisek :

I was going through the code and yes, the URL encoding is something we do for the query parameters as you can see here. And URL encoding is a common practice that should be followed.

I am trying to wrap my head around the need to not encode it.

[ishleenk17]

Also , I used json_exporter with both encoded and non-encoded URL. It works for both the cases.

[bartoszcisek]

@ishleenk17 @belimawr Thanks for looking into it. I performed tests again and I can confirm that described configuration works. Unfortunately I distilled my configuration too much to the point that it does not describe problem correctly. Sorry for that.

Broader context

Provided configuration describes one type of exporter. But I have multiple. Each listens on different port and require different target query. target parameter can't be provided via hints. My initial approach was to use condition to pick selected part of agent config. But on what variable I should make conditional? kubernetes.deployment.name might be a convenient one. The problem is that enabling hints turns off other kubernetes provider metadata. The only metadata that is left is container_id. Moreover hints can only be turned on/off for whole agent and influences all other integrations.

Used solution

Solution that worked consists of two steps:

1. Extract hints related integrations to separate agent deployment.
2. Use username field to encode configuration types. We don't use username in or configs but it has a benefit of being recognized by hints code. Resulting configuration attached below.

Possible next steps

1. Agent already accepts processors definitions via hints. If similar approach would be applied to query parameter, configuration could be simplified greatly.
2. Term providers.kubernetes.hints.enable: true is misleading to say at least. It does not enable, it switches between type of metadata source. At least to me 'enable' imply additive action, while after 'enabling' hints, conditions based on regular kubernetes provider variables stop working. I understand this is beta feature. Maybe a bit more emphasis in documentation on how it actually works would be helpful.

Working configuration

Pod config

podAnnotations:
  co.elastic.hints/package: prometheus
  co.elastic.hints/host: '${kubernetes.pod.ip}:7979'
  co.elastic.hints/username: 'json_exporter'

Agent config

inputs:
  - id: prometheus/metrics-prometheus-${kubernetes.hints.container_id}
    name: prometheus-1
    type: prometheus/metrics
    use_output: default
    meta:
      package:
        name: prometheus
        version: ${local.prometheus_integration_version}
    data_stream:
      namespace: ${env.ENVIRONMENT}
    processors:
      - drop_fields:
          fields: ["/^kubernetes.node./", "/^kubernetes.namespace_labels/", "kubernetes.namespace_uid"]
      - add_fields:
          fields:
            name: ${env.ENVIRONMENT}
          target: orchestrator.cluster
    streams:
      - condition: ${kubernetes.hints.prometheus.enabled} == true and length(${kubernetes.hints.prometheus.username}) == 0
        data_stream:
          dataset: prometheus.collector
          type: metrics
        hosts:
          - ${kubernetes.hints.prometheus.collector.host|kubernetes.hints.prometheus.host|'localhost:9090'}
        metrics_path: ${kubernetes.hints.prometheus.metrics_path|'/metrics'}
        metricsets:
          - collector
        period: ${kubernetes.hints.prometheus.collector.period|kubernetes.hints.prometheus.period|'60s'}
        rate_counters: true
        timeout: ${kubernetes.hints.prometheus.collector.timeout|kubernetes.hints.prometheus.timeout|'3s'}
        use_types: true
        query:
          format: prometheus
      - condition: ${kubernetes.hints.prometheus.enabled} == true and ${kubernetes.hints.prometheus.username} == 'json_exporter'
        data_stream:
          dataset: prometheus.collector
          type: metrics
        hosts:
          - '${kubernetes.hints.prometheus.host}'
        metrics_path: '/probe'
        metricsets:
          - collector
        period: '60s'
        timeout: '3s'
        use_types: true
        query:
          target: http://127.0.0.1:3000/json/endpoint