Allow url in prometheus query variables

[bartoszcisek]

Describe the enhancement:

Currently prometheus integration query config variable accept variables, but escape them before creating query.

  - id: prometheus/metrics-prometheus-${kubernetes.hints.container_id}
    name: prometheus-1
    type: prometheus/metrics
    use_output: default
    meta:
      package:
        name: prometheus
        version: 1.12.0
    data_stream:
      namespace: default
    processors:
      - drop_fields:
          fields: ["/^kubernetes.node./", "/^kubernetes.namespace_labels/", "kubernetes.namespace_uid"]
      - add_fields:
          fields:
            name: default
          target: orchestrator.cluster
    streams:
      - condition: ${kubernetes.hints.prometheus.collector.enabled} == true or ${kubernetes.hints.prometheus.enabled} == true
        data_stream:
          dataset: prometheus.collector
          type: metrics
        hosts:
          - ${kubernetes.hints.prometheus.collector.host|kubernetes.hints.prometheus.host|'localhost:9090'}
        metrics_filters.exclude: null
        metrics_filters.include: null
        metrics_path: '/probe'
        metricsets:
          - collector
        period: ${kubernetes.hints.prometheus.collector.period|kubernetes.hints.prometheus.period|'60s'}
        rate_counters: true
        timeout: ${kubernetes.hints.prometheus.collector.timeout|kubernetes.hints.prometheus.timeout|'3s'}
        use_types: true
        query:
          target: "http://127.0.0.1:3000/json/endpoint"

Results in query to json-exporter:

http://10.1.1.1:9090/probe?target=http%3A%2F%2F127.0.0.1%3A3000%2Fjson%2Fendpoint

Describe a specific use case for the enhancement or feature:

json_exporter, blackbox_exporter allow unified configuration where specific endpoint url is passed via target variable.

What is the definition of done?

Query variables allow URL type values, which are not escaped.

[jlind23]

@elastic/obs-cloudnative-monitoring don't you think this should be transferred to the https://github.com/elastic/integrations repository?

[gizas]

Yes I agree it falls into under the category of Prometheus Integration.

cc @elastic/obs-infraobs-integrations that are the owners

[gizas]

Having second thoughts for the implementation, the enhancement might need a variable substitution in agent side. But because it will need also some changes on the prometheus integration itself it is better to have the initial issue tracked in integrations

[belimawr]

I added some comments in the integration issue (https://github.com/elastic/integrations/issues/7886#issuecomment-1792000387). The integration seems to be working as expected (at least on my understanding). It looks to be a miss configuration.