search

DataDog / integrations-core

Core integrations of the Datadog Agent
BSD 3-Clause "New" or "Revised" License
913 stars 1.39k forks source link

openmetric and prometheus agent requests.exceptions.InvalidURL: Failed to parse: http://%%host%%:2112/metrics #12678

Closed thomas-weishaupl-rft closed 2 years ago

thomas-weishaupl-rft commented 2 years ago

Output of the info page

===============
Agent (v7.34.0)
===============

  Status date: 2022-08-04 16:14:04.187 UTC (1659629644187)
  Agent start: 2022-08-04 16:13:32.887 UTC (1659629612887)
  Pid: 421325
  Go Version: go1.16.12
  Python Version: 3.8.11
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: INFO

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    System time: 2022-08-04 16:14:04.187 UTC (1659629644187)

  Host Info
  =========
    bootTime: 2022-08-04 10:21:05 UTC (1659608465000)
    kernelArch: x86_64
    kernelVersion: 4.18.0-305.25.1.el8_4.x86_64
    os: linux
    platform: ubuntu
    platformFamily: debian
    platformVersion: 21.10
    procs: 933
    uptime: 5h52m40s
    virtualizationRole: guest
    virtualizationSystem: docker

  Hostnames
  =========
    cluster-name: cluster4
    host_aliases: [xxxxxxxx-cluster4]
    hostname: xxxxxxxx.com
    socket-fqdn: datadog-dc9xd
    socket-hostname: datadog-dc9xd
    host tags:
      cluster_name:cluster4
      kube_cluster_name:cluster4
      kube_node_role:worker
    hostname provider: container
    unused hostname providers:
      aws: not retrieving hostname from AWS: the host is not an ECS instance and other providers already retrieve non-default hostnames
      azure: azure_hostname_style is set to 'os'
      configuration/environment: hostname is empty
      gce: unable to retrieve hostname from GCE: GCE metadata API error: Get "http://169.254.169.254/computeMetadata/v1/instance/hostname": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

  Metadata
  ========
    agent_version: 7.34.0
    config_apm_dd_url:
    config_dd_url:
    config_logs_dd_url:
    config_logs_socks5_proxy_address:
    config_no_proxy: [localhost .....com .....com 10.186.129.61 .....com .....net localhost.localdomain 10.33.0.1]
    config_process_dd_url:
    config_proxy_http:
    config_proxy_https: http://webproxy....
    config_site:
    feature_apm_enabled: false
    feature_cspm_enabled: false
    feature_cws_enabled: true
    feature_logs_enabled: true
    feature_networks_enabled: true
    feature_process_enabled: false
    flavor: agent
    hostname_source: container
    install_method_installer_version: kubernetes sample manifests
    install_method_tool: kubernetes sample manifests
    install_method_tool_version: kubernetes sample manifests
    logs_transport: HTTP

=========
Collector
=========

  Running Checks
  ==============

    cilium (1.10.2)
    ---------------
      Instance ID: cilium:2585dc84b30b0b9 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cilium.d/auto_conf.yaml
      Total Runs: 1
      Metric Samples: Last Run: 1,043, Total: 1,043
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 1
      Average Execution Time : 168ms
      Last Execution Date : 2022-08-04 16:13:50 UTC (1659629630000)
      Last Successful Execution Date : 2022-08-04 16:13:50 UTC (1659629630000)
      metadata:
        version.major: 1
        version.minor: 11
        version.patch: 7
        version.raw: 1.11.7-cee.2
        version.release: cee.2
        version.scheme: semver

    container
    ---------
      Instance ID: container [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/container.d/conf.yaml.default
      Total Runs: 1
      Metric Samples: Last Run: 590, Total: 590
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 21ms
      Last Execution Date : 2022-08-04 16:13:55 UTC (1659629635000)
      Last Successful Execution Date : 2022-08-04 16:13:55 UTC (1659629635000)

    cpu
    ---
      Instance ID: cpu [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 9, Total: 11
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-08-04 16:14:02 UTC (1659629642000)
      Last Successful Execution Date : 2022-08-04 16:14:02 UTC (1659629642000)

    disk (4.5.2)
    ------------
      Instance ID: disk:e5dffb8bef24336f [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/disk.d/conf.yaml.default
      Total Runs: 1
      Metric Samples: Last Run: 416, Total: 416
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 41ms
      Last Execution Date : 2022-08-04 16:13:54 UTC (1659629634000)
      Last Successful Execution Date : 2022-08-04 16:13:54 UTC (1659629634000)

    docker
    ------
      Instance ID: docker [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 1,190, Total: 2,380
      Events: Last Run: 1, Total: 1
      Service Checks: Last Run: 1, Total: 2
      Average Execution Time : 98ms
      Last Execution Date : 2022-08-04 16:14:01 UTC (1659629641000)
      Last Successful Execution Date : 2022-08-04 16:14:01 UTC (1659629641000)

    file_handle
    -----------
      Instance ID: file_handle [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/file_handle.d/conf.yaml.default
      Total Runs: 1
      Metric Samples: Last Run: 5, Total: 5
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-08-04 16:13:53 UTC (1659629633000)
      Last Successful Execution Date : 2022-08-04 16:13:53 UTC (1659629633000)

    io
    --
      Instance ID: io [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/io.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 80, Total: 106
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-08-04 16:14:00 UTC (1659629640000)
      Last Successful Execution Date : 2022-08-04 16:14:00 UTC (1659629640000)

    kubelet (7.1.1)
    ---------------
      Instance ID: kubelet:5bbc63f3938c02f4 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/kubelet.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 1,424, Total: 2,807
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 4, Total: 8
      Average Execution Time : 478ms
      Last Execution Date : 2022-08-04 16:14:02 UTC (1659629642000)
      Last Successful Execution Date : 2022-08-04 16:14:02 UTC (1659629642000)

    load
    ----
      Instance ID: load [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/load.d/conf.yaml.default
      Total Runs: 1
      Metric Samples: Last Run: 6, Total: 6
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-08-04 16:13:52 UTC (1659629632000)
      Last Successful Execution Date : 2022-08-04 16:13:52 UTC (1659629632000)

    memory
    ------
      Instance ID: memory [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/memory.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 20, Total: 40
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-08-04 16:13:59 UTC (1659629639000)
      Last Successful Execution Date : 2022-08-04 16:13:59 UTC (1659629639000)

    network (2.5.0)
    ---------------
      Instance ID: network:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/network.d/conf.yaml.default
      Total Runs: 1
      Metric Samples: Last Run: 185, Total: 185
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 13ms
      Last Execution Date : 2022-08-04 16:13:51 UTC (1659629631000)
      Last Successful Execution Date : 2022-08-04 16:13:51 UTC (1659629631000)

    ntp
    ---
      Instance ID: ntp:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/ntp.d/conf.yaml.default
      Total Runs: 1
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 1
      Average Execution Time : 20.051s
      Last Execution Date : 2022-08-04 16:14:01 UTC (1659629641000)
      Last Successful Execution Date : 2022-08-04 16:14:01 UTC (1659629641000)

    openmetrics (1.16.3)
    --------------------
      Instance ID: openmetrics:tetragon:1361b0125907fe68 [ERROR]
      Configuration Source: file:/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml
      Total Runs: 2
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 2
      Average Execution Time : 9ms
      Last Execution Date : 2022-08-04 16:13:56 UTC (1659629636000)
      Last Successful Execution Date : Never
      Error: Failed to parse: http://%%host%%:2112/metrics
      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py", line 382, in prepare_url
          scheme, auth, host, port, path, query, fragment = parse_url(url)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/util/url.py", line 394, in parse_url
          return six.raise_from(LocationParseError(source_url), None)
        File "<string>", line 3, in raise_from
      urllib3.exceptions.LocationParseError: Failed to parse: http://%%host%%:2112/metrics

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 1033, in run
          self.check(instance)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/v2/base.py", line 60, in check
          scraper.scrape()
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/v2/scraper.py", line 238, in scrape
          for metric in self.consume_metrics(runtime_data):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/v2/scraper.py", line 258, in consume_metrics
          for metric in metric_parser:
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/v2/scraper.py", line 276, in parse_metrics
          for metric in self.parse_metric_families(line_streamer):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/prometheus_client/parser.py", line 174, in text_fd_to_metric_families
          for line in fd:
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/v2/scraper.py", line 337, in stream_connection_lines
          with self.get_connection() as connection:
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/v2/scraper.py", line 358, in get_connection
          response = self.send_request()
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/v2/scraper.py", line 385, in send_request
          return self.http.get(self.endpoint, **kwargs)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py", line 341, in get
          return self._request('get', url, options)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py", line 405, in _request
          response = self.make_request_aia_chasing(request_method, method, url, new_options, persist)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py", line 411, in make_request_aia_chasing
          response = request_method(url, **new_options)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/api.py", line 76, in get
          return request('get', url, params=params, **kwargs)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/api.py", line 61, in request
          return session.request(method=method, url=url, **kwargs)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/sessions.py", line 528, in request
          prep = self.prepare_request(req)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/sessions.py", line 456, in prepare_request
          p.prepare(
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py", line 316, in prepare
          self.prepare_url(url, params)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py", line 384, in prepare_url
          raise InvalidURL(*e.args)
      requests.exceptions.InvalidURL: Failed to parse: http://%%host%%:2112/metrics

    prometheus (3.3.1)
    ------------------
      Instance ID: prometheus:tetragon_prometheus:8fc94d70befe0f34 [ERROR]
      Configuration Source: file:/etc/datadog-agent/conf.d/prometheus.d/conf.yaml
      Total Runs: 2
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 2
      Average Execution Time : 3ms
      Last Execution Date : 2022-08-04 16:14:03 UTC (1659629643000)
      Last Successful Execution Date : Never
      Error: Failed to parse: http://%%host%%:2112/metrics
      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py", line 382, in prepare_url
          scheme, auth, host, port, path, query, fragment = parse_url(url)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/util/url.py", line 394, in parse_url
          return six.raise_from(LocationParseError(source_url), None)
        File "<string>", line 3, in raise_from
      urllib3.exceptions.LocationParseError: Failed to parse: http://%%host%%:2112/metrics

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 1033, in run
          self.check(instance)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/prometheus/base_check.py", line 106, in check
          scraper.process(
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/prometheus/mixins.py", line 408, in process
          for metric in self.scrape_metrics(endpoint, instance=instance):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/prometheus/mixins.py", line 370, in scrape_metrics
          response = self.poll(endpoint, instance=instance)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/prometheus/mixins.py", line 579, in poll
          response = handler.get(endpoint, extra_headers=headers, stream=False)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py", line 341, in get
          return self._request('get', url, options)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py", line 405, in _request
          response = self.make_request_aia_chasing(request_method, method, url, new_options, persist)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py", line 411, in make_request_aia_chasing
          response = request_method(url, **new_options)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/api.py", line 76, in get
          return request('get', url, params=params, **kwargs)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/api.py", line 61, in request
          return session.request(method=method, url=url, **kwargs)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/sessions.py", line 528, in request
          prep = self.prepare_request(req)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/sessions.py", line 456, in prepare_request
          p.prepare(
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py", line 316, in prepare
          self.prepare_url(url, params)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py", line 384, in prepare_url
          raise InvalidURL(*e.args)
      requests.exceptions.InvalidURL: Failed to parse: http://%%host%%:2112/metrics

    uptime
    ------
      Instance ID: uptime [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 1, Total: 2
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-08-04 16:13:58 UTC (1659629638000)
      Last Successful Execution Date : 2022-08-04 16:13:58 UTC (1659629638000)

========
JMXFetch
========

  Information
  ==================
  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    Cluster: 0
    ClusterRole: 0
    ClusterRoleBinding: 0
    CronJob: 0
    DaemonSet: 0
    Deployment: 0
    Dropped: 0
    HighPriorityQueueFull: 0
    Job: 0
    Node: 0
    PersistentVolume: 0
    PersistentVolumeClaim: 0
    Pod: 0
    ReplicaSet: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Role: 0
    RoleBinding: 0
    Service: 0
    ServiceAccount: 0
    StatefulSet: 0

  Transaction Successes
  =====================
    Total number: 4
    Successes By Endpoint:
      check_run_v1: 1
      intake: 2
      series_v1: 1

  On-disk storage
  ===============
    On-disk storage is disabled. Configure `forwarder_storage_max_size_in_bytes` to enable it.

  API Keys status
  ===============
    API key ending with ecefd: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.eu - API Key ending with:
      - ecefd

==========
Logs Agent
==========

    Reliable: Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.eu on port 443
    BytesSent: 7.648172e+06
    EncodedBytesSent: 216510
    LogsProcessed: 3701
    LogsSent: 3369

  kube-system/anthos-multinet-xszf7/multus
  ----------------------------------------
    - Type: file
      Identifier: d82afc0dd2746b957dd1782ae70290130803c3183003b418cb12138dbcda3d8f
      Path: /var/log/pods/kube-system_anthos-multinet-xszf7_828af163-7a1c-4a29-b552-93022a58d1e4/multus/*.log
      Status: OK
        1 files tailed out of 1 files matching
      Inputs:
        /var/log/pods/kube-system_anthos-multinet-xszf7_828af163-7a1c-4a29-b552-93022a58d1e4/multus/0.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  kube-system/localpv-r2gz9/provisioner
  -------------------------------------
    - Type: file
      Identifier: bd7de4b4b6a46f9eb78b74bf21bc8a816bfee8fe405c9fba1211fad36194d474
      Path: /var/log/pods/kube-system_localpv-r2gz9_a04d7398-03b7-4c68-87fe-40e4ffe1026d/provisioner/*.log
      Status: OK
        2 files tailed out of 2 files matching
      Inputs:
        /var/log/pods/kube-system_localpv-r2gz9_a04d7398-03b7-4c68-87fe-40e4ffe1026d/provisioner/46.log
        /var/log/pods/kube-system_localpv-r2gz9_a04d7398-03b7-4c68-87fe-40e4ffe1026d/provisioner/45.log
      BytesRead: 1392
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  monitoring-datadog/datadog-dc9xd/init-volume
  --------------------------------------------
    - Type: file
      Identifier: a16a73686adfe82076f00edb9a1d85ae36a7219a2387ace97fa2a6751aabeb67
      Path: /var/log/pods/monitoring-datadog_datadog-dc9xd_9b098301-30f3-452d-bc47-81c38f25306a/init-volume/*.log
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  monitoring-datadog/datadog-dc9xd/trace-agent
  --------------------------------------------
    - Type: file
      Identifier: da53885936254ecfa260107c58211de22fdf6fd360f70c1ece94564ed741b23c
      Path: /var/log/pods/monitoring-datadog_datadog-dc9xd_9b098301-30f3-452d-bc47-81c38f25306a/trace-agent/*.log
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

 ...

  monitoring-datadog/datadog-dc9xd/system-probe
  ---------------------------------------------
    - Type: file
      Identifier: 0748e82ee9ea5b3f252050045a3f10394d75836ff3bedcb94ea3a36ac73cfd6a
      Path: /var/log/pods/monitoring-datadog_datadog-dc9xd_9b098301-30f3-452d-bc47-81c38f25306a/system-probe/*.log
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  monitoring-datadog/datadog-dc9xd/init-config
  --------------------------------------------
    - Type: file
      Identifier: c0801e389bcac4ff52dd4873697304fc4df2a49ca6915198ef1efdb83324d16c
      Path: /var/log/pods/monitoring-datadog_datadog-dc9xd_9b098301-30f3-452d-bc47-81c38f25306a/init-config/*.log
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  monitoring-datadog/datadog-dc9xd/process-agent
  ----------------------------------------------
    - Type: file
      Identifier: 63d992ea85172db97c2c16ab4a77d8bdc898d9d967a637668743b667e733fc8b
      Path: /var/log/pods/monitoring-datadog_datadog-dc9xd_9b098301-30f3-452d-bc47-81c38f25306a/process-agent/*.log
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  ...

  monitoring-datadog/datadog-dc9xd/agent
  --------------------------------------
    - Type: file
      Identifier: 62b7c2df5600433ef5769e38523122c534caef43c5ffb19c4d9c1c3d9c2b06df
      Path: /var/log/pods/monitoring-datadog_datadog-dc9xd_9b098301-30f3-452d-bc47-81c38f25306a/agent/*.log
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  ...

  monitoring-datadog/datadog-dc9xd/security-agent
  -----------------------------------------------
    - Type: file
      Identifier: ffa3a275df4fc12404e4743f198310d51a018dd9887fe07f851caa6dbd9a4ac3
      Path: /var/log/pods/monitoring-datadog_datadog-dc9xd_9b098301-30f3-452d-bc47-81c38f25306a/security-agent/*.log
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  ...

============
System Probe
============
System Probe is running

=========
APM Agent
=========
  Status: Running
  Pid: 421387
  Uptime: 30 seconds
  Mem alloc: 10,560,224 bytes
  Hostname: .....com
  Receiver: 0.0.0.0:8126
  Endpoints:
    https://trace.agent.datadoghq.eu

  Receiver (previous minute)
  ==========================
    No traces received in the previous minute.
    Default priority sampling rate: 100.0%

  Writer (previous minute)
  ========================
    Traces: 0 payloads, 0 traces, 0 events, 0 bytes
    Stats: 0 payloads, 0 stats buckets, 0 bytes

=========
Aggregator
=========
  Checks Metric Sample: 7,637
  Dogstatsd Metric Sample: 899
  Event: 2
  Events Flushed: 2
  Number Of Flushes: 1
  Series Flushed: 1,703
  Service Check: 39
  Service Checks Flushed: 16
=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 898
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 77,917
  Udp Packet Reading Errors: 0
  Udp Packets: 243
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 1
  Unterminated Metric Errors: 0

=====================
Datadog Cluster Agent
=====================

  - Datadog Cluster Agent endpoint detected: https://10.33.242.151:5005
  Successfully connected to the Datadog Cluster Agent.
  - Running: 1.19.0+commit.083a221

=============
Autodiscovery
=============
  Enabled Features
  ================
    docker
    kubernetes

Additional environment details (Operating System, Cloud provider, etc):

Steps to reproduce the issue:

  1. use %%host%% in conf.yaml of

Describe the results you received: are not able to scrap metric from host endpoint as url does not translate into valid url host variable does not get replaced

Describe the results you expected: connect to host endpoint and scrap metric from host

Additional information you deem important (e.g. issue happens only occasionally): based on https://www.datadoghq.com/blog/monitor-prometheus-metrics/ the %%host%% should be a valid variable

is there another way to use the env variable DD_KUBERNETES_KUBELET_HOST as target host to scrap metrics from?

fanny-jiang commented 2 years ago

Hi @thomas-weishaupl-rft, thanks for reaching out! You might be missing the ad_identifiers: section in the conf.yaml files, which is used to define container identifiers for the agent to properly interpolate the template variables and run the check. Please see our autodiscovery documentation for files here: https://docs.datadoghq.com/containers/kubernetes/integrations/?tab=file

To investigate your issue further, can you please open a support ticket? I'll close this issue for now while our Support team assists you.