search

DataDog / datadog-agent

Main repository for Datadog Agent
https://docs.datadoghq.com/
Apache License 2.0
2.8k stars 1.18k forks source link

istio integration is reporting: an error scraping endpoint http://istiod.istio-system:15014/metrics #11765

Open JingnanJia opened 2 years ago

JingnanJia commented 2 years ago

Running Checks ：：：

istio (3.15.0)

  Instance ID: istio:2025de15087306ad [OK]
  Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
  Total Runs: 100
  Metric Samples: Last Run: 0, Total: 0
  Events: Last Run: 0, Total: 0
  Service Checks: Last Run: 0, Total: 0
  Average Execution Time : 8ms
  Last Execution Date : 2022-04-22 09:13:43 UTC (1650618823000)
  Last Successful Execution Date : 2022-04-22 09:13:43 UTC (1650618823000)

  Instance ID: istio:56779aab59a544fb [OK]
  Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
  Total Runs: 99
  Metric Samples: Last Run: 0, Total: 0
  Events: Last Run: 0, Total: 0
  Service Checks: Last Run: 0, Total: 0
  Average Execution Time : 26ms
  Last Execution Date : 2022-04-22 09:13:36 UTC (1650618816000)
  Last Successful Execution Date : 2022-04-22 09:13:36 UTC (1650618816000)

  Instance ID: istio:8729cd9886ab901a [OK]
  Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
  Total Runs: 99
  Metric Samples: Last Run: 0, Total: 0
  Events: Last Run: 0, Total: 0
  Service Checks: Last Run: 0, Total: 0
  Average Execution Time : 8ms
  Last Execution Date : 2022-04-22 09:13:44 UTC (1650618824000)
  Last Successful Execution Date : 2022-04-22 09:13:44 UTC (1650618824000)

  Instance ID: istio:91b6a082c13712ad [OK]
  Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
  Total Runs: 100
  Metric Samples: Last Run: 0, Total: 0
  Events: Last Run: 0, Total: 0
  Service Checks: Last Run: 0, Total: 0
  Average Execution Time : 7ms
  Last Execution Date : 2022-04-22 09:13:45 UTC (1650618825000)
  Last Successful Execution Date : 2022-04-22 09:13:45 UTC (1650618825000)

  Instance ID: istio:d9ee14fc721d19a6 [OK]
  Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
  Total Runs: 99
  Metric Samples: Last Run: 0, Total: 0
  Events: Last Run: 0, Total: 0
  Service Checks: Last Run: 0, Total: 0
  Average Execution Time : 7ms
  Last Execution Date : 2022-04-22 09:13:37 UTC (1650618817000)
  Last Successful Execution Date : 2022-04-22 09:13:37 UTC (1650618817000)

Describe what happened: When I installed istio on k8s and edited the conf.yaml file according to the configuration link(https://app.datadoghq.com/account/settings#integrations/istio)

[root@d]# kubectl get pod -n istio-system NAME READY STATUS RESTARTS AGE istio-egressgateway-585dfcbbdb-p5lnj 1/1 Running 0 24h istio-ingressgateway-6cd9bf697d-2zkr9 1/1 Running 0 24h istiod-c555c64c4-j5zrs 1/1 Running 0 24h

Datadog’s istio integration is reporting: Instance #istio:898bfcf1da407bd9[ERROR]: [{"message": "There was an error scraping endpoint http://istiod.istio-system:15014/metrics: HTTPConnectionPool(host='istiod.istio-system', port=15014): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb371e86370>: Failed to establish a new connection: [Errno -2] Name or service not known'))", "traceback": "Traceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 1071, in run\n self.check(instance)\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/v2/base.py\", line 67, in check\n raise_from(type(e)(\"There was an error scraping endpoint {}: {}\".format(endpoint, e)), None)\n File \"\", line 3, in raise_from\nrequests.exceptions.ConnectionError: There was an error scraping endpoint http://istiod.istio-system:15014/metrics: HTTPConnectionPool(host='istiod.istio-system', port=15014): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb371e86370>: Failed to establish a new connection: [Errno -2] Name or service not known'))\n"}]

k8s V1.16 istio V13.2

how to deal with this issue? thanks

andrein19 commented 1 month ago

This is related to your Datadog proxy configuration. Endpoint "istiod.istio-system" is not into $$DD_PROXY_NO_PROXY. so dd agent tries to go through proxy to reach the endpoint and throws 403.

Check $DD_PROXY_NO_PROXY. It should contain ".svc.cluster.local" . Hence you can either configure your scrape endpoint as "istio.istio-system.svc.cluster.com" or add "istio.istio-system" to $DD_PROXY_NO_PROXY which will allow agent to connect to respective endpoint.