Open axot opened 4 years ago
Output of the info page (if this is a bug)
# agent status Getting the status from the agent. =============== Agent (v6.14.1) =============== Status date: 2019-11-25 08:04:57.021342 UTC Agent start: 2019-11-05 04:05:44.527832 UTC Pid: 1 Go Version: go1.12.9 Python Version: 2.7.16 Check Runners: 4 Log Level: WARN Paths ===== Config File: /etc/datadog-agent/datadog.yaml conf.d: /etc/datadog-agent/conf.d checks.d: /etc/datadog-agent/checks.d Clocks ====== NTP offset: -53µs System UTC time: 2019-11-25 08:04:57.021342 UTC Host Info ========= bootTime: 2019-11-05 03:11:53.000000 UTC kernelVersion: 4.15.0-1034-gke os: linux platform: debian platformFamily: debian platformVersion: 10.1 procs: 60 uptime: 53m53s Hostnames ========= host_aliases: [gke-xxx-test-app-0-app-red-0-2c62eb0b-jlpf.yyy gke-xxx-test-app-0-app-red-0-2c62eb0b-jlpf-xxx-test-app-0] hostname: gke-xxx-test-app-0-app-red-0-2c62eb0b-jlpf.c.yyy.internal socket-fqdn: datadog-f49ddd458-6zk92 socket-hostname: datadog-f49ddd458-6zk92 host tags: gke-xxx-test-app-0-4060f4d1-node prd-app zone:asia-northeast1-b instance-type:n1-standard-64 internal-hostname:gke-xxx-test-app-0-app-red-0-2c62eb0b-jlpf.c.yyy.internal instance-id:5444726203643418295 project:yyy numeric_project_id:824996079768 cluster-location:asia-northeast1 gci-ensure-gke-docker:true cluster-name:xxx-test-app-0 gci-update-strategy:update_disabled kube-labels:beta.kubernetes.io/fluentd-ds-ready=true,cloud.google.com/gke-nodepool=app-red-0,cloud.google.com/gke-os-distribution=ubuntu,server_group=red created-by:projects/824996079768/zones/asia-northeast1-b/instanceGroupManagers/gke-xxx-test-app-0-app-red-0-2c62eb0b-grp enable-oslogin:false cluster-uid:4060f4d13b73b0d6375b30d37b5c05676cd2eaae63ce5cdcc73ab215d3fa3584 disable-legacy-endpoints:true instance-template:projects/824996079768/global/instanceTemplates/gke-xxx-test-app-0-app-red-0-2c62eb0b hostname provider: gce unused hostname providers: configuration/environment: hostname is empty ========= Collector ========= Running Checks ============== cpu --- Instance ID: cpu [OK] Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default Total Runs: 116,156 Metric Samples: Last Run: 6, Total: 696,930 Events: Last Run: 0, Total: 0 Service Checks: Last Run: 0, Total: 0 Average Execution Time : 0s disk (2.5.0) ------------ Instance ID: disk:89011cceb1f16288 [OK] Configuration Source: file:/etc/datadog-agent/conf.d/disk.yaml Total Runs: 116,157 Metric Samples: Last Run: 200, Total: 1 M Events: Last Run: 0, Total: 0 Service Checks: Last Run: 0, Total: 0 Average Execution Time : 95ms docker ------ Instance ID: docker [OK] Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default Total Runs: 116,157 Metric Samples: Last Run: 578, Total: 1 M Events: Last Run: 0, Total: 1,909 Service Checks: Last Run: 1, Total: 116,157 Average Execution Time : 135ms file_handle ----------- Instance ID: file_handle [OK] Configuration Source: file:/etc/datadog-agent/conf.d/file_handle.d/conf.yaml.default Total Runs: 116,156 Metric Samples: Last Run: 5, Total: 580,780 Events: Last Run: 0, Total: 0 Service Checks: Last Run: 0, Total: 0 Average Execution Time : 0s io -- Instance ID: io [OK] Configuration Source: file:/etc/datadog-agent/conf.d/io.d/conf.yaml.default Total Runs: 116,157 Metric Samples: Last Run: 65, Total: 1 M Events: Last Run: 0, Total: 0 Service Checks: Last Run: 0, Total: 0 Average Execution Time : 0s kubelet (3.3.2) --------------- Instance ID: kubelet:d884b5186b651429 [OK] Configuration Source: file:/etc/datadog-agent/conf.d/kubelet.d/conf.yaml.default Total Runs: 116,156 Metric Samples: Last Run: 628, Total: 1 M Events: Last Run: 0, Total: 0 Service Checks: Last Run: 4, Total: 464,624 Average Execution Time : 375ms load ---- Instance ID: load [OK] Configuration Source: file:/etc/datadog-agent/conf.d/load.d/conf.yaml.default Total Runs: 116,157 Metric Samples: Last Run: 6, Total: 696,942 Events: Last Run: 0, Total: 0 Service Checks: Last Run: 0, Total: 0 Average Execution Time : 0s memory ------ Instance ID: memory [OK] Configuration Source: file:/etc/datadog-agent/conf.d/memory.d/conf.yaml.default Total Runs: 116,156 Metric Samples: Last Run: 17, Total: 1 M Events: Last Run: 0, Total: 0 Service Checks: Last Run: 0, Total: 0 Average Execution Time : 0s network (1.11.4) ---------------- Instance ID: network:e0204ad63d43c949 [OK] Configuration Source: file:/etc/datadog-agent/conf.d/network.d/conf.yaml.default Total Runs: 116,157 Metric Samples: Last Run: 79, Total: 1 M Events: Last Run: 0, Total: 0 Service Checks: Last Run: 0, Total: 0 Average Execution Time : 2ms ntp --- Instance ID: ntp:7e1f812862c38a66 [OK] Configuration Source: file:/etc/datadog-agent/conf.d/ntp.yaml Total Runs: 116,157 Metric Samples: Last Run: 1, Total: 116,157 Events: Last Run: 0, Total: 0 Service Checks: Last Run: 1, Total: 116,157 Average Execution Time : 0s uptime ------ Instance ID: uptime [OK] Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default Total Runs: 116,157 Metric Samples: Last Run: 1, Total: 116,157 Events: Last Run: 0, Total: 0 Service Checks: Last Run: 0, Total: 0 Average Execution Time : 0s ======== JMXFetch ======== Initialized checks ================== no checks Failed checks ============= no checks ========= Forwarder ========= Transactions ============ CheckRunsV1: 116,156 Dropped: 0 DroppedOnInput: 0 Events: 0 HostMetadata: 0 IntakeV1: 10,447 Metadata: 0 Requeued: 1 Retried: 1 RetryQueueSize: 0 Series: 0 ServiceChecks: 0 SketchSeries: 0 Success: 242,759 TimeseriesV1: 116,156 Transaction Errors ================== Total number: 1 Errors By Type: API Keys status =============== API key ending with 2c40c: API Key valid ========== Endpoints ========== https://app.datadoghq.com - API Key ending with: - 2c40c ========== Logs Agent ========== Logs Agent is not running ========= Aggregator ========= Checks Metric Sample: 176.4 M Dogstatsd Metric Sample: 21.7 M Event: 1,910 Events Flushed: 1,910 Number Of Flushes: 116,156 Series Flushed: 168.1 M Service Check: 1.8 M Service Checks Flushed: 1.9 M ========= DogStatsD ========= Event Packets: 0 Event Parse Errors: 0 Metric Packets: 21.7 M Metric Parse Errors: 0 Service Check Packets: 0 Service Check Parse Errors: 0 Udp Bytes: 1.9 G Udp Packet Reading Errors: 0 Udp Packets: 21.7 M Uds Bytes: 0 Uds Origin Detection Errors: 0 Uds Packet Reading Errors: 0 Uds Packets: 0
Describe what happened: datadog agent showed http: Accept error: request has been rate-limited, and then we can not see any apm events.
Describe what you expected: We want to know the root cause and how to solve it.
Steps to reproduce the issue:
Additional environment details (Operating System, Cloud provider, etc): datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:07:52.624988201Z 2019-11-25 06:07:52 UTC | TRACE | WARN | (pkg/trace/info/stats.go:111 in LogStats) | [lang:php lang_version:7.2.24 interpreter:fpm-fcgi tracer_version:0.33.0] -> traces_dropped(decoding_error:55). Enable debug logging for more details. datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:07:52.625030217Z 2019-11-25 06:07:52 UTC | TRACE | WARN | (pkg/trace/api/api.go:518 in watchdog) | CPU threshold exceeded (apm_config.max_cpu_percent: 50): 1 datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:02.624100679Z 2019-11-25 06:08:02 UTC | TRACE | WARN | (pkg/trace/api/api.go:518 in watchdog) | CPU threshold exceeded (apm_config.max_cpu_percent: 50): 1 datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.297445136Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 5ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.302707184Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 10ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.313077695Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 20ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.333479258Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 40ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.373653815Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 80ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.454073332Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 160ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.614451933Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 320ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.934798176Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 640ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:08.575108959Z 2019-11-25 06:08:08 UTC | TRACE | ERROR | (pkg/trace/agent/log.go:65 in ReceiveMessage) | Too many messages to log, skipping for a bit... datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:12.578466285Z 2019-11-25 06:08:12 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 1s datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:13.579612403Z 2019-11-25 06:08:13 UTC | TRACE | ERROR | (pkg/trace/api/api.go:372 in handleTraces) | Cannot decode v0.4 traces payload: unexpected EOF datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:13.579646122Z 2019-11-25 06:08:13 UTC | TRACE | ERROR | (pkg/trace/api/api.go:372 in handleTraces) | Cannot decode v0.4 traces payload: unexpected EOF
@axot was this datadog limiting you? We are running into something similar now.
https://docs.datadoghq.com/tracing/troubleshooting/?tab=java#max-connection-limit This is something that I found that might help you out.
Output of the info page (if this is a bug)
Describe what happened: datadog agent showed http: Accept error: request has been rate-limited, and then we can not see any apm events.
Describe what you expected: We want to know the root cause and how to solve it.
Steps to reproduce the issue:
Additional environment details (Operating System, Cloud provider, etc): datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:07:52.624988201Z 2019-11-25 06:07:52 UTC | TRACE | WARN | (pkg/trace/info/stats.go:111 in LogStats) | [lang:php lang_version:7.2.24 interpreter:fpm-fcgi tracer_version:0.33.0] -> traces_dropped(decoding_error:55). Enable debug logging for more details. datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:07:52.625030217Z 2019-11-25 06:07:52 UTC | TRACE | WARN | (pkg/trace/api/api.go:518 in watchdog) | CPU threshold exceeded (apm_config.max_cpu_percent: 50): 1 datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:02.624100679Z 2019-11-25 06:08:02 UTC | TRACE | WARN | (pkg/trace/api/api.go:518 in watchdog) | CPU threshold exceeded (apm_config.max_cpu_percent: 50): 1 datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.297445136Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 5ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.302707184Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 10ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.313077695Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 20ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.333479258Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 40ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.373653815Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 80ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.454073332Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 160ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.614451933Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 320ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:07.934798176Z 2019-11-25 06:08:07 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 640ms datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:08.575108959Z 2019-11-25 06:08:08 UTC | TRACE | ERROR | (pkg/trace/agent/log.go:65 in ReceiveMessage) | Too many messages to log, skipping for a bit... datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:12.578466285Z 2019-11-25 06:08:12 UTC | TRACE | ERROR | (pkg/trace/api/api.go:616 in Write) | http.Server: http: Accept error: request has been rate-limited; retrying in 1s datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:13.579612403Z 2019-11-25 06:08:13 UTC | TRACE | ERROR | (pkg/trace/api/api.go:372 in handleTraces) | Cannot decode v0.4 traces payload: unexpected EOF datadog-f49ddd458-6zk92 trace-agent 2019-11-25T06:08:13.579646122Z 2019-11-25 06:08:13 UTC | TRACE | ERROR | (pkg/trace/api/api.go:372 in handleTraces) | Cannot decode v0.4 traces payload: unexpected EOF