search

DataDog / dd-agent

Datadog Agent Version 5
https://docs.datadoghq.com/
Other
1.3k stars 812 forks source link

sampleRate from dd-trace-js is not taken into account in statistics #3897

Closed millerick closed 4 years ago

millerick commented 4 years ago

Output of the info page 

Getting the status from the agent.

2020-10-13 18:03:10 UTC | CORE | DEBUG | (pkg/util/log/log.go:473 in func1) | cloud_provider_metadata is set to [aws gcp azure alibaba] in agent configuration, trying endpoints for AWS Cloud Provider
2020-10-13 18:03:10 UTC | CORE | DEBUG | (pkg/util/log/log.go:473 in func1) | cloud_provider_metadata is set to [aws gcp azure alibaba] in agent configuration, trying endpoints for AWS Cloud Provider
===============
Agent (v7.22.1)
===============

  Status date: 2020-10-13 18:03:10.106111 UTC
  Agent start: 2020-10-02 03:15:54.198392 UTC
  Pid: 383
  Go Version: go1.13.11
  Python Version: 3.8.5
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: debug

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: -2.713ms
    System UTC time: 2020-10-13 18:03:10.106111 UTC

  Host Info
  =========
    bootTime: 2020-09-24 13:11:48.000000 UTC
    kernelArch: x86_64
    kernelVersion: 4.4.0-1101-aws
    os: linux
    platform: debian
    platformFamily: debian
    platformVersion: bullseye/sid
    procs: 555
    uptime: 182h4m13s
    virtualizationRole: guest
    virtualizationSystem: xen

  Hostnames
  =========
    ec2-hostname: ip-10-10-3-92.ec2.internal
    host_aliases: [ip-10-10-3-92.ec2.internal]
    hostname: ip-10-10-3-92.ec2.internal
    instance-id: i-0a3254cd1fb05093a
    socket-fqdn: dd-agent-lsbx2
    socket-hostname: dd-agent-lsbx2
    host tags:
      kubernetescluster:sandbox.k8s.centrio.com
      cluster_name:sandbox.k8s.centrio.com
      environment:sandbox
      cluster_type:kubernetes
      kube_node_role:node
    hostname provider: configuration

  Metadata
  ========
    cloud_provider: AWS
    hostname_source: configuration

=========
Collector
=========

  Running Checks
  ==============

    cpu
    ---
      Instance ID: cpu [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default
      Total Runs: 66,909
      Metric Samples: Last Run: 7, Total: 468,357
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-10-13 18:03:00.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:00.000000 UTC

    disk (2.10.1)
    -------------
      Instance ID: disk:e5dffb8bef24336f [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/disk.d/conf.yaml.default
      Total Runs: 66,909
      Metric Samples: Last Run: 286, Total: 19,143,170
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 55ms
      Last Execution Date : 2020-10-13 18:03:07.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:07.000000 UTC

    docker
    ------
      Instance ID: docker [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default
      Total Runs: 66,908
      Metric Samples: Last Run: 3,042, Total: 168,953,779
      Events: Last Run: 0, Total: 4,971
      Service Checks: Last Run: 1, Total: 66,908
      Average Execution Time : 1.358s
      Last Execution Date : 2020-10-13 18:03:00.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:00.000000 UTC

    etcd (2.4.1)
    ------------
      Instance ID: etcd:e6cc2959f8f11055 [OK]
      Configuration Source: kubelet:docker://2e8bbfe98827f98f6a944ac127d5dc788d5ac939bf891f2670d953568609c586
      Total Runs: 66,909
      Metric Samples: Last Run: 1,053, Total: 70,455,177
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 66,909
      Average Execution Time : 268ms
      Last Execution Date : 2020-10-13 18:03:01.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:01.000000 UTC
      metadata:
        version.major: 3
        version.minor: 3
        version.patch: 1
        version.raw: 3.3.1
        version.scheme: semver

    file_handle
    -----------
      Instance ID: file_handle [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/file_handle.d/conf.yaml.default
      Total Runs: 66,909
      Metric Samples: Last Run: 5, Total: 334,545
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-10-13 18:03:06.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:06.000000 UTC

    fluentd (1.9.1)
    ---------------
      Instance ID: fluentd:2323b6cd70c8e638 [OK]
      Configuration Source: kubelet:docker://122b0dcc018208c5f16519e50d236d5755bbf7d16c4ebd9d45f720c8dd0ae998
      Total Runs: 66,909
      Metric Samples: Last Run: 5, Total: 334,545
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 66,909
      Average Execution Time : 46ms
      Last Execution Date : 2020-10-13 18:03:09.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:09.000000 UTC
      metadata:
        version.major: 1
        version.minor: 11
        version.patch: 2
        version.raw: 1.11.2
        version.scheme: semver

    http_check (4.10.0)
    -------------------
      Instance ID: http_check:etcd:555f04d8819efe8d [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/http_check.yaml
      Total Runs: 66,908
      Metric Samples: Last Run: 5, Total: 334,540
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 2, Total: 133,816
      Average Execution Time : 121ms
      Last Execution Date : 2020-10-13 18:02:59.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:02:59.000000 UTC

      Instance ID: http_check:registry:70f468b5e2e82a72 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/http_check.yaml
      Total Runs: 66,909
      Metric Samples: Last Run: 5, Total: 334,545
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 2, Total: 133,818
      Average Execution Time : 82ms
      Last Execution Date : 2020-10-13 18:03:00.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:00.000000 UTC

      Instance ID: http_check:vault:78f6fdb97c23bf67 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/http_check.yaml
      Total Runs: 66,909
      Metric Samples: Last Run: 5, Total: 334,545
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 2, Total: 133,818
      Average Execution Time : 32ms
      Last Execution Date : 2020-10-13 18:03:07.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:07.000000 UTC

    io
    --
      Instance ID: io [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/io.d/conf.yaml.default
      Total Runs: 66,908
      Metric Samples: Last Run: 104, Total: 7,005,138
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 1ms
      Last Execution Date : 2020-10-13 18:02:58.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:02:58.000000 UTC

    istio (3.5.0)
    -------------
      Instance ID: istio:24171b3af241dcac [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 4,924
      Metric Samples: Last Run: 266, Total: 1,278,506
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 314ms
      Last Execution Date : 2020-10-13 18:03:09.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:09.000000 UTC

      Instance ID: istio:2eafd24a193bbdad [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 61,808
      Metric Samples: Last Run: 136, Total: 8,405,376
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 277ms
      Last Execution Date : 2020-10-13 18:03:01.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:01.000000 UTC

      Instance ID: istio:4987ffa6a81cb879 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 4,926
      Metric Samples: Last Run: 266, Total: 1,278,648
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 219ms
      Last Execution Date : 2020-10-13 18:02:55.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:02:55.000000 UTC

      Instance ID: istio:54f741acc44bba44 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 66,908
      Metric Samples: Last Run: 266, Total: 17,662,518
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 175ms
      Last Execution Date : 2020-10-13 18:02:58.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:02:58.000000 UTC

      Instance ID: istio:551564bf368b27c8 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 12
      Metric Samples: Last Run: 128, Total: 1,536
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 99ms
      Last Execution Date : 2020-10-13 18:03:04.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:04.000000 UTC

      Instance ID: istio:5883e5729525841f [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 61,809
      Metric Samples: Last Run: 136, Total: 8,405,384
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 304ms
      Last Execution Date : 2020-10-13 18:03:09.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:09.000000 UTC

      Instance ID: istio:5a7d66fad1b0c7a1 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 689
      Metric Samples: Last Run: 8, Total: 5,512
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 76ms
      Last Execution Date : 2020-10-13 18:03:02.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:02.000000 UTC

      Instance ID: istio:60d02d654582df26 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 692
      Metric Samples: Last Run: 8, Total: 5,536
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 147ms
      Last Execution Date : 2020-10-13 18:02:55.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:02:55.000000 UTC

      Instance ID: istio:6eabd3d89f43ec89 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 66,909
      Metric Samples: Last Run: 128, Total: 8,564,352
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 180ms
      Last Execution Date : 2020-10-13 18:03:05.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:05.000000 UTC

      Instance ID: istio:947b2c27fbf4d79b [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 66,908
      Metric Samples: Last Run: 142, Total: 9,500,936
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 126ms
      Last Execution Date : 2020-10-13 18:02:57.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:02:57.000000 UTC

      Instance ID: istio:9ed3860d14bfc6f [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 66,909
      Metric Samples: Last Run: 142, Total: 9,501,078
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 310ms
      Last Execution Date : 2020-10-13 18:03:01.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:01.000000 UTC

      Instance ID: istio:aea0971ba1098ee0 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 66,908
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 94ms
      Last Execution Date : 2020-10-13 18:02:56.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:02:56.000000 UTC

      Instance ID: istio:dd86f29b714c010e [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 23,024
      Metric Samples: Last Run: 190, Total: 7,286,640
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 359ms
      Last Execution Date : 2020-10-13 18:03:09.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:09.000000 UTC

      Instance ID: istio:f297d5631c71a9db [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 23,009
      Metric Samples: Last Run: 190, Total: 7,283,620
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 171ms
      Last Execution Date : 2020-10-13 18:02:59.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:02:59.000000 UTC

    kubelet (4.1.1)
    ---------------
      Instance ID: kubelet:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/kubelet.d/conf.yaml.default
      Total Runs: 66,901
      Metric Samples: Last Run: 2,710, Total: 151,295,704
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 4, Total: 267,604
      Average Execution Time : 1.048s
      Last Execution Date : 2020-10-13 18:03:06.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:06.000000 UTC

    load
    ----
      Instance ID: load [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/load.d/conf.yaml.default
      Total Runs: 66,908
      Metric Samples: Last Run: 6, Total: 401,448
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-10-13 18:02:57.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:02:57.000000 UTC

    memory
    ------
      Instance ID: memory [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/memory.d/conf.yaml.default
      Total Runs: 66,909
      Metric Samples: Last Run: 18, Total: 1,204,362
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-10-13 18:03:04.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:04.000000 UTC

    network (1.17.0)
    ----------------
      Instance ID: network:5c571333f400457d [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/network.d/conf.yaml.default
      Total Runs: 66,908
      Metric Samples: Last Run: 31, Total: 2,074,148
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 9ms
      Last Execution Date : 2020-10-13 18:02:56.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:02:56.000000 UTC

    nginx (3.8.1)
    -------------
      Instance ID: nginx:bc19242fd3bdbe8d [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/nginx.yaml
      Total Runs: 66,909
      Metric Samples: Last Run: 7, Total: 468,363
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 66,909
      Average Execution Time : 12ms
      Last Execution Date : 2020-10-13 18:03:08.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:08.000000 UTC
      metadata:
        version.major: 1
        version.minor: 15
        version.patch: 8
        version.raw: 1.15.8
        version.scheme: semver

    ntp
    ---
      Instance ID: ntp:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/ntp.d/conf.yaml.default
      Total Runs: 1,116
      Metric Samples: Last Run: 1, Total: 1,116
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 1,116
      Average Execution Time : 859ms
      Last Execution Date : 2020-10-13 18:01:01.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:01:01.000000 UTC

    openmetrics (1.9.0)
    -------------------
      Instance ID: openmetrics:node_exporter:d2f022cea06294a5 [OK]
      Configuration Source: kubelet:docker://aef8dc9dc234c6a99fc5a75cc075f67b0554b113cd994daea86ab1fbb1159476
      Total Runs: 66,909
      Metric Samples: Last Run: 6, Total: 401,454
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 66,909
      Average Execution Time : 136ms
      Last Execution Date : 2020-10-13 18:03:02.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:02.000000 UTC

    prometheus (3.3.0)
    ------------------
      Instance ID: prometheus:calico:4f701ba9713d0d8 [OK]
      Configuration Source: kubelet:docker://64f5befc751e6900f8c8646b0e6b5b87326a7ac8f8e3e84668497bee23d3522b
      Total Runs: 66,908
      Metric Samples: Last Run: 148, Total: 9,611,384
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 66,908
      Average Execution Time : 60ms
      Last Execution Date : 2020-10-13 18:02:55.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:02:55.000000 UTC

    uptime
    ------
      Instance ID: uptime [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
      Total Runs: 66,909
      Metric Samples: Last Run: 1, Total: 66,909
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-10-13 18:03:03.000000 UTC
      Last Successful Execution Date : 2020-10-13 18:03:03.000000 UTC

  Check Initialization Errors
  ===========================

      etcd (2.4.1)
      ------------

      instance 0:

        could not invoke 'etcd' python check constructor. New constructor API returned:
Traceback (most recent call last):
  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/etcd/etcd.py", line 86, in __init__
    super(Etcd, self).__init__(
  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 100, in __init__
    self.get_scraper_config(instance)
  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 122, in get_scraper_config
    raise CheckException("Unable to find prometheus URL in config file.")
datadog_checks.base.errors.CheckException: Unable to find prometheus URL in config file.
Deprecated constructor API returned:
__init__() got an unexpected keyword argument 'agentConfig'
  Loading Errors
  ==============
    etcd
    ----
      Core Check Loader:
        Check etcd not found in Catalog

      JMX Check Loader:
        check is not a jmx check, or unable to determine if it's so

      Python Check Loader:
        could not configure check instance for python check etcd: could not invoke 'etcd' python check constructor. New constructor API returned:
Traceback (most recent call last):
  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/etcd/etcd.py", line 86, in __init__
    super(Etcd, self).__init__(
  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 100, in __init__
    self.get_scraper_config(instance)
  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 122, in get_scraper_config
    raise CheckException("Unable to find prometheus URL in config file.")
datadog_checks.base.errors.CheckException: Unable to find prometheus URL in config file.
Deprecated constructor API returned:
__init__() got an unexpected keyword argument 'agentConfig'

========
JMXFetch
========

  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    CheckRunsV1: 66,909
    Connections: 0
    Containers: 0
    Deployments: 0
    Dropped: 0
    DroppedOnInput: 0
    Events: 0
    HostMetadata: 0
    IntakeV1: 10,368
    Metadata: 0
    Pods: 0
    Processes: 0
    RTContainers: 0
    RTProcesses: 0
    ReplicaSets: 0
    Requeued: 2
    Retried: 2
    RetryQueueSize: 0
    Series: 0
    ServiceChecks: 0
    Services: 0
    SketchSeries: 0
    Success: 275,022
    TimeseriesV1: 66,909

  Transaction Errors
  ==================
    Total number: 1
    Errors By Type:

  API Keys status
  ===============
    API key ending with 05c5f: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.com - API Key ending with:
      - 05c5f

==========
Logs Agent
==========

  Logs Agent is not running

=========
APM Agent
=========
  Status: Running
  Pid: 386
  Uptime: 1.003636e+06 seconds
  Mem alloc: 42,797,264 bytes
  Hostname: ip-10-10-3-92.ec2.internal
  Receiver: 0.0.0.0:8126
  Endpoints:
    https://trace.agent.datadoghq.com

  Receiver (previous minute)
  ==========================
    From go 1.14.7 (gc-amd64-linux), client v1.5.0
      Traces received: 13 (6,655 bytes)
      Spans received: 19

    From nodejs v12.14.1 (v8), client 0.20.3
      Traces received: 365 (3,488,319 bytes)
      Spans received: 7277

    From go 1.14.6 (gc-amd64-linux), client v1.2
      Traces received: 282 (115,761 bytes)
      Spans received: 289

    From go 1.14.7 (gc-amd64-linux), client v1.20.1
      Traces received: 864 (436,837 bytes)
      Spans received: 864

    From go 1.15.2 (gc-amd64-linux), client v1.2
      Traces received: 357 (256,525 bytes)
      Spans received: 613

    From nodejs v12.13.1 (v8), client 0.20.3
      Traces received: 226 (866,516 bytes)
      Spans received: 1562

    From nodejs v12.14.1 (v8), client 0.15.4
      Traces received: 39 (89,636 bytes)
      Spans received: 345

    Default priority sampling rate: 9.5%
    Priority sampling rate for 'service:blend-backend-http-client,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-backend-mongodb,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-backend-tcp,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-express,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-highpriority-aws-sqs,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-highpriority-http-client,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-highpriority-mongodb,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-highpriority-tcp,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-highresource-aws-sqs,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-highresource-http-client,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-highresource-mongodb,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-mediumpriority-http-client,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-mediumpriority-mongodb,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-mediumpriority-tcp,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-retry-aws-sqs,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-retry-http-client,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-retry-mongodb,env:sandbox': 100.0%
    Priority sampling rate for 'service:blend-retry-tcp,env:sandbox': 100.0%
    Priority sampling rate for 'service:borrower-sync-express,env:sandbox': 100.0%
    Priority sampling rate for 'service:borrower-sync-http-client,env:sandbox': 100.0%
    Priority sampling rate for 'service:chantest-proxy,env:sandbox': 100.0%
    Priority sampling rate for 'service:coordinate-organizer-express,env:sandbox': 100.0%
    Priority sampling rate for 'service:coordinate-organizer-proxy,env:sandbox': 100.0%
    Priority sampling rate for 'service:disclosures-worker-high-priority-aws-sqs,env:sandbox': 100.0%
    Priority sampling rate for 'service:files-worker,env:sandbox': 100.0%
    Priority sampling rate for 'service:files-worker-aws-sqs,env:sandbox': 100.0%
    Priority sampling rate for 'service:files-worker-http-client,env:sandbox': 100.0%
    Priority sampling rate for 'service:files-worker-mongodb,env:sandbox': 100.0%
    Priority sampling rate for 'service:home-insurance-custom-workflow-proxy,env:sandbox': 100.0%
    Priority sampling rate for 'service:movement-fees-express,env:sandbox': 100.0%
    Priority sampling rate for 'service:orgchart-dns,env:sandbox': 100.0%
    Priority sampling rate for 'service:orgchart-express,env:sandbox': 100.0%
    Priority sampling rate for 'service:orgchart-tcp,env:sandbox': 100.0%
    Priority sampling rate for 'service:osquery-fleet-proxy,env:sandbox': 13.2%

  Writer (previous minute)
  ========================
    Traces: 0 payloads, 0 traces, 0 events, 0 bytes
    Stats: 0 payloads, 0 stats buckets, 0 bytes

=========
Aggregator
=========
  Checks Metric Sample: 585,893,449
  Dogstatsd Metric Sample: 382,416,336
  Event: 6,137
  Events Flushed: 6,137
  Number Of Flushes: 66,909
  Series Flushed: 654,396,032
  Service Check: 3,330,579
  Service Checks Flushed: 3,397,488

=========
DogStatsD
=========
  Event Packets: 1,165
  Event Parse Errors: 37
  Metric Packets: 382,416,335
  Metric Parse Errors: 39
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 27,264,962,121
  Udp Packet Reading Errors: 0
  Udp Packets: 66,836,310
  Uds Bytes: 36,193,786,796
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 31,058,713

=====================
Datadog Cluster Agent
=====================

  - Datadog Cluster Agent endpoint detected: https://10.101.17.200:5005
  Successfully connected to the Datadog Cluster Agent.
  - Running: 1.8.0+commit.934d5c5

Additional environment details (Operating System, Cloud provider, etc): AWS, Kube running on EC2, Ubuntu 16.04

Steps to reproduce the issue:

  1. Send traces from dd-trace-js with sampleRate: 1
  2. Reduce the sampleRate by a factor of 4: sampleRate: 0.25

Describe the results you received: The number of hits registered by APM increased by a factor of four, even though there was no change to traffic pattern of the service. image

Describe the results you expected: There should not have been a change to the number of hits (within error). With sampleRate: 0.25, the service should be sending 25% the spans and the agent should adjust the statistics appropriately and there should not be a cliff visible like above (either upwards or downwards).

Additional information you deem important (e.g. issue happens only occasionally):

millerick commented 4 years ago

https://github.com/DataDog/datadog-agent/issues/6556