search

DataDog / datadog-agent

Main repository for Datadog Agent
https://docs.datadoghq.com/
Apache License 2.0
2.87k stars 1.21k forks source link

[BUG] DD_AGENT_VERSION="7.42.1" Has slow memory leak #15532

Closed gonace closed 1 year ago

gonace commented 1 year ago

Output of the info page (if this is a bug) See below

Describe what happened: The agents are slowly using up memory until the server has no memory left: datadog

We updated to 7.42.1 2023-02-03 21:00:00 and then restarted the agent 2023-02-06 11:00:00 and 2023-02-07 07:00:00 as one can see in the graph above.

After about 10h the agent consumes about 4.1Gb datadog-agent

Describe what you expected: Memory usage should return to normal after bursty events.

Steps to reproduce the issue: Not 100% sure other than just running it.

Additional environment details (Operating System, Cloud provider, etc):

===============
Agent (v7.42.1)
===============

  Status date: 2023-02-10 10:29:16.214 CET / 2023-02-10 09:29:16.214 UTC (1676021356214)
  Agent start: 2023-02-09 10:16:30.483 CET / 2023-02-09 09:16:30.483 UTC (1675934190483)
  Pid: 30044
  Go Version: go1.18.9
  Python Version: 3.8.16
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: info

  Paths
  =====
    Config File: C:\ProgramData\Datadog\datadog.yaml
    conf.d: C:\ProgramData\Datadog\conf.d
    checks.d: C:\ProgramData\Datadog\checks.d

  Clocks
  ======
    NTP offset: -6.723ms
    System time: 2023-02-10 10:29:16.214 CET / 2023-02-10 09:29:16.214 UTC (1676021356214)

  Host Info
  =========
    bootTime: 2022-10-31 10:23:32 CET / 2022-10-31 09:23:32 UTC (1667208212000)
    kernelArch: amd64
    os: windows
    platform: Windows Server 2016 Datacenter
    platformFamily: Windows Server 2016 Datacenter
    platformVersion: 10.0 Build 14393
    procs: 119
    uptime: 2423h53m1s

  Hostnames
  =========
    hostname: web-server-01
    socket-fqdn: web-server-01
    socket-hostname: web-server-01
    host tags:
      availability-zone:cygate
    hostname provider: os
    unused hostname providers:
      'hostname' configuration/environment: hostname is empty
      'hostname_file' configuration/environment: 'hostname_file' configuration is not enabled
      aws: not retrieving hostname from AWS: the host is not an ECS instance and other providers already retrieve non-default hostnames
      azure: azure_hostname_style is set to 'os'
      container: the agent is not containerized
      fargate: agent is not runnning on Fargate
      fqdn: 'hostname_fqdn' configuration is not enabled
      gce: unable to retrieve hostname from GCE: GCE metadata API error: Get "http://169.254.169.254/computeMetadata/v1/instance/hostname": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

  Metadata
  ========
    agent_version: 7.42.1
    config_apm_dd_url: 
    config_dd_url: 
    config_logs_dd_url: 
    config_logs_socks5_proxy_address: 
    config_no_proxy: []
    config_process_dd_url: 
    config_proxy_http: 
    config_proxy_https: 
    config_site: 
    feature_apm_enabled: false
    feature_cspm_enabled: false
    feature_cws_enabled: false
    feature_logs_enabled: false
    feature_networks_enabled: true
    feature_networks_gotls_enabled: false
    feature_networks_http_enabled: false
    feature_networks_https_enabled: false
    feature_otlp_enabled: false
    feature_process_enabled: false
    feature_processes_container_enabled: false
    flavor: agent
    hostname_source: os
    install_method_installer_version: windows_msi_gui
    install_method_tool: windows_msi_gui
    install_method_tool_version: windows_msi_gui

=========
Collector
=========

  Running Checks
  ==============

    aspdotnet (1.12.0)
    ------------------
      Instance ID: aspdotnet:663d833d2ec171e0 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\aspdotnet.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 43, Total: 240,960
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 5,811
      Average Execution Time : 8ms
      Last Execution Date : 2023-02-10 10:29:09 CET / 2023-02-10 09:29:09 UTC (1676021349000)
      Last Successful Execution Date : 2023-02-10 10:29:09 CET / 2023-02-10 09:29:09 UTC (1676021349000)

    cpu
    ---
      Instance ID: cpu [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\cpu.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 8, Total: 46,480
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2023-02-10 10:29:01 CET / 2023-02-10 09:29:01 UTC (1676021341000)
      Last Successful Execution Date : 2023-02-10 10:29:01 CET / 2023-02-10 09:29:01 UTC (1676021341000)

    disk (4.8.0)
    ------------
      Instance ID: disk:67cc0574430a16ba [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\disk.d\conf.yaml.default
      Total Runs: 5,811
      Metric Samples: Last Run: 8, Total: 46,488
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 1ms
      Last Execution Date : 2023-02-10 10:29:13 CET / 2023-02-10 09:29:13 UTC (1676021353000)
      Last Successful Execution Date : 2023-02-10 10:29:13 CET / 2023-02-10 09:29:13 UTC (1676021353000)

    dotnetclr (1.13.0)
    ------------------
      Instance ID: dotnetclr:57862cd098f50d15 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\dotnetclr.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 16, Total: 93,774
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 5,810
      Average Execution Time : 11ms
      Last Execution Date : 2023-02-10 10:29:08 CET / 2023-02-10 09:29:08 UTC (1676021348000)
      Last Successful Execution Date : 2023-02-10 10:29:08 CET / 2023-02-10 09:29:08 UTC (1676021348000)

    file_handle
    -----------
      Instance ID: file_handle [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\file_handle.d\conf.yaml.default
      Total Runs: 5,810
      Metric Samples: Last Run: 1, Total: 5,810
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 6ms
      Last Execution Date : 2023-02-10 10:29:05 CET / 2023-02-10 09:29:05 UTC (1676021345000)
      Last Successful Execution Date : 2023-02-10 10:29:05 CET / 2023-02-10 09:29:05 UTC (1676021345000)

    iis (2.18.0)
    ------------
      Instance ID: iis:161b43e899fd6323 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 217, Total: 1,260,751
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 67, Total: 389,270
      Average Execution Time : 11ms
      Last Execution Date : 2023-02-10 10:29:01 CET / 2023-02-10 09:29:01 UTC (1676021341000)
      Last Successful Execution Date : 2023-02-10 10:29:01 CET / 2023-02-10 09:29:01 UTC (1676021341000)

      Instance ID: iis:23cf5314249b96f8 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 217, Total: 1,260,968
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 68, Total: 395,148
      Average Execution Time : 21ms
      Last Execution Date : 2023-02-10 10:29:15 CET / 2023-02-10 09:29:15 UTC (1676021355000)
      Last Successful Execution Date : 2023-02-10 10:29:15 CET / 2023-02-10 09:29:15 UTC (1676021355000)

      Instance ID: iis:244ff3735b500076 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 569, Total: 3,305,567
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 87, Total: 505,470
      Average Execution Time : 40ms
      Last Execution Date : 2023-02-10 10:29:03 CET / 2023-02-10 09:29:03 UTC (1676021343000)
      Last Successful Execution Date : 2023-02-10 10:29:03 CET / 2023-02-10 09:29:03 UTC (1676021343000)

      Instance ID: iis:2e60f948c3e21f26 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 239, Total: 1,388,552
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 68, Total: 395,080
      Average Execution Time : 21ms
      Last Execution Date : 2023-02-10 10:29:06 CET / 2023-02-10 09:29:06 UTC (1676021346000)
      Last Successful Execution Date : 2023-02-10 10:29:06 CET / 2023-02-10 09:29:06 UTC (1676021346000)

      Instance ID: iis:36a66ed67a277ee1 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 239, Total: 1,388,791
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 68, Total: 395,148
      Average Execution Time : 18ms
      Last Execution Date : 2023-02-10 10:29:12 CET / 2023-02-10 09:29:12 UTC (1676021352000)
      Last Successful Execution Date : 2023-02-10 10:29:12 CET / 2023-02-10 09:29:12 UTC (1676021352000)

      Instance ID: iis:554916a00570a744 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 217, Total: 1,260,968
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 67, Total: 389,337
      Average Execution Time : 20ms
      Last Execution Date : 2023-02-10 10:29:15 CET / 2023-02-10 09:29:15 UTC (1676021355000)
      Last Successful Execution Date : 2023-02-10 10:29:15 CET / 2023-02-10 09:29:15 UTC (1676021355000)

      Instance ID: iis:5f58721914542890 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 217, Total: 1,260,968
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 68, Total: 395,148
      Average Execution Time : 16ms
      Last Execution Date : 2023-02-10 10:29:13 CET / 2023-02-10 09:29:13 UTC (1676021353000)
      Last Successful Execution Date : 2023-02-10 10:29:13 CET / 2023-02-10 09:29:13 UTC (1676021353000)

      Instance ID: iis:69751c2c9e9526c0 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 217, Total: 1,260,968
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 67, Total: 389,337
      Average Execution Time : 18ms
      Last Execution Date : 2023-02-10 10:29:11 CET / 2023-02-10 09:29:11 UTC (1676021351000)
      Last Successful Execution Date : 2023-02-10 10:29:11 CET / 2023-02-10 09:29:11 UTC (1676021351000)

      Instance ID: iis:76c87f041d8ed28e [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 217, Total: 1,260,968
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 67, Total: 389,337
      Average Execution Time : 14ms
      Last Execution Date : 2023-02-10 10:29:11 CET / 2023-02-10 09:29:11 UTC (1676021351000)
      Last Successful Execution Date : 2023-02-10 10:29:11 CET / 2023-02-10 09:29:11 UTC (1676021351000)

      Instance ID: iis:8a5f812e34cf9bbe [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 569, Total: 3,306,136
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 87, Total: 505,557
      Average Execution Time : 36ms
      Last Execution Date : 2023-02-10 10:29:10 CET / 2023-02-10 09:29:10 UTC (1676021350000)
      Last Successful Execution Date : 2023-02-10 10:29:10 CET / 2023-02-10 09:29:10 UTC (1676021350000)

      Instance ID: iis:8c5abcf23c4e8ebe [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 217, Total: 1,260,751
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 67, Total: 389,270
      Average Execution Time : 22ms
      Last Execution Date : 2023-02-10 10:29:03 CET / 2023-02-10 09:29:03 UTC (1676021343000)
      Last Successful Execution Date : 2023-02-10 10:29:03 CET / 2023-02-10 09:29:03 UTC (1676021343000)

      Instance ID: iis:8d387df0b15aed2c [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 217, Total: 1,260,968
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 67, Total: 389,337
      Average Execution Time : 18ms
      Last Execution Date : 2023-02-10 10:29:13 CET / 2023-02-10 09:29:13 UTC (1676021353000)
      Last Successful Execution Date : 2023-02-10 10:29:13 CET / 2023-02-10 09:29:13 UTC (1676021353000)

      Instance ID: iis:8f38e6a050bbe0fa [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 217, Total: 1,260,751
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 67, Total: 389,270
      Average Execution Time : 21ms
      Last Execution Date : 2023-02-10 10:29:06 CET / 2023-02-10 09:29:06 UTC (1676021346000)
      Last Successful Execution Date : 2023-02-10 10:29:06 CET / 2023-02-10 09:29:06 UTC (1676021346000)

      Instance ID: iis:b55a2393cb28f0a2 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 239, Total: 1,388,552
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 68, Total: 395,080
      Average Execution Time : 16ms
      Last Execution Date : 2023-02-10 10:29:04 CET / 2023-02-10 09:29:04 UTC (1676021344000)
      Last Successful Execution Date : 2023-02-10 10:29:04 CET / 2023-02-10 09:29:04 UTC (1676021344000)

      Instance ID: iis:b5af41125478e29f [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 217, Total: 1,260,751
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 68, Total: 395,080
      Average Execution Time : 21ms
      Last Execution Date : 2023-02-10 10:29:05 CET / 2023-02-10 09:29:05 UTC (1676021345000)
      Last Successful Execution Date : 2023-02-10 10:29:05 CET / 2023-02-10 09:29:05 UTC (1676021345000)

      Instance ID: iis:b8bc00b3e32f2fcd [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 239, Total: 1,388,791
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 68, Total: 395,148
      Average Execution Time : 21ms
      Last Execution Date : 2023-02-10 10:29:14 CET / 2023-02-10 09:29:14 UTC (1676021354000)
      Last Successful Execution Date : 2023-02-10 10:29:14 CET / 2023-02-10 09:29:14 UTC (1676021354000)

      Instance ID: iis:b9d99f617d4f5590 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 239, Total: 1,388,791
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 68, Total: 395,148
      Average Execution Time : 29ms
      Last Execution Date : 2023-02-10 10:29:10 CET / 2023-02-10 09:29:10 UTC (1676021350000)
      Last Successful Execution Date : 2023-02-10 10:29:10 CET / 2023-02-10 09:29:10 UTC (1676021350000)

      Instance ID: iis:ce83c7767d4d3142 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 217, Total: 1,260,751
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 67, Total: 389,270
      Average Execution Time : 21ms
      Last Execution Date : 2023-02-10 10:29:07 CET / 2023-02-10 09:29:07 UTC (1676021347000)
      Last Successful Execution Date : 2023-02-10 10:29:07 CET / 2023-02-10 09:29:07 UTC (1676021347000)

      Instance ID: iis:cf87ea1bda806047 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 217, Total: 1,260,751
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 68, Total: 395,080
      Average Execution Time : 22ms
      Last Execution Date : 2023-02-10 10:29:07 CET / 2023-02-10 09:29:07 UTC (1676021347000)
      Last Successful Execution Date : 2023-02-10 10:29:07 CET / 2023-02-10 09:29:07 UTC (1676021347000)

      Instance ID: iis:d027a20dfd901b9a [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 217, Total: 1,260,751
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 67, Total: 389,270
      Average Execution Time : 19ms
      Last Execution Date : 2023-02-10 10:29:08 CET / 2023-02-10 09:29:08 UTC (1676021348000)
      Last Successful Execution Date : 2023-02-10 10:29:08 CET / 2023-02-10 09:29:08 UTC (1676021348000)

      Instance ID: iis:d387803644bffe2d [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 239, Total: 1,388,552
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 68, Total: 395,080
      Average Execution Time : 117ms
      Last Execution Date : 2023-02-10 10:29:02 CET / 2023-02-10 09:29:02 UTC (1676021342000)
      Last Successful Execution Date : 2023-02-10 10:29:02 CET / 2023-02-10 09:29:02 UTC (1676021342000)

      Instance ID: iis:ddc3698145b3f38b [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 283, Total: 1,644,154
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 71, Total: 412,510
      Average Execution Time : 26ms
      Last Execution Date : 2023-02-10 10:29:05 CET / 2023-02-10 09:29:05 UTC (1676021345000)
      Last Successful Execution Date : 2023-02-10 10:29:05 CET / 2023-02-10 09:29:05 UTC (1676021345000)

      Instance ID: iis:e34c1a7bead887f2 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 217, Total: 1,260,751
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 67, Total: 389,270
      Average Execution Time : 19ms
      Last Execution Date : 2023-02-10 10:29:04 CET / 2023-02-10 09:29:04 UTC (1676021344000)
      Last Successful Execution Date : 2023-02-10 10:29:04 CET / 2023-02-10 09:29:04 UTC (1676021344000)

      Instance ID: iis:eefdf36fce54777 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 217, Total: 1,260,968
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 67, Total: 389,337
      Average Execution Time : 12ms
      Last Execution Date : 2023-02-10 10:29:09 CET / 2023-02-10 09:29:09 UTC (1676021349000)
      Last Successful Execution Date : 2023-02-10 10:29:09 CET / 2023-02-10 09:29:09 UTC (1676021349000)

      Instance ID: iis:f1041d5345c397f4 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 217, Total: 1,260,968
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 67, Total: 389,337
      Average Execution Time : 20ms
      Last Execution Date : 2023-02-10 10:29:14 CET / 2023-02-10 09:29:14 UTC (1676021354000)
      Last Successful Execution Date : 2023-02-10 10:29:14 CET / 2023-02-10 09:29:14 UTC (1676021354000)

      Instance ID: iis:ffbddb35f5c8b08d [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\iis.d\conf.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 283, Total: 1,644,437
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 71, Total: 412,581
      Average Execution Time : 23ms
      Last Execution Date : 2023-02-10 10:29:12 CET / 2023-02-10 09:29:12 UTC (1676021352000)
      Last Successful Execution Date : 2023-02-10 10:29:12 CET / 2023-02-10 09:29:12 UTC (1676021352000)

    io
    --
      Instance ID: io [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\io.d\conf.yaml.default
      Total Runs: 5,811
      Metric Samples: Last Run: 14, Total: 81,354
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2023-02-10 10:29:12 CET / 2023-02-10 09:29:12 UTC (1676021352000)
      Last Successful Execution Date : 2023-02-10 10:29:12 CET / 2023-02-10 09:29:12 UTC (1676021352000)

    memory
    ------
      Instance ID: memory [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\memory.d\conf.yaml.default
      Total Runs: 5,810
      Metric Samples: Last Run: 17, Total: 98,770
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 7ms
      Last Execution Date : 2023-02-10 10:29:04 CET / 2023-02-10 09:29:04 UTC (1676021344000)
      Last Successful Execution Date : 2023-02-10 10:29:04 CET / 2023-02-10 09:29:04 UTC (1676021344000)

    network (2.9.3)
    ---------------
      Instance ID: network:27b29dba7a40e75c [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\network.d\conf.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 39, Total: 228,292
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 158ms
      Last Execution Date : 2023-02-10 10:29:02 CET / 2023-02-10 09:29:02 UTC (1676021342000)
      Last Successful Execution Date : 2023-02-10 10:29:02 CET / 2023-02-10 09:29:02 UTC (1676021342000)

    ntp
    ---
      Instance ID: ntp:3c427a42a70bbf8 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\ntp.d\conf.yaml.default
      Total Runs: 97
      Metric Samples: Last Run: 1, Total: 97
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 97
      Average Execution Time : 264ms
      Last Execution Date : 2023-02-10 10:16:43 CET / 2023-02-10 09:16:43 UTC (1676020603000)
      Last Successful Execution Date : 2023-02-10 10:16:43 CET / 2023-02-10 09:16:43 UTC (1676020603000)

    tcp_check (4.6.0)
    -----------------
      Instance ID: tcp_check:TCP-SQL-SERVER:cc09a8d421291e9 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\tcp_check.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 1, Total: 5,811
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 5,811
      Average Execution Time : 9ms
      Last Execution Date : 2023-02-10 10:29:09 CET / 2023-02-10 09:29:09 UTC (1676021349000)
      Last Successful Execution Date : 2023-02-10 10:29:09 CET / 2023-02-10 09:29:09 UTC (1676021349000)

    uptime
    ------
      Instance ID: uptime [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\uptime.d\conf.yaml.default
      Total Runs: 5,810
      Metric Samples: Last Run: 1, Total: 5,810
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2023-02-10 10:29:11 CET / 2023-02-10 09:29:11 UTC (1676021351000)
      Last Successful Execution Date : 2023-02-10 10:29:11 CET / 2023-02-10 09:29:11 UTC (1676021351000)

    winproc
    -------
      Instance ID: winproc [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\winproc.d\conf.yaml.default
      Total Runs: 5,810
      Metric Samples: Last Run: 2, Total: 11,620
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 7ms
      Last Execution Date : 2023-02-10 10:29:03 CET / 2023-02-10 09:29:03 UTC (1676021343000)
      Last Successful Execution Date : 2023-02-10 10:29:03 CET / 2023-02-10 09:29:03 UTC (1676021343000)

    wmi_check (1.15.1)
    ------------------
      Instance ID: wmi_check:20ba17fb5026f9c5 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\wmi_check.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 2, Total: 11,620
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 55ms
      Last Execution Date : 2023-02-10 10:29:01 CET / 2023-02-10 09:29:01 UTC (1676021341000)
      Last Successful Execution Date : 2023-02-10 10:29:01 CET / 2023-02-10 09:29:01 UTC (1676021341000)

      Instance ID: wmi_check:5a82328704fd7841 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\wmi_check.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 2, Total: 11,620
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 267ms
      Last Execution Date : 2023-02-10 10:29:09 CET / 2023-02-10 09:29:09 UTC (1676021349000)
      Last Successful Execution Date : 2023-02-10 10:29:09 CET / 2023-02-10 09:29:09 UTC (1676021349000)

      Instance ID: wmi_check:6b028c080de842db [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\wmi_check.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 1, Total: 5,811
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 280ms
      Last Execution Date : 2023-02-10 10:29:16 CET / 2023-02-10 09:29:16 UTC (1676021356000)
      Last Successful Execution Date : 2023-02-10 10:29:16 CET / 2023-02-10 09:29:16 UTC (1676021356000)

      Instance ID: wmi_check:976f5443f2d0a18e [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\wmi_check.yaml
      Total Runs: 5,811
      Metric Samples: Last Run: 2, Total: 11,622
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 62ms
      Last Execution Date : 2023-02-10 10:29:14 CET / 2023-02-10 09:29:14 UTC (1676021354000)
      Last Successful Execution Date : 2023-02-10 10:29:14 CET / 2023-02-10 09:29:14 UTC (1676021354000)

      Instance ID: wmi_check:cd01bc6a57cef893 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\wmi_check.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 1, Total: 5,810
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 293ms
      Last Execution Date : 2023-02-10 10:29:08 CET / 2023-02-10 09:29:08 UTC (1676021348000)
      Last Successful Execution Date : 2023-02-10 10:29:08 CET / 2023-02-10 09:29:08 UTC (1676021348000)

      Instance ID: wmi_check:e272eb6addfdd470 [OK]
      Configuration Source: file:C:\ProgramData\Datadog\conf.d\wmi_check.yaml
      Total Runs: 5,810
      Metric Samples: Last Run: 114, Total: 661,881
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 159ms
      Last Execution Date : 2023-02-10 10:29:06 CET / 2023-02-10 09:29:06 UTC (1676021346000)
      Last Successful Execution Date : 2023-02-10 10:29:06 CET / 2023-02-10 09:29:06 UTC (1676021346000)

========
JMXFetch
========

  Information
  ==================
  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    Cluster: 0
    ClusterRole: 0
    ClusterRoleBinding: 0
    CronJob: 0
    CustomResource: 0
    CustomResourceDefinition: 0
    DaemonSet: 0
    Deployment: 0
    Dropped: 0
    HighPriorityQueueFull: 0
    Ingress: 0
    Job: 0
    Namespace: 0
    Node: 0
    PersistentVolume: 0
    PersistentVolumeClaim: 0
    Pod: 0
    ReplicaSet: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Role: 0
    RoleBinding: 0
    Service: 0
    ServiceAccount: 0
    StatefulSet: 0

  Transaction Successes
  =====================
    Total number: 11960
    Successes By Endpoint:
      check_run_v1: 5,810
      intake: 195
      metadata_v1: 145
      series_v1: 5,810

  On-disk storage
  ===============
    On-disk storage is disabled. Configure `forwarder_storage_max_size_in_bytes` to enable it.

  API Keys status
  ===============
    API key ending with 58d0e: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.com - API Key ending with:
      - 58d0e

==========
Logs Agent
==========

  Logs Agent is not running

============
System Probe
============
  Status: Not running or unreachable
  Error: temporary failure in system-probe-util, will retry later: Get "http://localhost:3333/debug/stats": dial tcp [::1]:3333: connectex: No connection could be made because the target machine actively refused it.

=============
Process Agent
=============

  Version: 7.42.1
  Status date: 2023-02-10 10:29:18.121 CET / 2023-02-10 09:29:18.121 UTC (1676021358121)
  Process Agent Start: 2023-02-09 10:16:54.273 CET / 2023-02-09 09:16:54.273 UTC (1675934214273)
  Pid: 12552
  Go Version: go1.18.9
  Build arch: amd64
  Log Level: info
  Enabled Checks: [process_discovery connections]
  Allocated Memory: 12,372,016 bytes
  Hostname: web-server-01

  =================
  Process Endpoints
  =================
    https://process.datadoghq.com - API Key ending with:
        - 58d0e

  =========
  Collector
  =========
    Last collection time: 2023-02-10 10:28:59
    Docker socket: 
    Number of processes: 0
    Number of containers: 0
    Process Queue length: 0
    RTProcess Queue length: 0
    Connections Queue length: 0
    Event Queue length: 0
    Pod Queue length: 0
    Process Bytes enqueued: 0
    RTProcess Bytes enqueued: 0
    Connections Bytes enqueued: 0
    Event Bytes enqueued: 0
    Pod Bytes enqueued: 0
    Drop Check Payloads: []

=========
APM Agent
=========

  Status: Not running or unreachable on localhost:8126.
  Error: Get "http://localhost:8126/debug/vars": dial tcp [::1]:8126: connectex: No connection could be made because the target machine actively refused it.

==========
Aggregator
==========
  Checks Metric Sample: 40,485,407
  Dogstatsd Metric Sample: 223,635
  Event: 1
  Events Flushed: 1
  Number Of Flushes: 5,810
  Series Flushed: 40,147,473
  Service Check: 10,726,375
  Service Checks Flushed: 10,731,192

=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 223,634
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 24,538,729
  Udp Packet Reading Errors: 0
  Udp Packets: 113,229
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 0
  Unterminated Metric Errors: 0

====
OTLP
====

  Status: Not enabled
  Collector status: Not running
gonace commented 1 year ago

The amount of memory used by the Datadog agent has doubled since this issue was created datadog

We had to downgrade to version 7.42.0 since the client since it will consume almost all memory on the servers in a few days, we'll update this issue if the problem exists in the older (7.42.0) version as well.

gonace commented 1 year ago

The same problem appears with version 7.42.0 with 7.96Gb memory after about 24 hours! datadog

We'll try version 7.41.1

vickenty commented 1 year ago

This issue tracker is primarily used to track bugs in the Agent codebase to completion. For issues directly related to your use of the agent, we have a dedicated team who can investigate your reports directly. Please contact Datadog support and and send them a flare demonstrating the issue.

Thanks!

gonace commented 1 year ago

@vickenty we have not changed our datadog agent config and it works just fine in 7.41.1 so something have changed in the datadog agent.