search

DataDog / datadog-agent

Main repository for Datadog Agent
https://docs.datadoghq.com/
Apache License 2.0
2.83k stars 1.19k forks source link

Error running check disk: No such file or directory: '/host/proc/self #7004

Open jbcpollak opened 3 years ago

jbcpollak commented 3 years ago

Running the datadog agent container on Docker for Windows with the WSL2 backend, I get an error that /host/proc/self is not found.

I am using docker-compose, with the following config:

  datadog:
    image: datadog/agent:7
    hostname: ${HOSTNAME}
    network_mode: "host"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /etc/passwd:/etc/passwd:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
    environment:
      - DD_API_KEY=${DATADOG_KEY}
      - DD_SITE=datadoghq.com
      - DD_PROCESS_AGENT_ENABLED=true

Describe what happened:

Error running check disk: [{"message": "[Errno 2] No such file or directory: '/host/proc/self'", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 863, in run\n    self.check(instance)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/disk/disk.py\", line 109, in check\n    for part in psutil.disk_partitions(all=self._include_all_devices):\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/psutil/__init__.py\", line 2027, in disk_partitions\n    return _psplatform.disk_partitions(all)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/psutil/_pslinux.py\", line 1167, in disk_partitions\n    mounts_path = os.path.realpath(\"%s/self/mounts\" % procfs_path)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/posixpath.py\", line 391, in realpath\n    path, ok = _joinrealpath(filename[:0], filename, {})\n  File \"/opt/datadog-agent/embedded/lib/python3.8/posixpath.py\", line 439, in _joinrealpath\n    path, ok = _joinrealpath(path, os.readlink(newpath), seen)\nFileNotFoundError: [Errno 2] No such file or directory: '/host/proc/self'\n"}]

Describe what you expected:

No error

Steps to reproduce the issue:

setup docker agent as above

Additional environment details (Operating System, Cloud provider, etc):

This is Windows 10 (Home, if that matters), WSL2, Docker for Desktop and Ubuntu 20.04

I belief the issue appears similar to this bug but I don't think they are related.

The agent status report is:


===============
Agent (v7.24.0)
===============

  Status date: 2020-12-16 22:22:38.284207 UTC
  Agent start: 2020-12-16 21:53:14.244158 UTC
  Pid: 388
  Go Version: go1.14.7
  Python Version: 3.8.5
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: info

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: 3.604616s
    System UTC time: 2020-12-16 22:22:38.284207 UTC

  Host Info
  =========
    bootTime: 2020-12-12 18:15:40.000000 UTC
    kernelArch: x86_64
    kernelVersion: 4.19.128-microsoft-standard
    os: linux
    platform: debian
    platformFamily: debian
    platformVersion: bullseye/sid
    procs: 23
    uptime: 99h37m39s

  Hostnames
  =========
    hostname: docker-desktop
    socket-fqdn: docker-desktop
    socket-hostname: docker-desktop
    hostname provider: container
    unused hostname providers:
      aws: not retrieving hostname from AWS: the host is not an ECS instance and other providers already retrieve non-default hostnames
      configuration/environment: hostname is empty
      gce: unable to retrieve hostname from GCE: Get "http://169.254.169.254/computeMetadata/v1/instance/hostname": dial tcp 169.254.169.254:80: connect: connection refused

  Metadata
  ========
    hostname_source: container

=========
Collector
=========

  Running Checks
  ==============

    cpu
    ---
      Instance ID: cpu [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default
      Total Runs: 118
      Metric Samples: Last Run: 7, Total: 820
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-12-16 22:22:33.000000 UTC
      Last Successful Execution Date : 2020-12-16 22:22:33.000000 UTC

    disk (4.0.0)
    ------------
      Instance ID: disk:e5dffb8bef24336f [ERROR]
      Configuration Source: file:/etc/datadog-agent/conf.d/disk.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 3ms
      Last Execution Date : 2020-12-16 22:22:25.000000 UTC
      Last Successful Execution Date : Never
      Error: [Errno 2] No such file or directory: '/host/proc/self'
      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 863, in run
          self.check(instance)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/disk/disk.py", line 109, in check
          for part in psutil.disk_partitions(all=self._include_all_devices):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/psutil/__init__.py", line 2027, in disk_partitions
          return _psplatform.disk_partitions(all)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/psutil/_pslinux.py", line 1167, in disk_partitions
          mounts_path = os.path.realpath("%s/self/mounts" % procfs_path)
        File "/opt/datadog-agent/embedded/lib/python3.8/posixpath.py", line 391, in realpath
          path, ok = _joinrealpath(filename[:0], filename, {})
        File "/opt/datadog-agent/embedded/lib/python3.8/posixpath.py", line 439, in _joinrealpath
          path, ok = _joinrealpath(path, os.readlink(newpath), seen)
      FileNotFoundError: [Errno 2] No such file or directory: '/host/proc/self'

    docker
    ------
      Instance ID: docker [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 10, Total: 1,170
      Events: Last Run: 0, Total: 5
      Service Checks: Last Run: 1, Total: 117
      Average Execution Time : 4ms
      Last Execution Date : 2020-12-16 22:22:32.000000 UTC
      Last Successful Execution Date : 2020-12-16 22:22:32.000000 UTC

    file_handle
    -----------
      Instance ID: file_handle [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/file_handle.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 5, Total: 585
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-12-16 22:22:24.000000 UTC
      Last Successful Execution Date : 2020-12-16 22:22:24.000000 UTC

    io
    --
      Instance ID: io [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/io.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 78, Total: 9,072
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-12-16 22:22:31.000000 UTC
      Last Successful Execution Date : 2020-12-16 22:22:31.000000 UTC

    load
    ----
      Instance ID: load [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/load.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 6, Total: 702
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-12-16 22:22:23.000000 UTC
      Last Successful Execution Date : 2020-12-16 22:22:23.000000 UTC

    memory
    ------
      Instance ID: memory [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/memory.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 18, Total: 2,106
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-12-16 22:22:30.000000 UTC
      Last Successful Execution Date : 2020-12-16 22:22:30.000000 UTC

    network (1.19.0)
    ----------------
      Instance ID: network:5c571333f400457d [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/network.d/conf.yaml.default
      Total Runs: 118
      Metric Samples: Last Run: 31, Total: 3,658
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 1ms
      Last Execution Date : 2020-12-16 22:22:37.000000 UTC
      Last Successful Execution Date : 2020-12-16 22:22:37.000000 UTC

    ntp
    ---
      Instance ID: ntp:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/ntp.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 1, Total: 2
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 2
      Average Execution Time : 275ms
      Last Execution Date : 2020-12-16 22:08:19.000000 UTC
      Last Successful Execution Date : 2020-12-16 22:08:19.000000 UTC

    uptime
    ------
      Instance ID: uptime [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
      Total Runs: 117
      Metric Samples: Last Run: 1, Total: 117
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-12-16 22:22:29.000000 UTC
      Last Successful Execution Date : 2020-12-16 22:22:29.000000 UTC

========
JMXFetch
========

  Information
  ==================
  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    Deployments: 0
    Dropped: 0
    DroppedOnInput: 0
    Nodes: 0
    Pods: 0
    ReplicaSets: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Services: 0

  Transaction Successes
  =====================
    Total number: 248
    Successes By Endpoint:
      check_run_v1: 117
      intake: 14
      series_v1: 117

  API Keys status
  ===============
    API key ending with b65ec: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.com - API Key ending with:
      - b65ec

==========
Logs Agent
==========

  Logs Agent is not running

=========
APM Agent
=========
  Status: Running
  Pid: 393
  Uptime: 1764 seconds
  Mem alloc: 32,904,032 bytes
  Hostname: docker-desktop
  Receiver: 0.0.0.0:8126
  Endpoints:
    https://trace.agent.datadoghq.com

  Receiver (previous minute)
  ==========================
    No traces received in the previous minute.
    Default priority sampling rate: 100.0%

  Writer (previous minute)
  ========================
    Traces: 0 payloads, 0 traces, 0 events, 0 bytes
    Stats: 0 payloads, 0 stats buckets, 0 bytes

=========
Aggregator
=========
  Checks Metric Sample: 20,346
  Dogstatsd Metric Sample: 12,244
  Event: 6
  Events Flushed: 6
  Number Of Flushes: 117
  Series Flushed: 23,534
  Service Check: 1,176
  Service Checks Flushed: 1,287

=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 12,243
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 892,508
  Udp Packet Reading Errors: 0
  Udp Packets: 1,180
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 0```
fazith27 commented 3 years ago

I am also facing the same issue with the same exact setup, running docker on Windoes with WSL2 enabled. Any suggestion to fix the issue. ddagent_1 | 2021-03-08 22:14:53 UTC | CORE | ERROR | (pkg/collector/runner/runner.go:292 in work) | Error running check disk: [{"message": "[Errno 2] No such file or directory: '/host/proc/self'", "traceback": "Traceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 901, in run\n self.check(instance)\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/disk/disk.py\", line 115, in check\n for part in psutil.disk_partitions(all=self._include_all_devices):\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/psutil/__init__.py\", line 2027, in disk_partitions\n return _psplatform.disk_partitions(all)\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/psutil/_pslinux.py\", line 1167, in disk_partitions\n mounts_path = os.path.realpath(\"%s/self/mounts\" % procfs_path)\n File \"/opt/datadog-agent/embedded/lib/python3.8/posixpath.py\", line 391, in realpath\n path, ok = _joinrealpath(filename[:0], filename, {})\n File \"/opt/datadog-agent/embedded/lib/python3.8/posixpath.py\", line 439, in _joinrealpath\n path, ok = _joinrealpath(path, os.readlink(newpath), seen)\nFileNotFoundError: [Errno 2] No such file or directory: '/host/proc/self'\n"}]

fazith27 commented 3 years ago

After adding the environment variable "DD_PROCFS_PATH: /proc", I am no longer seeing the error. Reference https://github.com/DataDog/integrations-core/issues/1550.