[BUG] Datadog IoT Agent fails to stream Docker container logs

[sfuerte]

Agent Environment

$ datadog-agent version
Agent 7.38.2 - Commit: ba442fd - Serialization version: v5.0.23 - Go version: go1.17.11

$ dpkg -l | grep datadog
ii  datadog-iot-agent                        1:7.38.2-1                     amd64        Datadog IoT Agent

Describe what happened: despite proper configuration and journal logs that Docker/containerd environments are detected, the Docker container logs are not showing in Datadog.

"proper configuration" - exactly the same config for full version of Datadog agent works fine.

Journal logs upon start of the IoT agent:

agent[3445156]: 2022-08-30 17:14:02 UTC | CORE | INFO | (pkg/util/log/log.go:571 in func1) | Features detected from environment: containerd,docker
agent[3445156]: 2022-08-30 17:14:02 UTC | CORE | INFO | (cmd/agent/app/run.go:251 in StartAgent) | Starting Datadog IoT Agent v7.38.2
..
agent[3445156]: 2022-08-30 17:14:03 UTC | CORE | INFO | (pkg/autodiscovery/listeners/environment.go:65 in createServices) | Listener created docker service from environment
agent[3445156]: 2022-08-30 17:14:03 UTC | CORE | INFO | (pkg/autodiscovery/listeners/environment.go:65 in createServices) | Listener created containerd service from environment
agent[3445156]: 2022-08-30 17:14:03 UTC | CORE | INFO | (pkg/autodiscovery/listeners/environment.go:72 in createServices) | Listener created container service from environment
...

# cat /etc/datadog-agent/conf.d/docker.d/conf.yaml
logs:
  - type: docker

as per Agent status output, Docker log agent stuck at "pending" status (excerpt):

$ datadog-agent status
===============
Agent (v7.38.2)
===============

  Status date: 2022-08-30 17:43:59.607 UTC (1661881439607)
  Agent start: 2022-08-30 17:14:02.045 UTC (1661879642045)
  Pid: 3.445156e+06
  Go Version: go1.17.11
  Python Version: n/a
  Build arch: amd64
  Agent flavor: iot_agent
  Check Runners: 4
  Log Level: info

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Host Info
  =========
    bootTime: 2022-08-19 18:48:57 UTC (1660934937000)
    hostId: d3619021-287d-4c03-b5a1-67da6a9b5f86
    kernelArch: x86_64
    kernelVersion: 5.18.0-0.bpo.1-cloud-amd64
    os: linux
    platform: debian
    platformFamily: debian
    platformVersion: 11.4

  Metadata
  ========
    agent_version: 7.38.2
    config_apm_dd_url:
    config_dd_url:
    config_logs_dd_url:
    config_logs_socks5_proxy_address:
    config_no_proxy: []
    config_process_dd_url:
    config_proxy_http:
    config_proxy_https:
    config_site:
    feature_apm_enabled: true
    feature_cspm_enabled: false
    feature_cws_enabled: false
    feature_logs_enabled: true
    feature_networks_enabled: false
    feature_networks_http_enabled: false
    feature_networks_https_enabled: false
    feature_otlp_enabled: false
    feature_process_enabled: false
    feature_processes_container_enabled: true
    flavor: iot_agent
    hostname_source: os
    install_method_installer_version: deb_package-iot
    install_method_tool: dpkg
    install_method_tool_version: dpkg-1.20.11

==========
Logs Agent
==========
  docker
  ----------
    - Type: docker
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

Describe what you expected: the same behaviour as for the full Agent version, i.e. Docker logs are streamed to Datadog Cloud.

Steps to reproduce the issue: see above

Additional environment details (Operating System, Cloud provider, etc):

$ lsb_release -ds
Debian GNU/Linux 11 (bullseye)

[maokitty]

I have meet the same situation , and not found the log "Could not setup the docker launcher" from my agent.log record which is from here https://docs.datadoghq.com/logs/guide/docker-logs-collection-troubleshooting-guide/#status-pending

my agent version: 7.34.0

[carlosroman]

Hi, IoT Agent doesn't support pulling logs from Docker directly but does support it using journald. For this to happen you need to change the type in your logs config and updated it with an extra flag as follows:

logs:
    - type: journald
      container_mode: true

This assumes you have set-up your Docker daemon to use journald as the logging driver. You may need to filter out the units you're not interested in. More details on this can be found in our docs here.