search

DataDog / datadog-agent

Main repository for Datadog Agent
https://docs.datadoghq.com/
Apache License 2.0
2.87k stars 1.2k forks source link

[BUG] Container agent will not tail container standard out via Docker socket unless configured to ignore file logs. #13250

Open gfl-chris opened 2 years ago

gfl-chris commented 2 years ago

Agent Environment

$ agent version
Agent 7.37.1 - Commit: 3c29612 - Serialization version: v5.0.22 - Go version: go1.17.11

With DD_LOGS_CONFIG_DOCKER_CONTAINER_USE_FILE = false:

$ agent status
Getting the status from the agent.

===============
Agent (v7.37.1)
===============

  Status date: 2022-08-25 15:48:05.318 UTC (1661442485318)
  Agent start: 2022-08-24 21:10:35.372 UTC (1661375435372)
  Pid: 379
  Go Version: go1.17.11
  Python Version: 3.8.11
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: info

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: 15µs
    System time: 2022-08-25 15:48:05.318 UTC (1661442485318)

  Host Info
  =========
    bootTime: 2021-12-21 20:22:02 UTC (1640118122000)
    hostId: ec2cee4f-5eb5-1245-daa9-72e6bb2fa2e3
    kernelArch: x86_64
    kernelVersion: 4.14.193-149.317.amzn2.x86_64
    os: linux
    platform: ubuntu
    platformFamily: debian
    platformVersion: 21.10
    procs: 416
    uptime: 5904h48m38s

  Hostnames
  =========
    ec2-hostname: ***CENSORED***
    host_aliases: [***CENSORED***]
    hostname: ***CENSORED***
    instance-id: ***CENSORED***
    socket-fqdn: ***CENSORED***
    socket-hostname: ***CENSORED***
    host tags:
      env:zd
    hostname provider: aws
    unused hostname providers:
      azure: azure_hostname_style is set to 'os'
      configuration/environment: hostname is empty
      gce: unable to retrieve hostname from GCE: GCE metadata API error: status code 404 trying to GET http://169.254.169.254/computeMetadata/v1/instance/hostname

  Metadata
  ========
    agent_version: 7.37.1
    cloud_provider: AWS
    config_apm_dd_url:
    config_dd_url:
    config_logs_dd_url:
    config_logs_socks5_proxy_address:
    config_no_proxy: []
    config_process_dd_url:
    config_proxy_http:
    config_proxy_https:
    config_site:
    feature_apm_enabled: true
    feature_cspm_enabled: false
    feature_cws_enabled: false
    feature_logs_enabled: true
    feature_networks_enabled: false
    feature_networks_http_enabled: false
    feature_networks_https_enabled: false
    feature_otlp_enabled: false
    feature_process_enabled: true
    feature_processes_container_enabled: false
    flavor: agent
    hostname_source: aws
    install_method_installer_version: docker
    install_method_tool: docker
    install_method_tool_version: docker
    logs_transport: HTTP

=========
Collector
=========

  Running Checks
  ==============

    container
    ---------
      Instance ID: container [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/container.d/conf.yaml.default
      Total Runs: 4,470
      Metric Samples: Last Run: 1,270, Total: 5,675,754
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 115ms
      Last Execution Date : 2022-08-25 15:47:55 UTC (1661442475000)
      Last Successful Execution Date : 2022-08-25 15:47:55 UTC (1661442475000)

    cpu
    ---
      ---8<---SNIP---8<---

    disk (4.7.0)
    ------------
      ---8<---SNIP---8<---

    docker
    ------
      Instance ID: docker [ERROR]
      Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default
      Total Runs: 4,458
      Metric Samples: Last Run: 1,952, Total: 5,684,912
      Events: Last Run: 2, Total: 322
      Service Checks: Last Run: 6, Total: 10,518
      Average Execution Time : 5.578s
      Last Execution Date : 2022-08-25 15:47:51 UTC (1661442471000)
      Last Successful Execution Date : 2022-08-25 15:42:55 UTC (1661442175000)
      Error: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.41/containers/json?all=1&limit=0": context deadline exceeded
      No traceback
      Warning: Error collecting containers: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.41/containers/json?all=1&limit=0": context deadline exceeded

    file_handle
    -----------
      ---8<---SNIP---8<---

    io
    --
      ---8<---SNIP---8<---

    load
    ----
      ---8<---SNIP---8<---

    memory
    ------
      ---8<---SNIP---8<---

    network (2.7.0)
    ---------------
      ---8<---SNIP---8<---

    ntp
    ---
      ---8<---SNIP---8<---

    uptime
    ------
      ---8<---SNIP---8<---

========
JMXFetch
========

  Information
  ==================
  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    Cluster: 0
    ClusterRole: 0
    ClusterRoleBinding: 0
    CronJob: 0
    DaemonSet: 0
    Deployment: 0
    Dropped: 0
    HighPriorityQueueFull: 0
    Ingress: 0
    Job: 0
    Node: 0
    PersistentVolume: 0
    PersistentVolumeClaim: 0
    Pod: 0
    ReplicaSet: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Role: 0
    RoleBinding: 0
    Service: 0
    ServiceAccount: 0
    StatefulSet: 0

  Transaction Successes
  =====================
    Total number: 9519
    Successes By Endpoint:
      check_run_v1: 4,469
      intake: 450
      metadata_v1: 112
      series_v1: 4,488

  On-disk storage
  ===============
    On-disk storage is disabled. Configure `forwarder_storage_max_size_in_bytes` to enable it.

  API Keys status
  ===============
    API key ending with a358f: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.com - API Key ending with:
      - a358f

==========
Logs Agent
==========

    Reliable: Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.com on port 443
    BytesSent: 3.484499586e+10
    EncodedBytesSent: 1.361145086e+09
    LogsProcessed: 5.1564684e+07
    LogsSent: 5.1564672e+07

  host_logs
  ---------
    - Type: file
      Path: /var/log/amazon/efs/*.log
      Status: OK
        2 files tailed out of 2 files matching
      Inputs:
        /var/log/amazon/efs/mount.log
        /var/log/amazon/efs/mount-watchdog.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/amazon/ssm/*.log
      Status: OK
        2 files tailed out of 2 files matching
      Inputs:
        /var/log/amazon/ssm/errors.log
        /var/log/amazon/ssm/amazon-ssm-agent.log
      BytesRead: 90353
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 19
      24h Peak Latency (ms): 19
    - Type: file
      Path: /var/log/ansible.log
      Status: OK
      Inputs:
        /var/log/ansible.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/audit/audit.log
      Status: OK
      Inputs:
        /var/log/audit/audit.log
      BytesRead: 108513
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/awsagent-update.log
      Status: OK
      Inputs:
        /var/log/awsagent-update.log
      BytesRead: 58431
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 2
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/boot.log
      Status: OK
      Inputs:
        /var/log/boot.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/cloud-init*.log
      Status: OK
        2 files tailed out of 2 files matching
      Inputs:
        /var/log/cloud-init.log
        /var/log/cloud-init-output.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/cron
      Status: OK
      Inputs:
        /var/log/cron
      BytesRead: 9928
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/dmesg
      Status: OK
      Inputs:
        /var/log/dmesg
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/yum.log
      Status: OK
      Inputs:
        /var/log/yum.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/messages
      Status: OK
      Inputs:
        /var/log/messages
      BytesRead: 4.6857999e+07
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 31
      24h Peak Latency (ms): 31
    - Type: file
      Path: /var/log/secure
      Status: OK
      Inputs:
        /var/log/secure
      BytesRead: 847
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  container_collect_all
  ---------------------
    - Type: docker
      Status: OK
      Inputs:
        a4fe0061cb60a1c9d22ba34df280b0abaf5cea1b5d3898c17790e8a2edcbdad4
        5e93a9166aa15e990673c331f146e3e34764cda9792d9c58d607d301e767326d
        246167c911f49edf3dd01e19952e35ce0985dd41d77369a5fd654aa1b69d3f4b
        9b6ee9856ea9235b4d3e5901d074e025c0d50f04274372f4e04b0f52d1cf375a
        fe3a4e876b7874960e47d3b3efb865e6d8c3a59c28a2ca12ab1a97069ac09b3f
        fc6ce667f35ce5fd81810e5eb9e373c82e0e7245d6416c5346bf6af5b9d865dc
        e533c40fb59f42e426c2ae24d7b054cc7ba993c72a1971af544e4984cd073849
        d3dd46859031d48ac88d83176070242075e4775243f18216deb888203fb87400
        cc8fa99918b807b08558d9c22a98a308b25586154b9e09b16dab566e09e54ad9
        2586f5d5ce2f40f7e59454b2ad1df0f9c76374ca75f39b3e65470cf12327bdeb
        52406f23a7627d7fbf470e067222b7f1465af69fc55597a1f5ab1c53a3f99ae7
        62b6b596457bc0cd436a44d16f2a4f98829f83ec2e85808aa2631c9d2c6c3c1d
        ec6e553d525addd000a8eedf8ab19069090079112cb6b96eac0ff08b36e89c58
        b85fe718dc30e8107ddedc71da7bdec13c90e23a79e726611f66c8f42ba77331
        117cc3cd7e34dd28dcb87a21f6f1fcf6d98d5eade438d24c143f7cad4159da8f
        3983c9645acff0fadc9f4213fe4edb487611801863cb87af81ad7d8f2ad072af
        55607e38e37f183b31ca9d68c01c759fa969a6fb6e9994995ee79947901119e3
        b6e492405daedfaa5cc180d1bf0db0b82b6bf207f92c487b24b37356d85cfe7b
        bc0ebf1173323b55342dbf487b1cd9c8942129d8aebfb96bd28c32856afb017a
        0946f1ed0d49297b50365cada2c1962647b844c8ca71d0f3e2387ce53e41b047
        4e68ae3dd1258c7379e6eeded9c13e40e9a6816a0604de641dd8c50aff53681c
        e56a50d04da5ed4afcb750bb9ab4c7b36302cb1b455f7c20da9c537f225077b6
        07020c3d4354b64352dd8bc4928b7b459ff0eb332e81f6b68db89ebf0090d742
        10555110ba540c0ab53ef09c6780d54e957e2bd202915271b17280a8a58e020f
        0555684b030305b8325902fcbbbdee098208f8b0679b1beafc5fea24d9b7024b
        6648e584224a971f4b245782fa43b4a86bf1963cc63758745df3e23e83e177ed
        71df287c6f6e6a7cfcc0d069e5c8c29dbdfc483f16420056a7e8fca66e9b5eab
        0789bdba4cb38a88ebbaefdbc19f2bdd79a42d510655c30a59eeeb03fb740702
        ee92ccf7b8c7a733529851052bfef81ad5af6c3c927fe4ac7000454303db67ee
        11aeb269b89b0f7487a242a900f3e810da0b3e5c564a1398cef6afd3c54f1c15
        5c9e5e823e6d7a02ad87879fcdf6e654c55505c5b37391c8d851bd79ed93479e
        11285731a45a051dc7b7143d27aa44e5d24d9699536593af4a78357b20b522b1
        5256a6a1ca98c9f9fd5d8f81f298d7cde424cc9eb657b17a01d279a312629c25
        5fd6751fa0093acc7ff8cb3e725f73785f530137d304bee5eb64c4c3fade6798
        0421228cb4dfac86a9205f185d26e76a1c5c254af77afc6ef6cd66351c260784
        dce07a9b7ec7bed17e4058122a8da462ed71f68dd03e1c425e0edfd90a7396b1
        7bde5e32cacea2451e25cbbbb6b3eb704807628e416d8ce1faa05130c22de921
        b6da397d839b4bd3de252cf5da23affa9f99b146f8c49e09d97115b41a9e318f
        a6def963c723ebf955f93d0b449a0fe4f0b614eb9dc27443227580249bb989bb
        ab98bac4c2dc5c0d5d72fbb385ca3de2f19f9ebc3807a94ed255efeb4074a3b1
        da8153b1aec7fa4a34fd61ff1109ddf93d3bc67f4c32616704b10a0f55cfa857
        d19117f7c3cf3c281d3af9f54807c49d1810f0e237c2e1a62ea79082b66a2e48
        361568d8f96d35fe0007e0f6381c1bc5c802d60e75d9924da285c206fe167399
        dc79e9c73bcfaa77b3c2919ff0f61ffcd8dc079f6b27e966bce0f58f212b2bf0
        b6628692374adf9eb66d4fe9cd87046271dd1f08f9eb97220a3b144556faff30
        605702ceae3ba66356a4330b3ac64694b3dcd1bea2e95e042cc8c05c6f3d701b
        5449cdc2f3db52d76329fdd150a30022c7397e2005f450769c8b55e02ed4fbc5
        1bb0b7849af5b81054c88e2171b72597b139ff46b7be79f1b4da4a415f308b2a
        15e1a035bfce23144d79eeb226c5eadf49bd08d9fcb5887a1c89fec0bd3b91b1
        9054680254da2c7f895a832efb00cef67a43d9ad155315c4b887e04fe580b3f9
        df7012a4e5b2fbc788c7328560546f824283be993d005eaad78de438b675e1fd
        4a3062b1993822c8bdf888f3d09dc7dec42b9fe42cbbc490578dae64c07167e9
        c3c075f156476a24ba4953ca0d432be573e356849811b0e21ef60443c516a8a4
        dce88280b77cd2873be849960666d74903fb59bb018e1af2e0f248869b262088
      BytesRead: 1.1815268024e+10
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
      Container Info:
        Container ID: 9054680254da, Image: ***CENSORED***, Created: 2022-07-29T20:57:11.277884166Z, Tailing from the Docker socket
        Container ID: 361568d8f96d, Image: ***CENSORED***, Created: 2022-08-24T15:47:06.528007486Z, Tailing from the Docker socket
        Container ID: 0555684b0303, Image: ***CENSORED***, Created: 2022-08-24T14:48:34.195769677Z, Tailing from the Docker socket
        Container ID: 246167c911f4, Image: ***CENSORED***, Created: 2022-08-24T18:17:43.007959844Z, Tailing from the Docker socket
        Container ID: 55607e38e37f, Image: ***CENSORED***, Created: 2022-05-10T19:13:12.467865596Z, Tailing from the Docker socket
        Container ID: 15e1a035bfce, Image: ***CENSORED***, Created: 2022-08-24T18:20:27.651935152Z, Tailing from the Docker socket
        Container ID: 5449cdc2f3db, Image: ***CENSORED***, Created: 2022-08-24T16:00:16.569392514Z, Tailing from the Docker socket
        Container ID: 11aeb269b89b, Image: ***CENSORED***, Created: 2022-08-24T15:20:23.847004742Z, Tailing from the Docker socket
        Container ID: dce07a9b7ec7, Image: ***CENSORED***, Created: 2022-07-29T20:57:34.273568968Z, Tailing from the Docker socket
        Container ID: b6628692374a, Image: ***CENSORED***, Created: 2022-08-25T14:30:58.758078474Z, Tailing from the Docker socket
        Container ID: ec6e553d525a, Image: ***CENSORED***, Created: 2022-08-24T15:20:43.64021512Z, Tailing from the Docker socket
        Container ID: 0421228cb4df, Image: ***CENSORED***, Created: 2022-05-10T19:11:30.597104922Z, Tailing from the Docker socket
        Container ID: 11285731a45a, Image: ***CENSORED***, Created: 2022-05-10T19:13:25.772934031Z, Tailing from the Docker socket
        Container ID: fe3a4e876b78, Image: ***CENSORED***, Created: 2022-05-18T06:11:43.341455284Z, Tailing from the Docker socket
        Container ID: 7bde5e32cace, Image: ***CENSORED***, Created: 2022-08-24T15:47:02.715066058Z, Tailing from the Docker socket
        Container ID: 07020c3d4354, Image: ***CENSORED***, Created: 2022-07-29T20:58:02.697173281Z, Tailing from the Docker socket
        Container ID: d19117f7c3cf, Image: ***CENSORED***, Created: 2022-08-24T18:28:49.977451986Z, Tailing from the Docker socket
        Container ID: 6648e584224a, Image: ***CENSORED***, Created: 2022-08-23T21:56:32.2166805Z, Tailing from the Docker socket
        Container ID: 52406f23a762, Image: ***CENSORED***, Created: 2022-08-24T15:09:10.425020416Z, Tailing from the Docker socket
        Container ID: e56a50d04da5, Image: ***CENSORED***, Created: 2022-05-10T19:12:19.812343865Z, Tailing from the Docker socket
        Container ID: 10555110ba54, Image: ***CENSORED***, Created: 2022-08-23T21:40:51.083567383Z, Tailing from the Docker socket
        Container ID: c3c075f15647, Image: ***CENSORED***, Created: 2022-07-22T18:18:20.381344489Z, Tailing from the Docker socket
        Container ID: dc79e9c73bcf, Image: ***CENSORED***, Created: 2021-12-21T20:25:37.778806476Z, Tailing from the Docker socket
        Container ID: b6e492405dae, Image: ***CENSORED***, Created: 2022-08-10T03:47:11.507800959Z, Tailing from the Docker socket
        Container ID: b85fe718dc30, Image: ***CENSORED***, Created: 2022-08-23T22:34:48.995056585Z, Tailing from the Docker socket
        Container ID: a4fe0061cb60, Image: ***CENSORED***, Created: 2022-07-29T20:57:02.794026231Z, Tailing from the Docker socket
        Container ID: 3983c9645acf, Image: ***CENSORED***, Created: 2022-08-24T15:39:47.954459348Z, Tailing from the Docker socket
        Container ID: 605702ceae3b, Image: ***CENSORED***, Created: 2021-12-21T20:25:45.089286791Z, Tailing from the Docker socket
        Container ID: 5fd6751fa009, Image: ***CENSORED***, Created: 2022-08-24T18:15:28.995551098Z, Tailing from the Docker socket
        Container ID: dce88280b77c, Image: ***CENSORED***, Created: 2022-05-17T19:12:51.569138744Z, Tailing from the Docker socket
        Container ID: 71df287c6f6e, Image: ***CENSORED***, Created: 2022-05-18T06:10:54.296942559Z, Tailing from the Docker socket
        Container ID: 5e93a9166aa1, Image: ***CENSORED***, Created: 2022-05-18T06:11:13.266987523Z, Tailing from the Docker socket
        Container ID: b6da397d839b, Image: ***CENSORED***, Created: 2022-08-24T15:08:55.337568217Z, Tailing from the Docker socket
        Container ID: 117cc3cd7e34, Image: ***CENSORED***, Created: 2022-07-29T20:58:02.70876994Z, Tailing from the Docker socket
        Container ID: 0789bdba4cb3, Image: ***CENSORED***, Created: 2022-08-23T21:56:32.204388156Z, Tailing from the Docker socket
        Container ID: bc0ebf117332, Image: ***CENSORED***, Created: 2022-08-23T23:07:15.595798611Z, Tailing from the Docker socket
        Container ID: 1bb0b7849af5, Image: ***CENSORED***, Created: 2022-08-23T21:56:35.076223977Z, Tailing from the Docker socket
        Container ID: e533c40fb59f, Image: ***CENSORED***, Created: 2022-07-29T20:58:02.716845301Z, Tailing from the Docker socket
        Container ID: ee92ccf7b8c7, Image: ***CENSORED***, Created: 2022-08-24T21:10:34.729557048Z, Tailing from the Docker socket
        Container ID: df7012a4e5b2, Image: ***CENSORED***, Created: 2022-08-24T15:29:37.80597281Z, Tailing from the Docker socket
        Container ID: 9b6ee9856ea9, Image: ***CENSORED***, Created: 2022-05-10T19:12:19.67657078Z, Tailing from the Docker socket
        Container ID: fc6ce667f35c, Image: ***CENSORED***, Created: 2022-08-23T23:07:20.036172451Z, Tailing from the Docker socket
        Container ID: 4a3062b19938, Image: ***CENSORED***, Created: 2022-08-24T16:00:23.186565276Z, Tailing from the Docker socket
        Container ID: 62b6b596457b, Image: ***CENSORED***, Created: 2022-08-24T16:00:29.324709425Z, Tailing from the Docker socket
        Container ID: 2586f5d5ce2f, Image: ***CENSORED***, Created: 2022-08-24T15:29:40.847587134Z, Tailing from the Docker socket
        Container ID: d3dd46859031, Image: ***CENSORED***, Created: 2022-07-29T20:58:02.697329086Z, Tailing from the Docker socket
        Container ID: 5c9e5e823e6d, Image: ***CENSORED***, Created: 2022-08-24T15:29:44.423586862Z, Tailing from the Docker socket
        Container ID: ab98bac4c2dc, Image: ***CENSORED***, Created: 2022-05-10T19:12:19.774731462Z, Tailing from the Docker socket
        Container ID: 4e68ae3dd125, Image: ***CENSORED***, Created: 2022-08-24T15:46:52.057871032Z, Tailing from the Docker socket
        Container ID: da8153b1aec7, Image: ***CENSORED***, Created: 2022-08-24T14:55:31.673532052Z, Tailing from the Docker socket
        Container ID: 0946f1ed0d49, Image: ***CENSORED***, Created: 2022-08-24T15:20:21.737225918Z, Tailing from the Docker socket
        Container ID: 5256a6a1ca98, Image: ***CENSORED***, Created: 2022-07-29T20:57:34.281099032Z, Tailing from the Docker socket
        Container ID: a6def963c723, Image: ***CENSORED***, Created: 2022-07-29T20:57:19.695708312Z, Tailing from the Docker socket
        Container ID: cc8fa99918b8, Image: ***CENSORED***, Created: 2022-07-29T20:58:02.705284603Z, Tailing from the Docker socket

=============
Process Agent
=============

  Version: 7.37.1
  Status date: 2022-08-25 15:48:05.324 UTC (1661442485324)
  Process Agent Start: 2022-08-24 21:10:35.564 UTC (1661375435564)
  Pid: 381
  Go Version: go1.17.11
  Build arch: amd64
  Log Level: info
  Enabled Checks: [process rtprocess]
  Allocated Memory: 19,823,976 bytes
  Hostname: ***CENSORED***

  =================
  Process Endpoints
  =================
    https://process.datadoghq.com - API Key ending with:
        - a358f

  =========
  Collector
  =========
    Last collection time: 2022-08-25 15:47:58
    Docker socket: /var/run/docker.sock
    Number of processes: 342
    Number of containers: 54
    Process Queue length: 0
    RTProcess Queue length: 0
    Pod Queue length: 0
    Process Bytes enqueued: 0
    RTProcess Bytes enqueued: 0
    Pod Bytes enqueued: 0
    Drop Check Payloads: []
=========
APM Agent
=========
  Status: Running
  Pid: 380
  Uptime: 67049 seconds
  Mem alloc: 23,126,176 bytes
  Hostname: ***CENSORED***
  Receiver: 0.0.0.0:8126
  Endpoints:
    https://trace.agent.datadoghq.com

  Receiver (previous minute)
  ==========================
    From go 1.17.9 (gc-amd64-linux), client v1.40.1
      Traces received: 37 (19,163 bytes)
      Spans received: 37

    From java 1.8.0_342 (OpenJDK 64-Bit Server VM), client 0.99.0~c3bbd7bd60
      Traces received: 8 (8,158 bytes)
      Spans received: 20

    Default priority sampling rate: 100.0%
    Priority sampling rate for 'service:***CENSORED***': 100.0%
    Priority sampling rate for 'service:***CENSORED***': 100.0%
    Priority sampling rate for 'service:***CENSORED***': 100.0%
    Priority sampling rate for 'service:***CENSORED***': 100.0%
    Priority sampling rate for 'service:***CENSORED***': 100.0%
    Priority sampling rate for 'service:***CENSORED***': 100.0%
    Priority sampling rate for 'service:***CENSORED***': 100.0%
    Priority sampling rate for 'service:***CENSORED***': 100.0%

  Writer (previous minute)
  ========================
    Traces: 0 payloads, 0 traces, 0 events, 0 bytes
    Stats: 0 payloads, 0 stats buckets, 0 bytes

=========
Aggregator
=========
  Checks Metric Sample: 15,862,555
  Dogstatsd Metric Sample: 1,419,968
  Event: 92
  Events Flushed: 92
  Number Of Flushes: 4,469
  Series Flushed: 14,361,507
  Service Check: 49,292
  Service Checks Flushed: 53,750
=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 1,419,967
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 286,549,921
  Udp Packet Reading Errors: 0
  Udp Packets: 705,585
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 0
  Unterminated Metric Errors: 0

=============
Autodiscovery
=============
  Enabled Features
  ================
    docker

====
OTLP
====

  Status: Not enabled
  Collector status: Not running

the agent attach to the Docker socket to tail containers' stdout/stderr and send container logs to HQ. But with DD_LOGS_CONFIG_DOCKER_CONTAINER_USE_FILE not set:

$ agent status
Getting the status from the agent.

===============
Agent (v7.37.1)
===============

  Status date: 2022-08-25 17:39:38.794 UTC (1661449178794)
  Agent start: 2022-08-25 17:38:48.927 UTC (1661449128927)
  Pid: 378
  Go Version: go1.17.11
  Python Version: 3.8.11
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: info

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: 8µs
    System time: 2022-08-25 17:39:38.794 UTC (1661449178794)

  Host Info
  =========
    bootTime: 2021-12-21 20:22:03 UTC (1640118123000)
    hostId: ec235ee9-0396-d41d-6e8f-f756dbe92eb2
    kernelArch: x86_64
    kernelVersion: 4.14.193-149.317.amzn2.x86_64
    os: linux
    platform: ubuntu
    platformFamily: debian
    platformVersion: 21.10
    procs: 233
    uptime: 5925h16m52s

  Hostnames
  =========
    ec2-hostname: ***CENSORED***
    host_aliases: [***CENSORED***]
    hostname: ***CENSORED***
    instance-id: ***CENSORED***
    socket-fqdn: ***CENSORED***
    socket-hostname: ***CENSORED***
    host tags:
      env:zd
    hostname provider: aws
    unused hostname providers:
      azure: azure_hostname_style is set to 'os'
      configuration/environment: hostname is empty
      gce: unable to retrieve hostname from GCE: GCE metadata API error: status code 404 trying to GET http://169.254.169.254/computeMetadata/v1/instance/hostname

  Metadata
  ========
    agent_version: 7.37.1
    cloud_provider: AWS
    config_apm_dd_url:
    config_dd_url:
    config_logs_dd_url:
    config_logs_socks5_proxy_address:
    config_no_proxy: []
    config_process_dd_url:
    config_proxy_http:
    config_proxy_https:
    config_site:
    feature_apm_enabled: true
    feature_cspm_enabled: false
    feature_cws_enabled: false
    feature_logs_enabled: true
    feature_networks_enabled: false
    feature_networks_http_enabled: false
    feature_networks_https_enabled: false
    feature_otlp_enabled: false
    feature_process_enabled: true
    feature_processes_container_enabled: false
    flavor: agent
    hostname_source: aws
    install_method_installer_version: docker
    install_method_tool: docker
    install_method_tool_version: docker
    logs_transport: HTTP

=========
Collector
=========

  Running Checks
  ==============

    container
    ---------
      Instance ID: container [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/container.d/conf.yaml.default
      Total Runs: 3
      Metric Samples: Last Run: 790, Total: 1,918
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 120ms
      Last Execution Date : 2022-08-25 17:39:25 UTC (1661449165000)
      Last Successful Execution Date : 2022-08-25 17:39:25 UTC (1661449165000)

    cpu
    ---
      ---8<---SNIP---8<---

    disk (4.7.0)
    ------------
      ---8<---SNIP---8<---

    docker
    ------
      Instance ID: docker [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default
      Total Runs: 3
      Metric Samples: Last Run: 612, Total: 1,599
      Events: Last Run: 6, Total: 39
      Service Checks: Last Run: 1, Total: 3
      Average Execution Time : 261ms
      Last Execution Date : 2022-08-25 17:39:31 UTC (1661449171000)
      Last Successful Execution Date : 2022-08-25 17:39:31 UTC (1661449171000)

    file_handle
    -----------
      ---8<---SNIP---8<---

    io
    --
      ---8<---SNIP---8<---

    load
    ----
      ---8<---SNIP---8<---

    memory
    ------
      ---8<---SNIP---8<---

    network (2.7.0)
    ---------------
      ---8<---SNIP---8<---

    ntp
    ---
      ---8<---SNIP---8<---

    uptime
    ------
      ---8<---SNIP---8<---

========
JMXFetch
========

  Information
  ==================
  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    Cluster: 0
    ClusterRole: 0
    ClusterRoleBinding: 0
    CronJob: 0
    DaemonSet: 0
    Deployment: 0
    Dropped: 0
    HighPriorityQueueFull: 0
    Ingress: 0
    Job: 0
    Node: 0
    PersistentVolume: 0
    PersistentVolumeClaim: 0
    Pod: 0
    ReplicaSet: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Role: 0
    RoleBinding: 0
    Service: 0
    ServiceAccount: 0
    StatefulSet: 0

  Transaction Successes
  =====================
    Total number: 10
    Successes By Endpoint:
      check_run_v1: 3
      intake: 4
      series_v1: 3

  On-disk storage
  ===============
    On-disk storage is disabled. Configure `forwarder_storage_max_size_in_bytes` to enable it.

  API Keys status
  ===============
    API key ending with a358f: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.com - API Key ending with:
      - a358f

==========
Logs Agent
==========

    Reliable: Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.com on port 443
    BytesSent: 1.917999e+06
    EncodedBytesSent: 114783
    LogsProcessed: 3401
    LogsSent: 3401

  host_logs
  ---------
    - Type: file
      Path: /var/log/amazon/efs/*.log
      Status: OK
        2 files tailed out of 2 files matching
      Inputs:
        /var/log/amazon/efs/mount.log
        /var/log/amazon/efs/mount-watchdog.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/amazon/ssm/*.log
      Status: OK
        2 files tailed out of 2 files matching
      Inputs:
        /var/log/amazon/ssm/errors.log
        /var/log/amazon/ssm/amazon-ssm-agent.log
      BytesRead: 256
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/ansible.log
      Status: OK
      Inputs:
        /var/log/ansible.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/audit/audit.log
      Status: OK
      Inputs:
        /var/log/audit/audit.log
      BytesRead: 168343
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 2
      24h Peak Latency (ms): 2
    - Type: file
      Path: /var/log/awsagent-update.log
      Status: OK
      Inputs:
        /var/log/awsagent-update.log
      BytesRead: 1
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/boot.log
      Status: OK
      Inputs:
        /var/log/boot.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/cloud-init*.log
      Status: OK
        2 files tailed out of 2 files matching
      Inputs:
        /var/log/cloud-init.log
        /var/log/cloud-init-output.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/cron
      Status: OK
      Inputs:
        /var/log/cron
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/dmesg
      Status: OK
      Inputs:
        /var/log/dmesg
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/yum.log
      Status: OK
      Inputs:
        /var/log/yum.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Path: /var/log/messages
      Status: OK
      Inputs:
        /var/log/messages
      BytesRead: 115373
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 16
      24h Peak Latency (ms): 16
    - Type: file
      Path: /var/log/secure
      Status: OK
      Inputs:
        /var/log/secure
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  container_collect_all
  ---------------------
    - Type: file
      Identifier: c03c6e1d62c22a5a7a61098ed7da60438bf5b8ac546a384821a51d0fe4c94b80
      Path: /var/lib/docker/containers/c03c6e1d62c22a5a7a61098ed7da60438bf5b8ac546a384821a51d0fe4c94b80/c03c6e1d62c22a5a7a61098ed7da60438bf5b8ac546a384821a51d0fe4c94b80-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: dee3213f0200909884b050181d04a707ff8e2f5772539230be8537cb22ec09b3
      Path: /var/lib/docker/containers/dee3213f0200909884b050181d04a707ff8e2f5772539230be8537cb22ec09b3/dee3213f0200909884b050181d04a707ff8e2f5772539230be8537cb22ec09b3-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      Inputs:
        /var/lib/docker/containers/dee3213f0200909884b050181d04a707ff8e2f5772539230be8537cb22ec09b3/dee3213f0200909884b050181d04a707ff8e2f5772539230be8537cb22ec09b3-json.log
      BytesRead: 605861
      Average Latency (ms): 2
      24h Average Latency (ms): 2
      Peak Latency (ms): 17
      24h Peak Latency (ms): 17
    - Type: file
      Identifier: 9d1f7da70beac9e6319c4ad8e491333e4d1cf1dc39faed2e8b89f91743b17c2d
      Path: /var/lib/docker/containers/9d1f7da70beac9e6319c4ad8e491333e4d1cf1dc39faed2e8b89f91743b17c2d/9d1f7da70beac9e6319c4ad8e491333e4d1cf1dc39faed2e8b89f91743b17c2d-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: b8a472b7a5b91021c37c6a173a706dc500ddfcd4dabea9e3251409b6f0ed93ac
      Path: /var/lib/docker/containers/b8a472b7a5b91021c37c6a173a706dc500ddfcd4dabea9e3251409b6f0ed93ac/b8a472b7a5b91021c37c6a173a706dc500ddfcd4dabea9e3251409b6f0ed93ac-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 0e4e53b07ff7647aa5c30662de5935ac4810c514ee43fe739d0dfd7c54299418
      Path: /var/lib/docker/containers/0e4e53b07ff7647aa5c30662de5935ac4810c514ee43fe739d0dfd7c54299418/0e4e53b07ff7647aa5c30662de5935ac4810c514ee43fe739d0dfd7c54299418-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      Inputs:
        /var/lib/docker/containers/0e4e53b07ff7647aa5c30662de5935ac4810c514ee43fe739d0dfd7c54299418/0e4e53b07ff7647aa5c30662de5935ac4810c514ee43fe739d0dfd7c54299418-json.log
      BytesRead: 32119
      Average Latency (ms): 2
      24h Average Latency (ms): 2
      Peak Latency (ms): 10
      24h Peak Latency (ms): 10
    - Type: file
      Identifier: 5f8beaaf9568e6ae0e19db054a13fc805f2410882cf9a40e24e9969613642aa3
      Path: /var/lib/docker/containers/5f8beaaf9568e6ae0e19db054a13fc805f2410882cf9a40e24e9969613642aa3/5f8beaaf9568e6ae0e19db054a13fc805f2410882cf9a40e24e9969613642aa3-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 08207dba4b3743c20914e0ddd69ab9516bd9ce2c733d51a20854c4f7dfdb7623
      Path: /var/lib/docker/containers/08207dba4b3743c20914e0ddd69ab9516bd9ce2c733d51a20854c4f7dfdb7623/08207dba4b3743c20914e0ddd69ab9516bd9ce2c733d51a20854c4f7dfdb7623-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: e19768d22749756abf7bd57bc2f7dc002da84ab6505170407dbb68ec35b6f835
      Path: /var/lib/docker/containers/e19768d22749756abf7bd57bc2f7dc002da84ab6505170407dbb68ec35b6f835/e19768d22749756abf7bd57bc2f7dc002da84ab6505170407dbb68ec35b6f835-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 77e5f9a37643b5bd19f0c906af91d65418b5cad58b696495b34f4bd1b68d55b9
      Path: /var/lib/docker/containers/77e5f9a37643b5bd19f0c906af91d65418b5cad58b696495b34f4bd1b68d55b9/77e5f9a37643b5bd19f0c906af91d65418b5cad58b696495b34f4bd1b68d55b9-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 9793b1d3392218b1920430e9b47acc596fc909c2b02cd7db81f3b79dd80bbd24
      Path: /var/lib/docker/containers/9793b1d3392218b1920430e9b47acc596fc909c2b02cd7db81f3b79dd80bbd24/9793b1d3392218b1920430e9b47acc596fc909c2b02cd7db81f3b79dd80bbd24-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 3ef9d9be4313719a242293d62948ab93140efa14141d2b3736ffae782c724b08
      Path: /var/lib/docker/containers/3ef9d9be4313719a242293d62948ab93140efa14141d2b3736ffae782c724b08/3ef9d9be4313719a242293d62948ab93140efa14141d2b3736ffae782c724b08-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 5cb2440ee3b9c35fe986688169beebef804c8ed992b54b355cade707f1909486
      Path: /var/lib/docker/containers/5cb2440ee3b9c35fe986688169beebef804c8ed992b54b355cade707f1909486/5cb2440ee3b9c35fe986688169beebef804c8ed992b54b355cade707f1909486-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: c7a6dbea183dfc292862e771b140b9cc4d749c28e734984886b8c6d42773dd5b
      Path: /var/lib/docker/containers/c7a6dbea183dfc292862e771b140b9cc4d749c28e734984886b8c6d42773dd5b/c7a6dbea183dfc292862e771b140b9cc4d749c28e734984886b8c6d42773dd5b-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: ac9318a7062137626a5b7b8874ed83342f1a823b45182710ae51379a0bd3efa3
      Path: /var/lib/docker/containers/ac9318a7062137626a5b7b8874ed83342f1a823b45182710ae51379a0bd3efa3/ac9318a7062137626a5b7b8874ed83342f1a823b45182710ae51379a0bd3efa3-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 0378e6487dbb3bb26943652f15050cbfabfd2016a520248522b05a90b49501d0
      Path: /var/lib/docker/containers/0378e6487dbb3bb26943652f15050cbfabfd2016a520248522b05a90b49501d0/0378e6487dbb3bb26943652f15050cbfabfd2016a520248522b05a90b49501d0-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 196ed3bc408c28b1e6d536f4eea471208970dd64c34fd89a2e232246899b1644
      Path: /var/lib/docker/containers/196ed3bc408c28b1e6d536f4eea471208970dd64c34fd89a2e232246899b1644/196ed3bc408c28b1e6d536f4eea471208970dd64c34fd89a2e232246899b1644-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: c8bf3df4c6e10c2a8391b5108062b5b94b2d05b1802729aef1a456542a38b711
      Path: /var/lib/docker/containers/c8bf3df4c6e10c2a8391b5108062b5b94b2d05b1802729aef1a456542a38b711/c8bf3df4c6e10c2a8391b5108062b5b94b2d05b1802729aef1a456542a38b711-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: bb0a50d200b06ae52344db108ceab11658902edcd00c3ab21a2fddbdf2036915
      Path: /var/lib/docker/containers/bb0a50d200b06ae52344db108ceab11658902edcd00c3ab21a2fddbdf2036915/bb0a50d200b06ae52344db108ceab11658902edcd00c3ab21a2fddbdf2036915-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 6915bb4a1f76a5b2b2abaa1db27c7577cad9234928284dda1ccfcd49982d3d38
      Path: /var/lib/docker/containers/6915bb4a1f76a5b2b2abaa1db27c7577cad9234928284dda1ccfcd49982d3d38/6915bb4a1f76a5b2b2abaa1db27c7577cad9234928284dda1ccfcd49982d3d38-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 8f7f879c6bbfc0b1c33e36fea760778119d26f76a1a796932c9f7e436bbf1fbf
      Path: /var/lib/docker/containers/8f7f879c6bbfc0b1c33e36fea760778119d26f76a1a796932c9f7e436bbf1fbf/8f7f879c6bbfc0b1c33e36fea760778119d26f76a1a796932c9f7e436bbf1fbf-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: d90c79ef633ec5baf9b158df3176f0d3f089190e73d5413865ac38a052b392da
      Path: /var/lib/docker/containers/d90c79ef633ec5baf9b158df3176f0d3f089190e73d5413865ac38a052b392da/d90c79ef633ec5baf9b158df3176f0d3f089190e73d5413865ac38a052b392da-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: cbe2ec74441f55d3820c6d47baaca63257df92d7031f8c5103b06022658d41f9
      Path: /var/lib/docker/containers/cbe2ec74441f55d3820c6d47baaca63257df92d7031f8c5103b06022658d41f9/cbe2ec74441f55d3820c6d47baaca63257df92d7031f8c5103b06022658d41f9-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 45d7f32f1f6a4393d64cbffdaeb9ce38028c94b737e19299396b5a9b36af5d5a
      Path: /var/lib/docker/containers/45d7f32f1f6a4393d64cbffdaeb9ce38028c94b737e19299396b5a9b36af5d5a/45d7f32f1f6a4393d64cbffdaeb9ce38028c94b737e19299396b5a9b36af5d5a-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 7bad0922049c0b50a335aefe780f28153368ade77dd7027d925b257b9235adfa
      Path: /var/lib/docker/containers/7bad0922049c0b50a335aefe780f28153368ade77dd7027d925b257b9235adfa/7bad0922049c0b50a335aefe780f28153368ade77dd7027d925b257b9235adfa-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 6b27be2a871d13c0468ad7d6921deb542d142d4e1037b03a9125399b1683dd6a
      Path: /var/lib/docker/containers/6b27be2a871d13c0468ad7d6921deb542d142d4e1037b03a9125399b1683dd6a/6b27be2a871d13c0468ad7d6921deb542d142d4e1037b03a9125399b1683dd6a-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 154b628a1d89a1af295fdc78ef6b6963e1b0b9f270179ab953cc304ae0383964
      Path: /var/lib/docker/containers/154b628a1d89a1af295fdc78ef6b6963e1b0b9f270179ab953cc304ae0383964/154b628a1d89a1af295fdc78ef6b6963e1b0b9f270179ab953cc304ae0383964-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: b2b4ec7d6854c6b8dd092dc33d0bb30d638c783c5e75c941a9fa49b7a83caf0c
      Path: /var/lib/docker/containers/b2b4ec7d6854c6b8dd092dc33d0bb30d638c783c5e75c941a9fa49b7a83caf0c/b2b4ec7d6854c6b8dd092dc33d0bb30d638c783c5e75c941a9fa49b7a83caf0c-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: f6dc88c8770f357d38b83261a7144b661c310eba8fd93a95d9f464954a094be1
      Path: /var/lib/docker/containers/f6dc88c8770f357d38b83261a7144b661c310eba8fd93a95d9f464954a094be1/f6dc88c8770f357d38b83261a7144b661c310eba8fd93a95d9f464954a094be1-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: eb42afeec7f57d13998323eab6a2b36b300c990908e1109aa6382c2445fdc3ac
      Path: /var/lib/docker/containers/eb42afeec7f57d13998323eab6a2b36b300c990908e1109aa6382c2445fdc3ac/eb42afeec7f57d13998323eab6a2b36b300c990908e1109aa6382c2445fdc3ac-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 3c2a51cb7e04a6ffcd11fc6d92fff728ea08cae67282b2a7ad99115cadf6e9e3
      Path: /var/lib/docker/containers/3c2a51cb7e04a6ffcd11fc6d92fff728ea08cae67282b2a7ad99115cadf6e9e3/3c2a51cb7e04a6ffcd11fc6d92fff728ea08cae67282b2a7ad99115cadf6e9e3-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 8b37a11ddbee94e2d8319bb66dbfed8d6a3827a5cd7b5e6e7bb5a7b9bdee4389
      Path: /var/lib/docker/containers/8b37a11ddbee94e2d8319bb66dbfed8d6a3827a5cd7b5e6e7bb5a7b9bdee4389/8b37a11ddbee94e2d8319bb66dbfed8d6a3827a5cd7b5e6e7bb5a7b9bdee4389-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: f6056f30a6dc307a299fe8d3266b58474766bb7902c3f0e815ef7b863075e1f0
      Path: /var/lib/docker/containers/f6056f30a6dc307a299fe8d3266b58474766bb7902c3f0e815ef7b863075e1f0/f6056f30a6dc307a299fe8d3266b58474766bb7902c3f0e815ef7b863075e1f0-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
    - Type: file
      Identifier: 68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de
      Path: /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log
      Status: Error: cannot read file /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: stat /var/lib/docker/containers/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de/68b4cb1441c5f13fed02778fcc4dba4981b04ba5eb1b42ef4d9b64dc7a8426de-json.log: no such file or directory
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

=============
Process Agent
=============

  Version: 7.37.1
  Status date: 2022-08-25 17:39:39.294 UTC (1661449179294)
  Process Agent Start: 2022-08-25 17:38:49.654 UTC (1661449129654)
  Pid: 381
  Go Version: go1.17.11
  Build arch: amd64
  Log Level: info
  Enabled Checks: [process rtprocess]
  Allocated Memory: 13,206,000 bytes
  Hostname: ***CENSORED***

  =================
  Process Endpoints
  =================
    https://process.datadoghq.com - API Key ending with:
        - a358f

  =========
  Collector
  =========
    Last collection time: 2022-08-25 17:39:32
    Docker socket: /var/run/docker.sock
    Number of processes: 196
    Number of containers: 33
    Process Queue length: 0
    RTProcess Queue length: 0
    Pod Queue length: 0
    Process Bytes enqueued: 0
    RTProcess Bytes enqueued: 0
    Pod Bytes enqueued: 0
    Drop Check Payloads: []
=========
APM Agent
=========
  Status: Running
  Pid: 376
  Uptime: 50 seconds
  Mem alloc: 10,893,416 bytes
  Hostname: ***CENSORED***
  Receiver: 0.0.0.0:8126
  Endpoints:
    https://trace.agent.datadoghq.com

  Receiver (previous minute)
  ==========================
    From unknown clients
      Traces received: 0 (1 bytes)
      Spans received: 0

  Writer (previous minute)
  ========================
    Traces: 0 payloads, 0 traces, 0 events, 0 bytes
    Stats: 0 payloads, 0 stats buckets, 0 bytes

=========
Aggregator
=========
  Checks Metric Sample: 8,057
  Dogstatsd Metric Sample: 784
  Event: 40
  Events Flushed: 40
  Number Of Flushes: 3
  Series Flushed: 5,557
  Service Check: 34
  Service Checks Flushed: 34
=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 783
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 144,236
  Udp Packet Reading Errors: 0
  Udp Packets: 390
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 0
  Unterminated Metric Errors: 0

=============
Autodiscovery
=============
  Enabled Features
  ================
    docker

====
OTLP
====

  Status: Not enabled
  Collector status: Not running

the agent won't attach to the Docker socket to tail containers' stdout/stderr and isn't sending container logs to HQ.

Describe what happened: We predominantly use the awslogs driver to send logs to CloudWatch and as you can see the agent fails to find the containers' log files (because there are none). I had to set DD_LOGS_CONFIG_DOCKER_CONTAINER_USE_FILE to false to make the agent finally collect container logs but now any container that uses the json driver won't have its logs collected (I think).

Describe what you expected: Based on my reading of the Docker log collection from file issues:

The Agent collects Docker logs from the log files on disk by default in versions 6.33.0/7.33.0+ so long as the log files on disk are accessible by the Agent. DD_LOGS_CONFIG_DOCKER_CONTAINER_USE_FILE can be set to false to disable this behavior.

The Agent tails containers from the Docker socket if it cannot access the logs file path.

It looks like the agent currently doesn't tail containers from the Docker socket if it cannot access the logs file path.

Steps to reproduce the issue: Do as I did above.

Additional environment details (Operating System, Cloud provider, etc):

dallin-wright-anthid commented 1 year ago

Wow this solved my problem I have legitimately been stuck on this for 4 work days now and after multiple support cases and escalations leading nowhere and going back and forth with the sales rep, this is the option which is a break from how the agent used to work.

For all those who come after, the standard agent configuration via datadog.yaml from the installation on Linux does not expose this option as far as I can tell. I had to uninstall and execute via docker to get it o work.

carlosroman commented 1 year ago

Hi I was wondering if this was still an issue with the Agent [7.40.0](https://github.com/DataDog/datadog-agent/releases/tag/7.40.0) and above? The reason I'm asking is that we did some work to improve the way the Agent collects logs from containers (behind a feature flag called logs_config.cca_in_ad) which was enabled by default in Agents 7.40.0 and above. I tried to recreate your issue and wasn't able to do so with version 7.42.1 of the Agent.

My config was just:

logs_enabled: true
logs_config:
  container_collect_all: true

And I'd get:

==========
Logs Agent
==========
    Reliable: Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.eu on port 443
    BytesSent: 12132
    EncodedBytesSent: 5172
    LogsProcessed: 19
    LogsSent: 19

  docker
  ------
    - Type: docker
      Service: info-only
      Source: random
      Status: OK
        The log file tailer could not be made, falling back to socket
      Inputs:
        2d3192ca134c50f393065f4e398528d2720f5a1794fdc93d7e27a08efb085d77
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0
      Bytes Read: 1814
      Lines Combined: 8
      MultiLine matches: 9

This was running on:

Linux ****.compute.internal 4.14.301-224.520.amzn2.x86_64 #1 SMP Fri Dec 9 09:57:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ docker version
Client:
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.18.6
 Git commit:        100c701
 Built:             Sat Dec  3 04:13:49 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.6
  Git commit:       a89b842
  Built:            Sat Dec  3 04:14:27 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        5fd4c4d144137e991c4acebb2146ab1483a97925
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
farioas commented 1 year ago

The issue still reproducible in dd 7.42.1. Steps to reproduce:

carlosroman commented 1 year ago

@farioas We found a bug in our Docker tailer that was fixed in 7.43.0 of the Agent (#15138). I was wondering if you could try again with that version?

farioas commented 1 year ago

Hi @carlosroman,

We're running 7.43.0 since Feb 23, no issues observed. Anyway we keep DD_LOGS_CONFIG_DOCKER_CONTAINER_USE_FILE=false.