Error while matching k8s services

[nhoughto]

====================
Collector (v 5.24.0)
====================

  Status date: 2018-05-23 06:04:03 (3s ago)
  Pid: 38
  Platform: Linux-4.14.19-coreos-x86_64-with-debian-9.4
  Python Version: 2.7.14, 64bit
  Logs: <stderr>, /var/log/datadog/collector.log

  Clocks
  ======

    NTP offset: Unknown ([Errno -8] Servname not supported for ai_socktype)
    System UTC time: 2018-05-23 06:04:07.113004

  Paths
  =====

    conf.d: /etc/dd-agent/conf.d
    checks.d: Not found

  Hostnames
  =========

    ec2-hostname: removed
    local-ipv4: removed
    socket-hostname: removed
    hostname: removed
    local-hostname: removed
    instance-id: removed
    socket-fqdn: removed

  Checks
  ======

    kube_dns (1.3.0)
    ----------------
      - instance #0 [OK]
      - Collected 44 metrics, 0 events & 0 service checks

    network (1.5.0)
    ---------------
      - instance #0 [OK]
      - Collected 260 metrics, 0 events & 0 service checks

    kubernetes (1.5.0)
    ------------------
      - instance #0 [OK]
      - Collected 782 metrics, 0 events & 3 service checks

    ntp (1.2.0)
    -----------
      - Collected 0 metrics, 0 events & 0 service checks

    disk (1.2.0)
    ------------
      - instance #0 [OK]
      - Collected 80 metrics, 0 events & 0 service checks

    kube_proxy (Unknown Wheel)
    --------------------------
      - Collected 0 metrics, 0 events & 0 service checks

    docker_daemon (1.10.0)
    ----------------------
      - instance #0 [OK]
      - Collected 222 metrics, 0 events & 1 service check

  Emitters
  ========

    - http_emitter [OK]

====================
Dogstatsd (v 5.24.0)
====================

  Status date: 2018-05-23 06:04:03 (4s ago)
  Pid: 31
  Platform: Linux-4.14.19-coreos-x86_64-with-debian-9.4
  Python Version: 2.7.14, 64bit
  Logs: <stderr>, /var/log/datadog/dogstatsd.log

  Flush count: 143
  Packet Count: 118268
  Packets per second: 29.0
  Metric count: 881
  Event count: 0
  Service check count: 0

====================
Forwarder (v 5.24.0)
====================

  Status date: 2018-05-23 06:04:04 (4s ago)
  Pid: 25
  Platform: Linux-4.14.19-coreos-x86_64-with-debian-9.4
  Python Version: 2.7.14, 64bit
  Logs: <stderr>, /var/log/datadog/forwarder.log

  Queue Size: 57665 bytes
  Queue Length: 3
  Flush Count: 476
  Transactions received: 334
  Transactions flushed: 331
  Transactions rejected: 0
  API Key Status: API Key is valid

======================
Trace Agent (v 5.24.0)
======================

  Pid: 24
  Uptime: 1473 seconds
  Mem alloc: 1156400 bytes

  Hostname: removed
  Receiver: 0.0.0.0:8126
  API Endpoint: https://trace.agent.datadoghq.com

  --- Receiver stats (1 min) ---

  --- Writer stats (1 min) ---

  Traces: 0 payloads, 0 traces, 0 bytes
  Stats: 0 payloads, 0 stats buckets, 0 bytes
  Services: 0 payloads, 0 services, 0 bytes

Additional environment details (Operating System, Cloud provider, etc): coreOS, Kube 1.9 from KOPS 1.9 on AWS.

Steps to reproduce the issue:

1. Configure dd-agent as daemonset as per documentations / helm chart
2. Enable event collection and leader election via kubernetes.yaml config
3. Look at container collector logs, /var/log/collector.log
4. Kubernetes Event collection fails

Describe the results you received:

2018-05-23 06:14:19 UTC | INFO | dd.collector | utils.cloud_metadata(cloud_metadata.py:275) | Attempting to get OpenStack meta_data.json
2018-05-23 06:14:19 UTC | INFO | dd.collector | utils.cloud_metadata(cloud_metadata.py:295) | Could not load meta_data.json, not OpenStack EC2 instance
2018-05-23 06:14:19 UTC | INFO | dd.collector | utils.pidfile(pidfile.py:35) | Pid file is: /opt/datadog-agent/run/dd-agent.pid
2018-05-23 06:14:19 UTC | INFO | dd.collector | collector(agent.py:559) | Agent version 5.24.0
2018-05-23 06:14:19 UTC | INFO | dd.collector | daemon(daemon.py:234) | Starting
2018-05-23 06:14:20 UTC | INFO | dd.collector | config(config.py:998) | no bundled checks.d path (checks provided as wheels): /opt/datadog-agent/agent/checks.d
2018-05-23 06:14:20 UTC | INFO | dd.collector | config(config.py:973) | Fetching service discovery check configurations.
2018-05-23 06:14:22 UTC | INFO | dd.collector | config(config.py:998) | no bundled checks.d path (checks provided as wheels): /opt/datadog-agent/agent/checks.d
2018-05-23 06:14:24 UTC | INFO | dd.collector | config(config.py:998) | no bundled checks.d path (checks provided as wheels): /opt/datadog-agent/agent/checks.d
2018-05-23 06:14:24 UTC | ERROR | dd.collector | collector(pod_service_mapper.py:104) | Error while matching k8s services: 'uid'
Traceback (most recent call last):
  File "/opt/datadog-agent/agent/utils/kubernetes/pod_service_mapper.py", line 82, in match_services_for_pod
    pod_id = pod_metadata['uid']
KeyError: 'uid'

Describe the results you expected: No errors, and the Kubernetes check to report events.

[nhoughto]

Disabling collect_service_tags in kubernetes.yaml config avoids the error it seems, not sure what the impact of that is though.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. Note that the issue will not be automatically closed, but this notification will remind us to investigate why there's been inactivity. Thank you for participating in the Datadog open source community.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. Note that the issue will not be automatically closed, but this notification will remind us to investigate why there's been inactivity. Thank you for participating in the Datadog open source community.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. Note that the issue will not be automatically closed, but this notification will remind us to investigate why there's been inactivity. Thank you for participating in the Datadog open source community.