otel collector failing to collect/export metrics when datadog exporter is unable to retrieve host metadata using gohai

michelleN commented 1 year ago

Component(s)

exporter/datadog

What happened?

Description

I'm not sure if this is a bug of if I am using datadog

Steps to Reproduce

Run otel collector using datadog exporter

Expected Result

Run without errors

Actual Result

The collector starts up fine and later I see errors related to being unable to fetch host metadata and I'm seeing that the metrics are neither being collected nor are they being exported.

Collector version

0.62.1

Environment information

Environment

OS: Ubuntu 22.04 Docker image: otel/opentelemetry-collector-contrib:0.62.1

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:<port>
exporters:
  file:
    path: ./metrics.json
  datadog:
    api:
      site: datadoghq.com
      key: *******
processors:
  batch:
extensions:
  health_check:
  pprof:
    endpoint: :<port>
  zpages:
    endpoint: :<port>
service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [file, datadog]

Log output

From stderr:

gohai/gohai.go:47   Failed to retrieve filesystem metadata  {"kind": "exporter", "data_type": "metrics", "name": "datadog", "error": "df failed to collect filesystem data: %!s(<nil>)"}

warn    gohai/gohai.go:66   Failed to retrieve platform metadata    {"kind": "exporter", "data_type": "metrics", "name": "datadog", "error": "exec: \"uname\": executable file not found in $PATH"}
2022-10-21T20:11:54.639Z    info    metadata/metadata.go:216    Sent host metadata  {"kind": "exporter", "data_type": "metrics", "name": "datadog"}

From stdout: [Debug] Error fetching info for pid 1: open /etc/passwd: no such file or directory



### Additional context

I'm trying to understand if this is expected behavior or a bug and what exactly is causing these issues. Is it a docker related config issue? I'm running on Nomad so I'm also wondering if it has anything to do with Nomad. Would appreciate any help here debugging/understanding/fixing. Thanks.

github-actions[bot] commented 1 year ago

Pinging code owners:

exporter/datadog: @KSerrania @mx-psi @gbbr @knusbaum @amenasria @dineshg13

See Adding Labels via Comments if you do not have permissions to add labels yourself.

dineshg13 commented 1 year ago

This is duplicate of https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/14186 , we will look into this and fix asap.

mx-psi commented 1 year ago

Thanks for reporting @michelleN, I am closing as a duplicate of #14186; feel free to add aditional details over there

open-telemetry / opentelemetry-collector-contrib