search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
3.02k stars 2.33k forks source link

receiver/hostmetrics reports metrics about the container instead of the host #16307

Closed liurui-1 closed 1 year ago

liurui-1 commented 1 year ago

Component(s)

receiver/hostmetrics

What happened?

Description

Some of the metrics got by receiver/hostmetrics are the metrics of the container itself. While we need the metrics for the host who owns the container. One typical example is system.filesystem.usage metric from filesystem ResourceMetrics. Here is the example, you can see it is the metric for the container not for the host. In the host, we do not have that file system with mount point to /etc/resolv.conf.

ResourceMetrics #1
Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/hostmetricsreceiver/filesystem 0.63.1

Metric #1
Descriptor:
     -> Name: system.filesystem.usage
     -> Description: Filesystem bytes used.
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative

NumberDataPoints #0
Data point attributes:
     -> device: Str(/dev/mapper/ubuntu--vg-root)
     -> mode: Str(rw)
     -> mountpoint: Str(/etc/resolv.conf)
     -> state: Str(used)
     -> type: Str(xfs)
StartTimestamp: 2022-11-11 04:34:25 +0000 UTC
Timestamp: 2022-11-15 01:16:37.599678646 +0000 UTC
Value: 119681974272

NumberDataPoints #1
Data point attributes:
     -> device: Str(/dev/mapper/ubuntu--vg-root)
     -> mode: Str(rw)
     -> mountpoint: Str(/etc/resolv.conf)
     -> state: Str(free)
     -> type: Str(xfs)
StartTimestamp: 2022-11-11 04:34:25 +0000 UTC
Timestamp: 2022-11-15 01:16:37.599678646 +0000 UTC
Value: 776874049536
...

Steps to Reproduce

Start the container and enable any exporter, you can find that metric system.filesystem.usage

Expected Result

I expect the metric system.filesystem.usage for the host instead of the container.

Actual Result

The metric system.filesystem.usage is for the container instead of the host.

Collector version

0.63.1

Environment information

Environment

Ubuntu 18.04

OpenTelemetry Collector configuration

receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
      memory:
      load:
      network:
      processes:
      process:

  hostmetrics/disk:
    collection_interval: 1m
    scrapers:
      disk:
      filesystem:
      paging:

exporters:
  otlp:
    endpoint: "10.21.15.86:4317"
    tls:
      insecure: true

  prometheus:
    endpoint: "0.0.0.0:8889"

  logging:
    verbosity: detailed

processors:
  batch:

extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    metrics:
      receivers: [hostmetrics, hostmetrics/disk]
      processors: [batch]
      exporters: [otlp, logging, prometheus]

Log output

2022/11/15 01:13:37 proto: duplicate proto type registered: jaeger.api_v2.PostSpansRequest
2022/11/15 01:13:37 proto: duplicate proto type registered: jaeger.api_v2.PostSpansResponse
2022-11-15T01:13:37.582Z        info    service/telemetry.go:110        Setting up own telemetry...
2022-11-15T01:13:37.582Z        info    service/telemetry.go:140        Serving Prometheus metrics      {"address": ":8888", "level": "basic"}
2022-11-15T01:13:37.582Z        info    components/components.go:30     In development component. May change in the future.     {"kind": "export
er", "data_type": "metrics", "name": "logging", "stability": "in development"}
2022-11-15T01:13:37.593Z        info    service/service.go:89   Starting otelcol-contrib...     {"Version": "0.63.1", "NumCPU": 8}
2022-11-15T01:13:37.593Z        info    extensions/extensions.go:42     Starting extensions...
2022-11-15T01:13:37.593Z        info    extensions/extensions.go:45     Extension is starting...        {"kind": "extension", "name": "pprof"}
2022-11-15T01:13:37.593Z        info    pprofextension@v0.63.0/pprofextension.go:71     Starting net/http/pprof server  {"kind": "extension", "n
ame": "pprof", "config": {"TCPAddr":{"Endpoint":":1888"},"BlockProfileFraction":0,"MutexProfileFraction":0,"SaveToFile":""}}
2022-11-15T01:13:37.593Z        info    extensions/extensions.go:49     Extension started.      {"kind": "extension", "name": "pprof"}
2022-11-15T01:13:37.593Z        info    extensions/extensions.go:45     Extension is starting...        {"kind": "extension", "name": "zpages"}
2022-11-15T01:13:37.593Z        info    zpagesextension/zpagesextension.go:64   Registered zPages span processor on tracer provider     {"kind":
 "extension", "name": "zpages"}
2022-11-15T01:13:37.594Z        info    zpagesextension/zpagesextension.go:74   Registered Host's zPages        {"kind": "extension", "name": "z
pages"}
2022-11-15T01:13:37.594Z        info    zpagesextension/zpagesextension.go:86   Starting zPages extension       {"kind": "extension", "name": "z
pages", "config": {"TCPAddr":{"Endpoint":":55679"}}}
2022-11-15T01:13:37.594Z        info    extensions/extensions.go:49     Extension started.      {"kind": "extension", "name": "zpages"}
2022-11-15T01:13:37.594Z        info    extensions/extensions.go:45     Extension is starting...        {"kind": "extension", "name": "health_ch
eck"}
2022-11-15T01:13:37.594Z        info    healthcheckextension@v0.63.0/healthcheckextension.go:44 Starting health_check extension {"kind": "extens
ion", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMet
adata":false,"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2022-11-15T01:13:37.594Z        info    extensions/extensions.go:49     Extension started.      {"kind": "extension", "name": "health_check"}
2022-11-15T01:13:37.594Z        info    pipelines/pipelines.go:74       Starting exporters...
2022-11-15T01:13:37.594Z        info    pipelines/pipelines.go:78       Exporter is starting... {"kind": "exporter", "data_type": "metrics", "na
me": "otlp"}
2022-11-15T01:13:37.595Z        info    pipelines/pipelines.go:82       Exporter started.       {"kind": "exporter", "data_type": "metrics", "na
me": "otlp"}
2022-11-15T01:13:37.595Z        info    pipelines/pipelines.go:78       Exporter is starting... {"kind": "exporter", "data_type": "metrics", "na
me": "logging"}
2022-11-15T01:13:37.595Z        info    pipelines/pipelines.go:82       Exporter started.       {"kind": "exporter", "data_type": "metrics", "na
me": "logging"}
2022-11-15T01:13:37.595Z        info    pipelines/pipelines.go:78       Exporter is starting... {"kind": "exporter", "data_type": "metrics", "na
me": "prometheus"}
2022-11-15T01:13:37.595Z        info    pipelines/pipelines.go:82       Exporter started.       {"kind": "exporter", "data_type": "metrics", "na
me": "prometheus"}
2022-11-15T01:13:37.595Z        info    pipelines/pipelines.go:86       Starting processors...
2022-11-15T01:13:37.595Z        info    pipelines/pipelines.go:90       Processor is starting...        {"kind": "processor", "name": "batch", "
pipeline": "metrics"}
2022-11-15T01:13:37.595Z        info    pipelines/pipelines.go:94       Processor started.      {"kind": "processor", "name": "batch", "pipeline
": "metrics"}
2022-11-15T01:13:37.595Z        info    pipelines/pipelines.go:98       Starting receivers...
2022-11-15T01:13:37.595Z        info    pipelines/pipelines.go:102      Receiver is starting... {"kind": "receiver", "name": "hostmetrics/disk",
 "pipeline": "metrics"}
2022-11-15T01:13:37.595Z        info    pipelines/pipelines.go:106      Receiver started.       {"kind": "receiver", "name": "hostmetrics/disk",
 "pipeline": "metrics"}
2022-11-15T01:13:37.595Z        info    pipelines/pipelines.go:102      Receiver is starting... {"kind": "receiver", "name": "hostmetrics", "pip
eline": "metrics"}

Additional context

No response

github-actions[bot] commented 1 year ago

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

dmitryax commented 1 year ago

This is expected behavior. If you run the receiver from a container -> container is a host. If you need to collect host metrics from a container, see https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver#collecting-host-metrics-from-inside-a-container-linux-only

dmitryax commented 1 year ago

Closing the issue as answered, feel free to reopen otherwise