Particular hostmetrics export failure: The start time must be before the end time

tmodelsk commented 3 years ago

Describe the bug Particular metrics (sum metrics) that are from hostmetrics receiver cannot be exported to google monitoring api. The error message is: "The start time must be before the end time" Affected hostmetrics are:

process.memory.physical_usage
process.memory.virtual_usage
system.disk.pending_operations
system.processes.count
system.processes.created
system.memory.usage
system.filesystem.usage
system.filesystem.inodes.usage
system.network.connections
system.paging.usage
system.cpu.load_average.1m
system.cpu.load_average.5m
system.cpu.load_average.15m

Other hostmetrics (ie: system.cpu.time) are properly exported.

Further investigations (debug & file exporter log) shows that all affected metrics:

don't have startTimeUnixNano field (file exporter log)
field StartTimestamp is 0 (debugging, pdata.Metrics data structure)
simple workaround described at the very bottom

Steps to reproduce Create metrics pipeline with hostmetrics receiver and googlecloud exporter

What did you expect to see? All hostmetrics values should be exported to monitoring api, given metrics values should be soon visible in metrics explorer GUI and queryable by rest api https://monitoring.googleapis.com/v3/projects/PROJECT_ID/timeSeries

What did you see instead? Errors in otel output console are like: rpc error: code = InvalidArgument desc = Field timeSeries[3].points[0].interval.start_time had an invalid value of \"2021-04-26T04:17:28.120024-07:00\": The start time must be before the end time (2021-04-26T04:17:28.120024-07:00) for the non-gauge metric 'external.googleapis.com/prometheus/process.memory.physical_usage'.;

Values of metrics mentioned above are not exported to google monitoring. Only some hostmetrics values (ie: system.cpu.time) are there.

What version did you use? Release versions 0.24.0 both for opentelemetry-collector-builder & opentelemetry-collector-contrib/exporter/googlecloudexporter

What config did you use? Otel config:

receivers:
  hostmetrics: 
    collection_interval: 30s
    scrapers:
      cpu:
      disk:
      load:
      filesystem:
      memory:
      network:
      paging:
      processes:
      process:
processors: 
  resource: 
    attributes:
      # gke settings   
exporters:
  googlecloud:
    project: PROJECT_ID
    metric:
      prefix: "external.googleapis.com/prometheus"
      skip_create_descriptor: false
service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resource]
      exporters: [googlecloud]

Additional context Simple working workaround (assign missing StartTimestamp := Timestamp - 1microSecond) is here: https://github.com/SabreOSS/opentelemetry-collector-contrib/commit/d6795c9b8da0cd77dc8b42c2f4e33066958b0aae With this one all hostmetrics values are exported.

bogdandrutu commented 3 years ago

@jsuereth what do you think about this?

jsuereth commented 3 years ago

Sorry I missed this comment. The OpenTelemetry GCP metrics exporter for Go does do the suggested fix, but the current collector exporter is based on OpenCensus where that was unnecessary. This should be considered a bug against the Google Cloud exporters in the collector.

dashpole commented 3 years ago

I think this should be fixed by https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/3360

open-telemetry / opentelemetry-collector-contrib

Particular hostmetrics export failure: The start time must be before the end time #3278