Resource detection for EC2 partially works if collector is running as a systemd service on an EC2

[stephenhong]

Describe the bug Hello team, I'm running the Otel collector as a systemd service on an ec2 instance. I have resource detection processor enabled on the collector with detectors: ["ec2"] and added configuration to gather tags for the EC2 instance. However it does not detect the EC2 information.

• If I run the collector plainly on the instance host and not as a systemd service or

• If I run the collector as a systemd service and only have detectors: ["ec2"] without the tags parameter

  the resource detection for EC2 starts working again.

Steps to reproduce Enable detectors: ["ec2"] under resource detection processor with the tags parameter in config.yaml Run Otel collector as a systemd service

What did you expect to see? I expected to see something similar to this in the collector startup log info internal/resourcedetection.go:150 detected resource information {"kind": "processor", "name": "resourcedetection", "pipeline": "traces", "resource": {"cloud.account.id":"XXXXXXXXXXXX","cloud.availability_zone":"us-east-1c","cloud.platform":"aws_ec2","cloud.provider":"aws","cloud.region":"us-east-1","host.id":"i-XXXXXXXXXXXXXXXXX","host.image.id":"ami-XXXXXXXXXXXXXXXXX","host.name":"ip-XX-XX-XX-XX","host.type":"m5.xlarge"}}

What did you see instead? warn internal/resourcedetection.go:141 failed to detect resource {"kind": "processor", "name": "resourcedetection", "pipeline": "traces", "error": "failed fetching ec2 instance tags: RequestError: send request failed\ncaused by: Post \"https://ec2.us-east-1.amazonaws.com/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}

What version did you use? Version: v0.58.0

What config did you use? Config:

receivers:
  hostmetrics:
    collection_interval: 10m
    scrapers:
      cpu:
      disk:
      load:
      filesystem:
      memory:
      network:
      paging:
      processes:
      process:
  docker_stats:
    collection_interval: 5m
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
    loglevel: debug
  otlp:
    endpoint: $OTLP_ENDPOINT

processors:
  batch:
  resourcedetection:
    detectors: ["ec2"]
    ec2:
      tags:
    - ^ASV$
        - ^BA$

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, otlp]
      processors: [batch, resourcedetection]
    metrics:
      receivers: [otlp, hostmetrics, docker_stats]
      exporters: [otlp]
      processors: [batch]
    logs:
      receivers: [otlp, filelog]
      exporters: [otlp]
      processors: [batch]

Environment OS: Linux 5.10.130-118.517.amzn2.x86_64 Compiler(if manually compiled): (e.g., "go 14.2")

Additional context Add any other context about the problem here.

[github-actions[bot]]

Pinging code owners: @jrcamp @Aneurysm9 @dashpole. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[EddieEldridge]

Can you share your system.d config? This works fine for me running via system.d.

[Aneurysm9]

Based on the included error this appears to be related to a failure to retrieve tag metadata from the EC2 API. The proposal here to use the IMDS to retrieve the tag data may help.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/resourcedetection: @jrcamp @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[fatsheep9146]

any update ? @tephenhong

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/resourcedetection: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions[bot]]

This issue has been closed as inactive because it has been stale for 120 days with no activity.