NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors when using AWS X-Ray Exporter

[swapnil-dargude-rp]

Describe the bug While using AWS Process credential provider, I encountered with an issue with the AWS OpenTelemetry Collector when trying to export metric and trace to AWS X-Ray . The collector fails with a error "NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors". Below are the details of my setup and the error logs.

OpenTelemetry Collector Configuration (otel-collector-config.yaml)

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:55681

processors:
  batch:

exporters:
  awsxray:
    region: 'us-east-1'
  logging:
    verbosity: detailed
  awsemf:
    log_group_name: "log_group_name"
    log_stream_name: "log_stream_name"
    region: "us-east-1"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsemf]

docker-compose.yaml

  sr-otel-collector:
    container_name: otel-collector
    image: public.ecr.aws/aws-observability/aws-otel-collector:latest
    command: ["--config", "/etc/otel-collector-config.yaml"]
    networks:
      - proxy
    volumes:
      - ./OtelCollector/otel-collector-config.yaml:/etc/otel-collector-config.yaml
      - ./config/.aws:/home/aoc/.aws
      - ./config/.aws/config:/home/aoc/.aws/config:ro
      - ./config/.aws/credentials:/home/aoc/.aws/credentials:ro
    environment:
      - AWS_PROFILE=rolesanywhere
      - AWS_REGION=us-east-1
      - HOME=/home/aoc
      - AWS_SDK_LOAD_CONFIG=1
      - AWS_CONFIG_FILE=/home/aoc/.aws/config
      - AWS_SHARED_CREDENTIALS_FILE=/home/aoc/.aws/credentials
    ports:
      - "55681:55681"
      - "4317:4317"
      - "4318:4318"

AWS ~/.aws/config:

[default]
region = us-east-1
output = json
[profile rolesanywhere]
credential_process = /home/aoc/.aws/aws_signing_helper credential-process --certificate /home/aoc/.aws/certificate.pem --private-key /home/aoc/.aws/decrypted_key.pem --trust-anchor-arn arn:aws:rolesanywhere:us-east-1:XXXXXXXX:trust-anchor/xxxx-xxxx-xxxx-xxxx --profile-arn arn:aws:rolesanywhere:us-east-1:XXXXXXXXX:profile/xxxx-xxxx-xxxx-xxxx --role-arn arn:aws:iam::XXXXXXXXXX:role/rolesanywhere-role
region = us-east-1
output = json

AWS ~/.aws/credentials:

[default]
aws_access_key_id=XXXXXXXXXXXXXXXX
aws_secret_access_key=xxxxxx
region= us-east-1

What did you expect to see? I expected the AWS OpenTelemetry Collector to successfully authenticate using AWS Process credential provider profile rolesanywhere and send trace data to AWS X-Ray.

What did you see instead?

Instead of the expected behavior, I encountered the following error in the logs:

{"kind": "exporter", "data_type": "traces", "name": "awsxray"}
2024-06-24T16:41:06.143Z    debug   awsxrayexporter@v0.98.0/awsxray.go:72   response error  {"kind": "exporter", "data_type": "traces", "name": "awsxray", "error": "NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"}
2024-06-24T16:41:06.143Z    debug   awsxrayexporter@v0.98.0/awsxray.go:79   response: {
}   {"kind": "exporter", "data_type": "traces", "name": "awsxray"}
2024-06-24T16:41:06.143Z    error   exporterhelper/common.go:292    Exporting failed. Rejecting data.   {"kind": "exporter", "data_type": "traces", "name": "awsxray", "error": "NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors", "rejected_items": 3}
go.opentelemetry.io/collector/exporter/exporterhelper.(*baseExporter).send
    go.opentelemetry.io/collector/exporter@v0.98.0/exporterhelper/common.go:292
go.opentelemetry.io/collector/exporter/exporterhelper.NewTracesRequestExporter.func1
    go.opentelemetry.io/collector/exporter@v0.98.0/exporterhelper/traces.go:134
go.opentelemetry.io/collector/consumer.ConsumeTracesFunc.ConsumeTraces
    go.opentelemetry.io/collector/consumer@v0.98.0/traces.go:25
go.opentelemetry.io/collector/processor/batchprocessor.(*batchTraces).export
    go.opentelemetry.io/collector/processor/batchprocessor@v0.98.0/batch_processor.go:414
go.opentelemetry.io/collector/processor/batchprocessor.(*shard).sendItems
    go.opentelemetry.io/collector/processor/batchprocessor@v0.98.0/batch_processor.go:261

Environment OS: Red Hat Enterprise Linux (Build 4.18.0-305.88.1.el8_4.x86_64) (64-bit) using docker container

Additional context I have setup IAM roles anywhere for process credential provider

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue was closed because it has been marked as stale for 30 days with no activity.