Sender failed {"kind": "processor", "name": "batch", "pipeline": "traces", "error": "NoCredentialProviders: no valid providers in chain.

[sharul121]

I am running awsotelcollector as docker service in AWS EC2 host and my EC2 machine is attached to an instance profile with a suitable role having permissions for AWS XRAY. Also i am using latest image of awsotelcollector from docker hub.

I have mounted .aws directory in the docker container for the awsotelcollector to take profile from ~/.aws/credentials file. The content of ~/.aws/credentials file is as follows:

[default]
role_arn = arn:aws:iam::xxx:role/xxx"
credential_source = Ec2InstanceMetadata
region = eu-central-1

However i cannot see traces been generated in AWS XRAY and the logs are filled with these errors:

2022-11-15T13:32:28.542Z    error   exporterhelper/queued_retry.go:361  Exporting failed. Try enabling retry_on_failure config option to retry on retryable errors  {"kind": "exporter", "data_type": "traces", "name": "awsxray", "error": "NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
    go.opentelemetry.io/collector@v0.62.1/exporter/exporterhelper/queued_retry.go:361
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
    go.opentelemetry.io/collector@v0.62.1/exporter/exporterhelper/traces.go:134
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send
    go.opentelemetry.io/collector@v0.62.1/exporter/exporterhelper/queued_retry.go:295
go.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2
    go.opentelemetry.io/collector@v0.62.1/exporter/exporterhelper/traces.go:113
go.opentelemetry.io/collector/consumer.ConsumeTracesFunc.ConsumeTraces
    go.opentelemetry.io/collector@v0.62.1/consumer/traces.go:36
go.opentelemetry.io/collector/service/internal/fanoutconsumer.(*tracesConsumer).ConsumeTraces
    go.opentelemetry.io/collector@v0.62.1/service/internal/fanoutconsumer/traces.go:77
go.opentelemetry.io/collector/processor/batchprocessor.(*batchTraces).export
    go.opentelemetry.io/collector@v0.62.1/processor/batchprocessor/batch_processor.go:262
go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).sendItems
    go.opentelemetry.io/collector@v0.62.1/processor/batchprocessor/batch_processor.go:176
go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).processItem
    go.opentelemetry.io/collector@v0.62.1/processor/batchprocessor/batch_processor.go:155
go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).startProcessingCycle
    go.opentelemetry.io/collector@v0.62.1/processor/batchprocessor/batch_processor.go:140
2022-11-15T13:32:28.542Z    error   exporterhelper/queued_retry.go:297  Exporting failed. Dropping data. Try enabling sending_queue to survive temporary failures.  {"kind": "exporter", "data_type": "traces", "name": "awsxray", "dropped_items": 70}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send
    go.opentelemetry.io/collector@v0.62.1/exporter/exporterhelper/queued_retry.go:297
go.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2
    go.opentelemetry.io/collector@v0.62.1/exporter/exporterhelper/traces.go:113
go.opentelemetry.io/collector/consumer.ConsumeTracesFunc.ConsumeTraces
    go.opentelemetry.io/collector@v0.62.1/consumer/traces.go:36
go.opentelemetry.io/collector/service/internal/fanoutconsumer.(*tracesConsumer).ConsumeTraces
    go.opentelemetry.io/collector@v0.62.1/service/internal/fanoutconsumer/traces.go:77
go.opentelemetry.io/collector/processor/batchprocessor.(*batchTraces).export
    go.opentelemetry.io/collector@v0.62.1/processor/batchprocessor/batch_processor.go:262
go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).sendItems
    go.opentelemetry.io/collector@v0.62.1/processor/batchprocessor/batch_processor.go:176
go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).processItem
    go.opentelemetry.io/collector@v0.62.1/processor/batchprocessor/batch_processor.go:155
go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).startProcessingCycle
    go.opentelemetry.io/collector@v0.62.1/processor/batchprocessor/batch_processor.go:140
2022-11-15T13:32:28.542Z    info    TracesExporter  {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 70}
2022-11-15T13:32:28.542Z    warn    batchprocessor/batch_processor.go:178   Sender failed   {"kind": "processor", "name": "batch", "pipeline": "traces", "error": "NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"}

Can Someone help why it is not able to find the credentials?

[rapphil]

Hi @sharul121 .

Since you are running docker in EC2, you can attach an IAM role to your instance and use this mechanism to provide credentials to the collector. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#working-with-iam-roles

This example will work out of the box if you use IAM Roles attached to the instance: https://github.com/aws-observability/aws-otel-collector/blob/main/examples/docker/docker-compose.yaml

You just need to install docker compose and run:

docker-compose up

If you still want to use a credentials and config file, you can use environment variables to explicitly set the path to those files: AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE. https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-where

Please notice that the collector process runs as root in the docker container. As such, if you want to use everything with defaults by not setting the previously mentioned environment variables, you have to mount the credentials/config file in the directory /root/.aws

NOTE: ~/.aws/credentials is used to store the credentials while the content that you shared in the comments is for the ~/.aws/config file: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-where

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue was closed because it has been marked as stale for 30 days with no activity.