rsmaso-aws
commented
2 years ago cc @mhausenblas
Closed rsmaso-aws closed 1 year ago
rsmaso-aws
commented
2 years ago cc @mhausenblas
mhausenblas
commented
2 years ago As per @willarmiros:
The underlying error comes from the AWS SDK for Go. It basically means the AWS SDK for Go can’t find credentials in its typical expected places, see for example, https://github.com/aws/aws-xray-daemon/issues/59
mhausenblas
commented
2 years ago I can reproduce the issue, tried different collector configs and methods to pass credentials as well as passing in NO_PROXY=169.254.169.254 to test if it's related to the metadata service. As long as there is no IAM role attached, the NoCredentialProviders: no valid providers in chain. Deprecated error occurs.
mhausenblas
commented
2 years ago 
jeromeinsf
commented
2 years ago Given this comment https://github.com/aws/aws-sdk-go/issues/2914#issuecomment-803177408 , how do you recommend tracking the ask on the SDK side of things? @mhausenblas @Aneurysm9 Also, is a PR thinkable ?
mhausenblas
commented
2 years ago @jeromeinsf as discussed last week, internal
github-actions[bot]
commented
2 years ago This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
rsmaso-aws
commented
2 years ago Hi @mhausenblas, is there any updates on this issue? Thanks in advance
mhausenblas
commented
2 years ago @rsmaso-aws it's on our backlog, hopefully this quarter, Q1/2023 otherwise.
rapphil
commented
2 years ago Hi @rsmaso-aws ,
You are not able to setup the credentials with sudo AWS_REGION=us-east-1 AWS_CONFIG_FILE="/home/ubuntu/.aws/credentials" /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -c ./config.yaml -a start or similar because aws-otel-collector-ctl is a control process and not the collector itself.
Here are a some options to solve your problem:
/opt/aws/aws-otel-collector/etc/.env. Each line in this file contains an env variable definition. You can append new env vars to that file so that it should look like:config=--config /opt/aws/aws-otel-collector/etc/config.yaml
AWS_SHARED_CREDENTIALS_FILE=/path/file/credentialssudo systemctl edit aws-otel-collector
# Add the AWS_SHARED_CREDENTIALS_FILE to the override of the systemd file. ref: https://www.freedesktop.org/software/systemd/man/systemd.service.html#Command%20lines
# Example of what you should add to the override:
[Service]
Environment="AWS_SHARED_CREDENTIALS_FILE=/path/to/creds"aoc user, is to do the following
sudo mkdir -p /home/aoc/.aws
sudo chown -R aoc:aoc /home/aoc
cp /path/credentials /home/aoc/.aws/credentialsAfter you set the environment variable, you can try to stop/start the collector and the credentials should be used.
The first two options bellow will also support specifying credentials through the env vars AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN, but I recommend using AWS_SHARED_CREDENTIALS_FILE.
We will update the documentation to detail how to install collector on-premises or in ec2 without profiles.
NOTE: AWS_SHARED_CREDENTIALS_FILE is what you are looking for to set the credentials file. AWS_CONFIG_FILE is used for storing configuration profiles.
RyanW8
commented
1 year ago We're trying to use the shared credentials approach when running the ADOT collector in EKS. We don't want to use IRSA so are using IAM credentials injected into the pod see below:
---
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: adot
spec:
image: <private-registry>/public-ecr/aws-observability/aws-otel-collector:v0.24.1
mode: deployment
serviceAccount: adot-collector
podAnnotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "<vault-role>"
vault.hashicorp.com/agent-cache-enable: "true"
vault.hashicorp.com/agent-revoke-on-shutdown: "true"
vault.hashicorp.com/agent-inject-secret-creds: "fake/path"
vault.hashicorp.com/agent-inject-template-creds: |
{{- with secret "aws-creds" }}
[default]
aws_access_key_id = {{index .Data "access_key" }}
aws_secret_access_key = {{index .Data "secret_key" }}
{{- end -}}
env:
- name: AWS_SHARED_CREDENTIALS_FILE
value: "/vault/secrets/creds"
config: |
extensions:
health_check:
receivers:
otlp:
protocols:
grpc:
http:
exporters:
awsxray:
region: eu-west-1
processors:
batch/traces:
timeout: 1s
send_batch_size: 50
service:
extensions:
- health_check
pipelines:
traces:
receivers:
- otlp
processors:
- batch/traces
exporters:
- awsxrayWe're still not getting traces into X-Ray
rapphil
commented
1 year ago Hi @RyanW8 . did you check the logs for this workload? If credentials was the issue, you should see an error log.
Moreover, can you try to use the logging exporter to verify if your application is really generating traces?
github-actions[bot]
commented
1 year ago This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
utahkay
commented
1 year ago I have the same problem, trying to run on EC2, following the instructions here. I'm running v0.28.0 of the aws-otel-collector.
I tried all three suggestions by @rapphil to specify credentials, but I still get the error
"error":"NoCredentialProviders: no valid providers in chain. Deprecated.The original issue description from @rsmas-aws says
The issue does NOT appear when there is an IAM role attached to the EC2 instance
Maybe I can use an IAM role; where can I find info on what needs to be in that role?
utahkay
commented
1 year ago I was able to find a good example policy here; using an IAM role for the EC2 instance, I am able to send metrics to CloudWatch.
github-actions[bot]
commented
1 year ago This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
lphaniKumar
commented
1 year ago I am seeing similar issue when using IAM Roles anywhere for an Onpremise system and not able to post the data to AWS CloudWatch using EMF Exporter. Is there a fix/workaround for this? @mhausenblas
rapphil
commented
1 year ago Hi @lphaniKumar
Please take a look into the updated documentation: https://aws-otel.github.io/docs/setup/on-premises#configuring-adot-collector-to-use-iam-roles-anywhere (make sure to refresh the page)
there was a miss in the documentation and setting the env var "AWS_SDK_LOAD_CONFIG=true" is necessary.
github-actions[bot]
commented
1 year ago This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
github-actions[bot]
commented
1 year ago This issue was closed because it has been marked as stale for 30 days with no activity.
Describe the bug When running the AWS OpenTelemetry collector v1.17.1+ in an on-premises setup or on an EC2 instance that has no IAM role attached, AOC is not able to recognize AWS credentials, no matter how they are provided (via env, file) and no matter what exporter for an AWS service (such as X-Ray, CW, AMP) is used fails with an
NoCredentialProviders: no valid providers in chainerror message in the logs.Steps to reproduce
/home/ubuntu/.aws/credentialsor export them using ENV vars.https://aws-otel-collector.s3.amazonaws.com/ubuntu/amd64/v0.18.0/aws-otel-collector.debsudo AWS_REGION=us-east-1 AWS_CONFIG_FILE="/home/ubuntu/.aws/credentials" /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -c ./config.yaml -a startAlternatively:sudo AWS_REGION=us-east-1 AWS_ACCESS_KEY_ID =*** AWS_SECRET_ACCESS_KEY=*** /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -c ./config.yaml -a startWhat did you expect to see? I expect the collector to properly authenticate and communicate with AWS X-Ray using provided credentials!
What did you see instead?
Environment
Tested on:
Additional context Fresh collector log after start:
Content of
config.yaml