Debugging OTEL agent on EC2 - logging exporter

[georges-git]

Hello @mxiamxia - What parameters i need to add in my OTEL config.yaml running on EC2 to spit out more detailed logs information to debug send failures. currently i have not set any logging info.

1) Currently i have in the exporters: exporters: awsxray: awsemf:

Shall i add logging, so it shall be like the below: exporters: awsxray: awsemf: logging: loglevel: debug

1) Is the above change correct?

2) Do i need to also add logging exporter as shown below ? service: pipelines: traces: receivers: [otlp,awsxray] processors: [batch/traces] exporters: [awsxray, logging] metrics: receivers: [otlp] processors: [batch/metrics] exporters: [awsemf, logging]

3) After my debugging is done, shall i remove this logging exporter from both the places? What is the penalty if i leave it?

[georges-git]

Hello , Can anyone respond to my above query?

[mxiamxia]

Hi George, to your questions,

1. Yes
2. Yes
3. You should remove it for your production env, it prints lots of debugging logs could affect your service performance

[sethAmazon]

From the docs enable debugging log add a key value pair into /opt/aws/aws-otel-collector/etc/extracfg.txt and restart collector

echo "loggingLevel=DEBUG" | sudo tee -a /opt/aws/aws-otel-collector/etc/extracfg.txt sudo /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -a stop sudo /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -a start

[georges-git]

Attaching the logs with DEBUG level - aws-otel-collector.zip

[javierm2k9]

Hello

In my case, I want to debug the Otel collector using the implementation of Otel inside of the Cloudwacth agent running in one ec2 windows server 2016 instance.

I'm following these steps

& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -o default -s  

Add-Content C:\ProgramData\Amazon\AmazonCloudWatchAgent\CWAgentOtelCollector\Configs\extracfg.txt "loggingLevel=DEBUG"

& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a stop

& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a start

but can't see any log in this folder:

C:\ProgramData\Amazon\AmazonCloudWatchAgent\CWAgentOtelCollector\Logs

The status of otel collector is ok

& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a status

"cwoc_status": "running", "cwoc_starttime": "2021-09-08T15:21:04", "cwoc_configstatus": "configured", "version": "1.247349.0b251399"

Any idea how to achieve this?

Thanks

[mxiamxia]

Did you see any log file in C:\\ProgramData\\Amazon\\AWSOTelCollector\\Logs\\ directory? Just in case that CW/OTel Composite Agent didn't override the log file path successfully.

[javierm2k9]

Hi @mxiamxia

I didn't see any logs and in fact, that path doesn't exist in my ec2 instance. Maybe that path is related to OTEL agent when you are not using Cloudwatch agent for open telemetry collector.

I'm using this approach https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-open-telemetry.html

Thanks

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue was closed because it has been marked as stale for 30 days with no activity.