search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.25k stars 1.93k forks source link

OOM error caused by AzureMonitorLogRecordExporter #40546

Open rafalp88 opened 4 weeks ago

rafalp88 commented 4 weeks ago

Below line cause OOM error. When DEBUG level is on, there will be some kind of recursive invocation. LogRecords logging causes more log entries which causes sending more and more LogRecords.

https://github.com/Azure/azure-sdk-for-java/blob/174041384dd09ac1bb23a07655f4a966af7c777a/sdk/monitor/azure-monitor-opentelemetry-exporter/src/main/java/com/azure/monitor/opentelemetry/exporter/AzureMonitorLogRecordExporter.java#L57

github-actions[bot] commented 4 weeks ago

Thank you for your feedback. Tagging and routing to the team member best able to assist.

lmolkova commented 3 weeks ago

@rafalp88

Thank you for reporting this! As a mitigation step, could you please disable verbose logs from com.azure.monitor.opentelemetry.exporter? If you haven't done it already.

@heyams @trask could you please take a look?

it seems exporter logs with ClientLogger from core which, when such debug logs are enabled, produces enormous amount of logs.

We might need to figure out how to disable all logs from the exporter http pipeline completely or prevent the exporter from consuming its own logs in some other way. LMK if you need any help with ClientLogger.