Logs are missing during high loads

signalfx / splunk-otel-collector-chart

Splunk OpenTelemetry Collector for Kubernetes

Apache License 2.0

116 stars 146 forks source link

Logs are missing during high loads #945

Closed beth2016 closed 9 months ago

beth2016 commented 11 months ago

What happened?

Description

When we run performance tests (40k requests/min) , we see logs are intermittently dropped for 30-90 seconds. This issue started manifesting after we upgrade otel from 0.43 to 0.81.

Steps to Reproduce

Expected Result

Actual Result

Chart version

0.81

Environment information

Environment

Cloud: Openshift k8s version: 1.23

Chart configuration

No response

Log output

No response

Additional context

No response

matthewmodestino commented 11 months ago

Hey Beth! Hope all is well!

This could be caused by file rotation under high load on the node, or other scenarios.

Any logs from the agents affected that may help?

atoulme commented 10 months ago

You might be best served by opening a support case to follow up. Please post the support case ID in this issue so we can work with you directly.