search

SumoLogic / sumologic-otel-collector

Sumo Logic Distribution for OpenTelemetry Collector
Apache License 2.0
41 stars 37 forks source link

OTEL enrichment agent do not set labels or host metadata during scaling event #1275

Open rupeshnemade opened 12 months ago

rupeshnemade commented 12 months ago

We observed that whenever OTEL pod scales up we start seeing few logs with incomplete metadata (labels, host). Expected

image

Actual As we can see there are no labels & host present in Kuberenetes block

image

During scaling phase as OTEL pod doesn't have metadata from k8s API in its cache it sends logs without setting few metadata fields. This behaviour is valid for short-lived pods but not appropriate for running pods.

Is this Kubernetes problem or this can be fixed in OTEL?

swiatekm commented 11 months ago

Since this only happens for new otel Pods, I have a suspicion that it's because the k8sprocessor processed data without first having populated its Pod cache completely. As I suggested on Slack, can you try adding a delay to the otel container (for example via initialDelaySeconds in the readiness probe) and see if that helps?

If this is the case, we can consider explicitly waiting for caches to sync while starting the processor.

rupeshnemade commented 11 months ago

I tried updating initialDelaySeconds in OTEL pods forwarding logs to Sumo from default 5sec to 20 sec but it didn't help to alleviate the issue.