emissary-ingress / emissary

open source Kubernetes-native API gateway for microservices built on the Envoy Proxy
https://www.getambassador.io
Apache License 2.0
4.32k stars 685 forks source link

Opentelemetry driver missing spans #5573

Open himanshu2998 opened 4 months ago

himanshu2998 commented 4 months ago

Hi, I am using tracing service in emissary ingress https://www.getambassador.io/docs/emissary/latest/topics/running/services/tracing-service below is the config

{{- if .Values.tracing.enabled }}
apiVersion: getambassador.io/v3alpha1
kind:  TracingService
metadata:
  name:  tracing
  namespace: {{ include "ambassador.namespace" . }}
spec:
  service: {{ .Values.tracing.endpoint }}
  {{- if .Values.env }}
  ambassador_id: [{{ .Values.env.AMBASSADOR_ID | quote }}]
  {{- end }}
  driver: opentelemetry
  sampling:
    overall: {{ .Values.tracing.overall_sampling }}
{{- end }}

Although I am getting traces but for some traces some spans are missing due to which I am getting missing root spans. This can be due to high load, is there a way to configure timeouts and batching config for tracing data in the service for opentelemetry driver?

Screenshot 2024-02-15 at 4 39 54 PM
cindymullins-dw commented 4 months ago

Hi @himanshu2998 , which version are you on? In 3.4 and 3.5 we had a bug fix and a Community Advocate contribution which both impacted telemetry spans so that'd be one thing to check.

himanshu2998 commented 4 months ago

Can we also set propagation mode for opentelemetry driver?

himanshu2998 commented 4 months ago

Hi @himanshu2998 , which version are you on? In 3.4 and 3.5 we had a bug fix and a Community Advocate contribution which both impacted telemetry spans so that'd be one thing to check.

Hi @cindymullins-dw we are using 3.9 version for ingress

himanshu2998 commented 4 months ago

any updates here?