Closed Souheil-Yazji closed 1 year ago
Leaving a comment here with info I have synthesized from talking with Pat and Souheil. (if the two of you want to have comments leaving what you have learned go ahead but this will be my property to edit.
Caveats
OpenTelemetry is not an observability back-end like Jaeger or Prometheus. Instead, it supports exporting data to a variety of open source and commercial back-ends. which we can combine with APM. Using the opentelemetry operator we can set the mode
to be in sidecar to inject into our pods.
From above, OTel seems like what would generate the data, so in our current setup it would be like the istio sidecar generating and sending traces. Once applications have the OTel library in them (idk how feasible that is)
In the video "if you have a jaeger instrumentation library... you can just point it to the collector, and the collector would take advantage of that and send it to your destination"
In the demo (also in the video screenshotted above) they present OpenTel with Jaeger + Elastic
I don't think this solves the existing problem of needing a "span storage" area though...
Maybe I shouldn't be surprised but the storage component for zipkin is also elasticsearch (tested against es 6-7.x no action on this issue for ES 8.3.0) and cassandra
Rationale: Jaeger was proving difficult to work with in our environment, and the workarounds needed introduced more components than thought necessary. We are moving in favour of using Elastic APM which seems to provide similar functionality to Jager Because APM does not support using istio traces to get spans, we need span data somehow by either injecting APM Agents or using OpenTelemetry agents With talks of looking at OpenSearch as a path forward we decided on leveraging OpenTelemetry instead of digging too deep into Elastic which does support Otel. Do note however that envoy is looking to send Otel compatible data so possibly in the future we might be able to do away with whatever agent we add at current time of implementation.
Following bullets are to be split into tasks
Closing as completed
After correctly configuring the ES service thanks to Will, we ran into an issue where the
Jaeger Collector
, a component created by the operator, was running into the following error:"msg":"Failed to create span writer","error":"elastic: Error 400 (Bad Request): unknown key [template] in the template
After doing some research, it seems that the latest release of Jaeger (1.41) does not yet support ES v8.x+. There is a long-run issue open in the Jaeger repository for this. Also supporting this discovery https://www.jaegertracing.io/docs/1.41/deployment/#elasticsearch, does not include 8.
For reference, our ElasticSearch instance is currently at v8.5.0
According to ES, it should be possible to integrate https://www.elastic.co/guide/en/apm/guide/current/jaeger-integration.html