Jose-Matsuda
commented
1 year ago Leaving a comment here with info I have synthesized from talking with Pat and Souheil. (if the two of you want to have comments leaving what you have learned go ahead but this will be my property to edit.
Attempting to continue to leverage Jaeger
- We tried to use cassandra's k8ssandra to use as the span-storage-backend but found that when trying to do this on dev we ran into various errors (one was permission) that we were unsure how it happened as Souheil was able to start it up in minikube in 10 minutes. The helm install via terraform had exceeded 40 minutes in trying to get something going but nothing.
- As said above, leveraging APM is a way forward; albeit a bit weird I feel as APM has it's own offerings that are similar to Jaeger. Note that one of the devs of APM has stated that the only benefit of using Jaeger in this case is if you're already using Jaeger -- that means you don't have to change your instrumentation when talking to a user who wanted to integrate an existing APM setup with a possible new Jaeger setup
Other options to Jaeger
Leveraging APM
- We could use APM which seems to have similar features to Jaeger, and really depends on what we need out of this (do we just want those fancy span graphs?)
Caveats
- Currently it does not seem like APM supports getting traces directly from istio. If we wanted to use those istio traces we would need the jaeger-agent in some form to consume said traces.
- If we don't want jaeger at all, we would need to inject an agent into the pods to allow APM to get data (since cant get it from istio).
Leveraging OTel and APM
OpenTelemetry is not an observability back-end like Jaeger or Prometheus. Instead, it supports exporting data to a variety of open source and commercial back-ends. which we can combine with APM. Using the opentelemetry operator we can set the mode to be in sidecar to inject into our pods.
Leveraging OTel and Jaeger
-
From above, OTel seems like what would generate the data, so in our current setup it would be like the istio sidecar generating and sending traces. Once applications have the OTel library in them (idk how feasible that is)
-
In the video "if you have a jaeger instrumentation library... you can just point it to the collector, and the collector would take advantage of that and send it to your destination"
-
In the demo (also in the video screenshotted above) they present OpenTel with Jaeger + Elastic
-
I don't think this solves the existing problem of needing a "span storage" area though...
Zipkin
Maybe I shouldn't be surprised but the storage component for zipkin is also elasticsearch (tested against es 6-7.x no action on this issue for ES 8.3.0) and cassandra
After correctly configuring the ES service thanks to Will, we ran into an issue where the
Jaeger Collector, a component created by the operator, was running into the following error:"msg":"Failed to create span writer","error":"elastic: Error 400 (Bad Request): unknown key [template] in the templateAfter doing some research, it seems that the latest release of Jaeger (1.41) does not yet support ES v8.x+. There is a long-run issue open in the Jaeger repository for this. Also supporting this discovery https://www.jaegertracing.io/docs/1.41/deployment/#elasticsearch, does not include 8.
For reference, our ElasticSearch instance is currently at v8.5.0
According to ES, it should be possible to integrate https://www.elastic.co/guide/en/apm/guide/current/jaeger-integration.html