Closed Voxelot closed 1 year ago
Let me know if you need any help here, I've had copious amounts of experience with Prometheus operating on a large scale.
https://istio.io/latest/docs/tasks/observability/distributed-tracing/jaeger/
or
https://observability.thomasriley.co.uk/prometheus/configuring-prometheus/using-service-monitors/
I have looked into tracing with Linkerd recently which is becoming popular as its more lightweight and easier to adapt than Istio - https://linkerd.io/2.10/tasks/distributed-tracing/ - but the issue is Linkerd doesn't have in built Ingress like Istio which is needed for routing on backend and load balancing (ingress gateway)
would love to learn about @digorithm perspectives on this
I did tons of observability work at previous companies, and by far the best experience I had was with Prometheus + Grafana. So that's the one I've been personally using and advocating ever since.
We can get distributed tracing with Grafana: https://grafana.com/docs/grafana/latest/explore/trace-integration/ (supports Jaeger, tempo, and more), plus we get all the excellent Prometheus SDKs to add monitoring in our applications, Prometheus' node exporter for system metrics (no manual logging needed), and Grafana offers a lot of out-of-the-box dashboards for all of that.
Then we can opt to use Grafana cloud or self-host it, both are relatively simple to do.
From @Voxelot :
" For telemetry, we may want to look at instrumenting something like this in addition to application level traces: https://medium.com/tezedge/halving-the-tezedge-nodes-memory-usage-with-an-ebpf-based-memory-profiler-2bfd32f94f69"
I've been working on this issue this week.
I'm going to break this ticket into smaller tickets as there have been multiple steps to realizing our goals. So far this is how it shakes down (and some of these tasks are already 'done'):
@uberscott for future reference if there are lots of subtasks that don't have a resultant work artifact, I suggest using checklists within a single issue.
https://docs.github.com/en/issues/tracking-your-work-with-issues/quickstart#adding-a-task-list
@Voxelot You got it. (next time)
Added Grafana and now Grafana is also scraping prometheus (and I can see the counter data yippie!)
bridging tracing with opentelemetry is the holy grail (so devs don't need to recode for opentelemetry we can just reuse our existing tracing intrumentation.). The tracing-opentelemetry crate that is supposed to make this bridging easy does not seem to work with the latest version of opentelemetry....
I am still trying to figure out what needs to be done because I may need to fix the tracing-opentelemetry crate.
Feel free to open issues with opentelemetry and submit fixes if you have the context. However, if opentelemetry is too immature we may just need to roll with our own tracing setup.
https://node-beta-1.fuel.network/metrics - @uberscott
We use tracing as per our coding standard. However, we need to be able to monitor metrics using systems like prometheus and jaeger.
Investigate how we'd integrate our tracing infrastructure into telemetry using tools like tracing-opentelemtry and setup minimum viable telemetry for fuel-core.
Goals:
Related Tasks: