Che Tracing

[yarivlifchuk]

Summary

We propose a tracing mechanism that does not require changes to existing Che code. However, we do recommend standardizing the format in which trace are written. In addition, we propose an option to enable providing additional parameters to trace entries in a standard way, to improve supportability. To support this, industry-accepted open source components must be deployed to the K8S cluster with special focus on security aspect.

Description

Che epics [Complementary]: Logging - https://github.com/eclipse/che/issues/10290 Monitoring - #10329 Tracing - https://github.com/eclipse/che/issues/10288 Complementary epic to the opentracing & Jaeger

Che epics [to be reevaluated]: https://github.com/redhat-developer/rh-che/issues/718 It gives a view on the K8S Tracing best practice and the implication on Che agents and should be complementary epic.

Background

In a distributed system, a trace encapsulates the transaction’s state as it propagates through the system. Tracing helps gather timing data needed to troubleshoot latency problems in microservice architectures, and tells the story of a transaction or workflow as it propagates through a distributed system. Every transaction might reflect performance anomalies in an early phase when new services are being introduced by independent teams.

Tracers live in the applications. It records timing and metadata about operations that took place. They often instrument libraries, so that their use is transparent to users. For example, an instrumented web server records when it received a request and when it sent a response. The trace data collected is called a Span. Main benefits of distributed tracing:

• Distributed context propagation: illuminates the request’s path until its final destination.
• Out of the box infrastructure overview: how the interactions between services are done and their dependencies.
• Efficient and fast detection of latency and bottleneck issues.

https://sematext.com/blog/opentracing-distributed-tracing-emerging-industry-standard http://opentracing.io/documentation/pages/instrumentation

Standard API and Common Tracers

OpenTracing API - offer a consistent, unified and tracer-agnostic instrumentation API for a wide range of frameworks, platforms and programming languages. It abstracts away the differences among numerous tracer implementations, so shifting from an existing one to a new tracer system would only require configuration changes specific to that new tracer.

Common distributed tracers are Zipkin and Jaeger inspired on Google’s Dapper large-scale distributed tracing platform.

1. Zipkin Zipkin is an open-source distributed tracing solution implemented in Java and with OpenTracing compatible API. It manages both the collection and lookup of this data. Zipkin component is installed per cluster. It collects and display tracing data. Each node is installed with Linkerd as a DeamonSet that export tracing data. https://sematext.com/blog/opentracing-zipkin-as-distributed-tracer

2. Jaeger Inspired by Dapper and OpenZipkin, is a distributed tracing system released as open source by Uber Technologies. Jaeger extends more complex architecture for larger scale of requests and performance. Jaeger has better language coverage of OpenTracing-compatible clients, low memory footprint, and a modern, scalable design compared to Zipkin. Jaeger Collector and Query is installed per cluster and require a backing storage. The Jaeger Agent is deployed as a DeamonSet and receive traces via UDP. All pods running on a given node will send data to the same agent. If that's not suitable for workload, an alternative is to deploy the agent as a sidecar. https://github.com/jaegertracing/jaeger

Implementation recommendation

1. Write trace based on the relevant library by the Che agents and any relevant Applications.
2. Add custom environment params to the trace’s records, e.g. user’s tenant id.

Implementation

• [x] Implementation plan for Opentracing support for k8s / OpenShift infrastructures in Che https://github.com/eclipse/che/issues/11720
• [x] Workspace related operations
  • [x] workspace creation
  • [x] workspace start
  • [x] trace workspace deletion
  • [x] trace volume claim attachment
  • [x] trace creation of che plugins -> moved to https://github.com/eclipse/che/issues/12003
  • [x] define meaningful set of tags to apply to the traces. (Postponed)
  • [x] review https://github.com/eclipse/che/pull/11920 maybe we need some traces here
• [ ] Investigate tracing of wsmaster tomcat server shutdown https://github.com/eclipse/che/issues/11946
• [ ] Investigate tracing of DB migration https://github.com/eclipse/che/issues/11945
• [ ] Tracing operation made over JSON-RPC https://github.com/eclipse/che/issues/11923
• [ ] [Tracing] Provide more context information about workspace related operation https://github.com/eclipse/che/issues/12533
• [ ] Trace ChePlugins and how they are applied to workspaces https://github.com/eclipse/che/issues/12003
• [ ] Trace activity that is happened in workspace??? Do we need this????
• [ ] Connect jaeger authentication to keycloak https://github.com/eclipse/che/issues/12544
• [ ] Document tracing usecase https://github.com/eclipse/che/issues/12545
• [ ] Che monitoring/tracing in istio environment https://github.com/eclipse/che/issues/12546
• [ ] https://github.com/eclipse/che/issues/12739 [Traces] Machine create/start trace has to be more detailed
• [ ] https://github.com/eclipse/che/issues/12740 [Traces] Study if there is a way to track container start phases

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.