feat: implement opentelemetry for distributed tracing

namkyu1999 commented 2 months ago

What this PR does / why we need it:

This pr implements opentelemetry codes according to this proposal.

Parent context and OTEL endpoint are passed through env(TRACE_PARENT, OTEL_EXPORTER_OTLP_ENDPOINT)

Result is like below

relevant PRs

Checklist:

[ ] Fixes #
[ ] PR messages has document related information
[ ] Labelled this PR & related issue with breaking-changes tag
[ ] PR messages has breaking changes related information
[ ] Labelled this PR & related issue with requires-upgrade tag
[ ] PR messages has upgrade related information
[ ] Commit has unit tests
[ ] Commit has integration tests
[ ] E2E run Required for the changes

xavierW commented 2 months ago

邮件已收到，我会尽快阅读并回复，谢谢

namkyu1999 commented 2 weeks ago

@smitthakkar96

This is a great start however I don't see any place where we are marking span as error in case of failure, do we plan to do this in future iterations?

I will record errors like below

if err != nil {
    span.SetStatus(codes.Error, "operationThatCouldFail failed")
    span.RecordError(err)
}

But we have over 50 chaos faults, so it can be merged in the next PR (it is a time-consuming job)

Based on the experiment context it might be good to explore what span attributes might be relevant.

Does this mean adding a span attribute to help with future queries by attribute?

Lastly to be able to correlate logs with traces can we add traceID to logs?

sure, I will add it

smitthakkar96 commented 2 weeks ago

Does this mean adding a span attribute to help with future queries by attribute?

Span Attributes can be useful for queries but also they can serve as additional context when reviewing the spans. Let's take an example, you get an http request, you might wanna log the relevant headers, the request body and the path similarly in context of the experiments it might be helpful to log the parameters for the experiment, maybe the targets and any other data you think that might be helpful when looking at an error trace :)

The demo application might be a good inspiration, another source of inspiration is semantic attributes.

Lastly for future I would suggest we increase the coverage of instrumentation (can be a good task for LFX contributors) by creating spans for calls to external services such as AWS, GCP, K8s or any HTTP or DB calls this contrib package has some out of the box shims that can make it easy

But we have over 50 chaos faults, so it can be merged in the next PR (it is a time-consuming job)

Totally understandable, instrumentation quality and coverage can always be improved overtime. Your initiative paves the foundation we can build on top of, my comment was just suggestions/ideas if you have sometime maybe raise log them as Github issues so other contributors can pick them up

namkyu1999 commented 2 weeks ago

Thanks for your detailed comments @smitthakkar96 :) I am going to pave the foundation for collaborating with other contributors. Adopting a contrib package is also good idea

litmuschaos / litmus-go

feat: implement opentelemetry for distributed tracing #706