MatthewVaadin
commented
1 year ago The more easy part of this seems to be propagating the trace context. I have experimented with adding a meta tag to the head section of each page (using an IndexHtmlRequestListener). I have verified that the document load spans created by the frontend instrumentation pick this up and use the trace ID.
The difficult part is how to make sure that the trace ID is then correctly forwarded to the observability tool. I have tried to use the OpenTelemetry SDK to create SpanData instances, which are then passed to the exporter. This is made possible by a customization API provided by the OTel SDK. The problem here is that because this is all happening within the agent, some of the SDK classes are being "shaded" and cause some method calls to fail.
I am looking into implementing a custom tracer that will be able to create its own spans without using the SpanData class.
In the solution for client-side instrumentation (#150), traces and spans that are created by the frontend instrumentation are sent to the server in a JSON format. The agent then picks up the JSON and uses the data to create new spans, which then in turn are sent to the configured observability tool.
The first problem with this is that the request to send the JSON to the server is also instrumented by the agent. This means that the frontend spans created by the agent are nested under the wrong parent. We need to both filter out any Observability Kit requests and create the spans under the correct parent. This might be solved by propagating the trace content from the server to the frontend.
The second problem is that the trace ID, span ID and parent ID (if there is one) for the original spans are lost when the agent creates new spans. We need to find a way to transfer these to the new span or find a way to forward the original spans rather than recreating them.