Context Propagation broken for Pulsar Functions

[damienburke]

Describe the bug Pulsar functions can consume messages and produce messages, where a receive and publisher Span will be created respectively. Re context propagation, the receive Span is created with the trace context of the incoming message (so works as expected) - but the trace context of the outgoing message (i.e. the publisher Span) has a different trace ID set.

Steps to reproduce I created a specific github repo for this - https://github.com/damienburke/pulsar_otel_function. But in short, I have a docker compose env that creates a pulsar broker and deploys an instrumented pulsar function. I have a script that then publishes a message to this function, where upon I can see in logs that the trace IDs differ.

What did you expect to see? The publisher Span created by the Pulsar Function should have the same Trace ID as the receive Span.

What did you see instead? The publisher Span created by the Pulsar Function had a different Trace ID from the receive Span.

What version are you using? 1.28 of the Java Agent 2.11.1 of the Pulsar Broker

Environment Docker Compose

Additional context I put some effort into making this easily re-createable via my github repo, so if any issues getting that working please let me know.

Thanks

[trask]

cc @tjiuming

[dao-jun]

@damienburke This is an unusual usage, the pulsar plugin is designed for pulsar-java-client, so I just enhanced client library. It looks like you want to mount the otel-agent on pulsar-function worker, right? It should be a feature request.

[dao-jun]

@damienburke I've created a branch dev/pulsar_func and enhanced PulsarFunction, you can take a try. If it works fine, please let me know.

[damienburke]

Thanks @dao-jun I just want to unpack and clarify some of what you mentioned.

This is an unusual usage, the pulsar plugin is designed for pulsar-java-client So not sure what you mean by unusual usage. Do you mean instrumenting a pulsar function in general is unusual? Because this instrumentation of the pulsar function is instrumenting the pulsar-java-client. Further, its this instrumentation that is creating the Spans I described, but for some reason the trace context is not being picked up by the (pulsar function) producer.

Or to take a step back, could you recommend a better approach such that OTel trace can be propagated through pulsar functions?

so I just enhanced client library. So as pr my description, the pulsar function does create Spans for the consumer and producer, but does not propagate the context between them. Your update looks like it will create a process Span, and will also pass along the (consumer) context. Is that correct? Or if not, please describe what the branch is doing - so I can learn.

It looks like you want to mount the otel-agent on pulsar-function worker, right? It should be a feature request. Interesting. I thought it would be natural for users to just to this themselves.

_I've created a branch dev/pulsar_func and enhanced PulsarFunction, you can take a try. If it works fine, please let me know._ Awesome, thanks. Would u know, do these remote branches create a build and/or how can I test? I did a quick search around project and could not find much info..

[dao-jun]

@damienburke Pulsar function is a sub-process created by pulsar broker, so I did't handle this case before. Because it seems a little strange, in general, java-agent only mounted on user-applications but not on a middleware.

But I realized that this usage may also make sense through the issue.

Awesome, thanks. Would u know, do these remote branches create a build and/or how can I test? I did a quick search around project and could not find much info..

You can clone the branch to you local, and build it. Then you can get the otel-java-agent package, mount it on pulsar function and test.