search

taskforcesh / bullmq-pro-support

Support repository for BullMQ Pro edition.
1 stars 0 forks source link

Tracing support for bullmq-pro #75

Open sakari opened 1 month ago

sakari commented 1 month ago

Hi

We use bullmq-pro and elastic-apm for tracing. To get this working we have implemented a separate layer for continuing traces from scheduling a job to worker running the job. This however requires us to do this boilerplate for each project we work on. Would there be any possibility of having an autoinstrumenter for this? Or if you know of a project that does that already that would be even better.

Like I imagine that we could store the current tracing context to the job object when calling .add and then set the parent from that data inside the Worker when starting to work on the job. This would require monkeypatching the bullmq-pro classes so some support from taskforcesh to at least avoid breaking that functionality would be nice -- even better would be to have this be part of the bullmq-pro project.

Our manual setup uses the job data to carry this information and has wrapper around the worker function to set the context. However this requires us to have one more thing to remember and maintain.

manast commented 1 month ago

@sakari we are planning to add pluggable support for telemetry frameworks such as open-telemetry. It would be quite valuable for us if you could show us an example of how you are currently wrapping the data so that we can design the most flexible solution that covers most cases used by BullMQ users.

sakari commented 4 weeks ago

we have a layer on top of bullmq that allows us to have middlewares. One such middleware changes the job data to contain the tracing information using elastic apm currentTraceParent. Another middleware envelopes the worker function and sets the tracing context using startTransaction.

However elastic-apm autoinstrumenting would be much simpler setup and would also cover cases where we do not for some reason use the layer on top of bullmq. If we could have a way to add "header" information to the job that does not mess with the job data I think monkeypatching .add and Worker to do the necessary would be enough and would limit the amount of setup we have to do to minimum.

manast commented 4 weeks ago

With autoinstrumenting you mean using this? https://github.com/elastic/apm-agent-nodejs From what I see here: https://www.elastic.co/guide/en/apm/agent/nodejs/current/supported-technologies.html it seems that if BullMQ Pro supported OTel it should work automatically.

sakari commented 3 weeks ago

yes. yes it seems so if I read the doc correctly.