Hangfire scheduling does not get instrumentated

[lucasoares]

Issue with OpenTelemetry.Instrumentation.Hangfire

List of all OpenTelemetry NuGet packages and version that you are using (e.g. OpenTelemetry 1.3.2):

    <PackageReference Include="OpenTelemetry.Instrumentation.Hangfire" Version="1.6.0-beta.1" />
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.4.0-beta.3" />
    <PackageReference Include="OpenTelemetry.Extensions.Propagators" Version="1.4.0-beta.3" />

    <PackageReference Include="Hangfire.Core" Version="1.8.7" />
    <PackageReference Include="HangFire.Redis.StackExchange" Version="1.9.3" />

    <!-- Only for Tests -->
    <PackageReference Include="Hangfire.MemoryStorage" Version="1.8.0" />

Runtime version (e.g. net462, net48, net6.0, net7.0 etc. You can find this information from the *.csproj file):

net6.0

Is this a feature request or a bug?

• [x] Feature Request
• [ ] Bug

What is the expected behavior?

Hangfire should create a span when scheduling the job.

What is the actual behavior?

Hangfire will not create any activity when scheduling a message. Scheduled jobs doesn't get traced, only when executed.

Additional Context

I think the current implementation of hangfire filter will not be able to notify when the job is scheduled.

[vishweshbankwar]

@fred2u Could you take a look at this?

[SitamMatt]

Hi, I encountered similar problem. In my case Reccuring Jobs didn't send any trace data. I've noticed that Reccuring Jobs are first handled by HangfireInstrumentationJobFilterAttribute.OnCreating method which does not occure when we just enqueue Job using BackgroundJobClient. Implementing your own JobFilter with the same implementation as HangfireInstrumentationJobFilterAttribute aside from OnCreating and OnCreated methods do the work. Note that this might not be a perfect workaround.

[jhartmann123]

It looks like the issue is partially solved in the meantime. Here's a repro with current versions of hangfire and otel:

OpenTelemetry.Hangfire.zip

The only thing that does not trace the job is when you trigger a recurring job manually on the Hangfire-UI.

What does work (tested in the repro) is

• Enqueuing a job
• Scheduling a job
• Job continuations
• Triggering recurring jobs via RecurringJobManager
• Triggering recurring jobs via Cron

When triggering the jobs via the UI, the methods called in the hangfire job filter are the same as for "automatic" jobs though,

// in that order
OnCreating
OnCreated
OnPerforming
OnPerformed

so it's not something blatantly obvious.

[jhartmann123]

It does get blatantly obvious as soon as you attach a debugger though. The problem starts in OnCreating about here: https://github.com/open-telemetry/opentelemetry-dotnet-contrib/blob/3a8cbafa44878cf24478142f314c0e715d5fc6ee/src/OpenTelemetry.Instrumentation.Hangfire/Implementation/HangfireInstrumentationJobFilterAttribute.cs#L103-L107

If you trigger the job from the UI, there is an activity from AspNetCore, which does not get traced, as AspNetCore-Instrumentation is not enabled. The Hangfire trace uses the parent context from AspNetCore, thus not being recorded.

As soon as you add AddAspNetCoreInstrumentation() to the otel config, all jobs get traced as expected.

However, there's a pitfall (for us at least): As we're not interested in Hangfire Dashboard traces, we've filtered them out, similar like this

services.AddOpenTelemetry()
        .AddAspNetCoreInstrumentation(o => o.Filter = (ctx) => !ctx.Request.Path.StartsWithSegments("/hangfire"))

Now at this point when testing tracing via the UI a few months ago, I thought it's a tracing issue of the Hangfire-Instrumentation, but the trace was just filtered out. When expanding the filter a bit, the manually triggered Hangfire-Jobs get traced again:

o.Filter = (ctx) => !ctx.Request.Path.StartsWithSegments("/hangfire") || ctx.Request.Method == "POST" && ctx.Request.Path == "/hangfire/recurring/trigger"

So long story short: It looks like this issue is solved in the meantime.