Jobs mixed-up when running parallel integration tests with ASP.NET Core in a single process

[corentinaltepe]

Hello,

We have an ASP.NET Core app which runs hangfire client and server. We have an integration tests suite using the WebApplicationFactory pattern and it works great.

When we run this test suite in six parallel processes with NCrunch to have it run faster, it works properly.

However, I've tried to run tests in parallel in a single process using xUnit and NUnit and both give errors. Let's say we have two instances of the app created with new WebApplicationFactory<Startup>().CreateClient() running concurrently on a same process, then we have two Hangfire servers on the same process. It looks like these two servers workers pick each other's jobs, messing with the test results, as the app is not designed to allow more than one worker per app.

It looks like Hangfire server has some static state, which is shared accross the different instances of the apps under tests. Is this behavior expected or am I doing something wrong here? I have looked on the internet for similar issues but couldn't find answers.

The things I have tried so far:

• isolate each app instance job creation and Hangfire server on a unique queue. The queue name is a Guid.NewGuid().ToString("N") generated at the application startup. Even in this configuration, Hangfire servers pick the jobs from the other app instances.
• define a separate Hangfire server name per app instance (although this is not required since v1.5), using again a unique Guid. No effect, we have the same problem
• mock IBackgroundJobClient in tests so that it doesn't schedule a job to hangfire but instead invokes the job's method immediately. This solves the problems, showing that something might be wrong with my Hangfire server configuration or shared static states. I'm not entirely happy with this solution as I would prefer having Hangfire client and server part of the system under tests in the test suite.
• in production, we currently use in-memory storage (I know, right? We'll change that soon), so I tried to use SQL Server storage in tests, with a separate database for each instance of the app under test. This did not solve the issue.

I've also tried several combinations of these but only mocking IBackgroundJobClient (bypassing Hangfire entirely) has brought results.

Would you have any input or recommendation? Thank you in advance.

[corentinaltepe]

Hi,

I've hit this issue again. I tried injecting a different instance of JobStorage in each application instance under test (test scope). But I still get the jobs mixed up. I believe this is cause by this specific line, which assigns the JobStorage into a static variable, which then crosses boundaries set by the test scopes.

public static IGlobalConfiguration<TStorage> UseStorage<TStorage>(
  [NotNull] this IGlobalConfiguration configuration,
  [NotNull] TStorage storage)
  where TStorage : JobStorage
  {
    if (configuration == null) throw new ArgumentNullException(nameof(configuration));
    if (storage == null) throw new ArgumentNullException(nameof(storage));

    return configuration.Use(storage, x => JobStorage.Current = x);
  }

[jufa2401]

@corentinaltepe

Did you manage to find a work-around? Our team is suspicious we have run into the same issue 😁

[domn1995]

Running into this issue as well. Would love to know if anyone has a workaround or if we have a suggested fix.

[corentinaltepe]

hi @jufa2401, hi @domn1995. Unfortunately I haven't been able to fix the issue nor work around it.

I still believe this line is reponsible for the static singleton pattern affecting parallel execution of several Hangfire Servers within the same process.

@odinserj, would you accept a PR to change the behavior within the Hangfire repository directly? The idea would be to replace the static singleton pattern with a non-static IoC container singleton pattern when possible. We'd have to be super careful about side-effects, as it would be a sensitive part of the code, and I'm not familiar with it.

[domn1995]

@corentinaltepe After deep diving the code I agree with you. @odinserj I would be happy to contribute a PR improving this design, because our code using hangfire is essentially untestable at the moment :(