Shared memory between created TestServers of individual WebApplicationFactories

[ascendise]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

When I start up my web app through a WebApplicationFactoy, it seems to share static fields inside the app with other apps that were created with another instance of WebApplicationFactory.

So for example if I set a static field during startup to 1 and then startup another app through a different factory and set a breakpoint before this field is set, this static field is already set to 1...

This seems unusual and there is no info about this in the docs. Is this intended behaviour?

Expected Behavior

When creating a web app, it should not remember in-memory state from the last run.

Steps To Reproduce

https://github.com/ascendise/webappfactory-share-memory-repro/tree/main

The relevant sections are Program.cs and the unit test.

Inside the Program.cs there is a static field initialized with 0 that is incremented during startup. The result is returned through an endpoint. The test asserts that this value is always 1 (as expected). But during the second run, the value is 2...

Exceptions (if any)

No response

.NET Version

8.0.101

Anything else?

dotnet_info.txt

[martincostello]

WebApplicationFactory instances all exist in the same process, so if there are any static fields they will retain their values during the test runs, just like any other static fields in any other type.

Your test runner would have to create and tear down an entire child process for each try of the test method to get the behaviour you seem to want it to give you.

[ascendise]

I think I understand now. But then, how are you supposed to write integration tests this way? The example tests from the docs do it the way I did (more or less), so they would have this issue too as soon as they have to deal with static fields, right?

It is not an unusual use case either, as I encountered this bug because I am using MongoDB and was registering ClassMaps during startup and those ClassMaps where saved in-memory inside a static dictionary field. And the tests failed during program startup because I was adding duplicates, because the ClassMaps were already added from a previous run.

My point is, that if the only way to run those tests, without risking sharing state, is to run every single test as an individual process, then this should be documented in the docs for ASP NET Core Integration Testing.

Unless you are telling me, that I somehow misconfigured my tests? Like I also tried explicitly running the tests in sequence (NUnit's [NonParallelizable]), having each test in it's own class, etc. just to make sure that I am not somehow sharing test state myself.

[martincostello]

This is a case where you need to design your code to not use static state, or factor it in a way that it doesn't create these issues of state management. In essence, you code as written isn't written to be testable in a way that isolates the tests from each other.

This isn't specific to TestServer or WebApplicationFactory - it's inherent to the way you've written your Program class. You would see the same behaviour without using them and running the code with any of the three main test frameworks for .NET.

The documentation wouldn't suggest to run each test in a dedicated process because that would be incredibly inefficient in terms of performance in a codebase of any meaningful size.

I wrote a sample application for integration testing with Minimal APIs and WebApplicationFactory a few years ago. Maybe take a look through it and get a feel for the way the application is written and uses the tests to avoid such state management problems: https://github.com/martincostello/dotnet-minimal-api-integration-testing

[ascendise]

So maybe I am just trying to do the integration testing differently than Microsoft, so just that we are clear what my goal is. I try to test my app by testing it as a whole. Interacting with it through HTTP, everything being persisted in a real "test db" that was setup, interacting with real systems etc.

I am trying to test the app in a way how it would be tested manually. Start the app up, setup my users, configure my stuff, try out my requests and assert the responses.

And the way how the app behaves in the tests, when using the WebApplicationFactory, deviates from how it works when running it for real. As in if I run it directly, the static fields would be reinitialized and not remember the state from the last run.

So by doing integration testing how it is documented, you are basically barring developers from using mutable static fields in any way, as they would run the risk of having deviating behaviour between the tests and running it directly.

I cannot estimate the effort/possibility of reinitializing all static fields when the TestServer is started, but as I previously mentioned, this kind of behaviour is, in my opinion, unexpected and not clearly documented. And it is something a developer that does integration testing will run in. As some libraries use static fields for example for configuration, like the C# MongoDb Driver

So what exactly is the course of action here? Is this intendet behaviour? Are you supposed to basically mock all in-memory state to make sure that none is shared between test runs? Is there a better way to do the kind of testing I intend to do?

[martincostello]

Integration testing as you describe it is also what the documentation and the sample I linked you to aim to do, which is: interact with the application would do from the outside over HTTP.

The difference is that when you run the server "for real" it is the only process and ends as soon as the Main method returns. This is not how test frameworks work. They start the process and then work through the tests one by one executing them. This means that any static fields will persist their values for the lifetime of the test process.

As I noted earlier, this is true for any type of tests using NUnit, xunit or MSTest, because they all run their tests in the same process and memory space as your code.

This is not a unique problem to WebApplicationFactory. It is still creating a server that you're interacting with using an HttpClient, but it is still in the same memory space as your test program, which is why you see the behaviour you do.

This same code would have the same behaviour if you called it from your tests and return 1, 2, 3 etc:

public class Program
{
    private static int _count;
    public static int Main()
    {
        return ++count;
    }
}

The fundamental issue here is with your code - it is not written in a manner that isolates state in a way that cannot be observed by subsequent test invocations in any of the popular .NET test frameworks.

You need to refactor, such as by having the count managed by an instance of a class that is registered with DI as a singleton for the lifetime of a single web application. Such a design would not persist state between invocations, even with a static main method, because once the Main() method exits the instance would go out of scope and be garbage collected. The second invocation of the test would create a new instance and the count would start at zero again, etc.

Or, even simpler, use a local variable and capture it into the lambda method of your endpoint:

internal class Program
{
    private static void Main(string[] args)
    {
        var builder = WebApplication.CreateBuilder(args);
        var app = builder.Build();

        int a = 0;
        app.MapGet("/test", () =>
        {
            ++a;
            return a;
        });
        app.Run();
    }
}

You should observe the above application will count up from 1 if you run it by itself, but always return 1 in your test (because the test creates a new server on each invocation) no matter how many times it is run.

[martincostello]

Notice also that:

1) there are no mocks; 2) no static fields are reset anywhere; 3) the example I linked you to earlier uses a SQLite database, so is if the level of complexity you mention regarding MongoDB etc.

Ultimately, you need to not use statics where you want behaviour not to persist in your tests. Part of software design also involves designing your code to be testable, and static state is fundamentally not friendly to repeatable isolated tests.

[ascendise]

So in the end, I have to somehow work around this. It is sadly not my code that stores the state inside static fields, but the MongoDB Driver when registering the class maps. There might be another way to configure it, depends on the library.

When it comes to integration testing, I normally don't keep this stuff in mind, because it normally does not matter, but in the case with BsonClassMapper, this became an issue. Thats why this behaviour annoys me a bit, because for a unit test it is pretty overseeable how state is managed, as you are directly working with the class you are testing. How the docs explained it, I understood it as the WebApplicationFactory starting up a test server that hosts the app as it's own independable process and the test just interacting with it through HTTP.

Well, thank you for the help. Seems like I mainly misinterpreted how the WebApplicationFactory works.

[dotnet-policy-service[bot]]

This issue has been resolved and has not had any activity for 1 day. It will be closed for housekeeping purposes.

See our Issue Management Policies for more information.