Decouple WebApplicationFactory and TestServer

[martincostello]

Describe the bug

As part of looking into the new features in ASP.NET Core 6 (top-level statements, minimal APIs, etc.) I've been looking at how to refactor the integration test approach I've been using with previous versions of .NET Core where WebApplicationFactory is available so that it works with the new approaches.

For integration tests where a UI is required, such as for browser automation tests, I've been tackling this by creating a derived WebApplicationFactory class to piggy-back its features to bootstrap an HTTP server using Kestrel so that there's a real HTTP port being listened to so that tools like Playwright and Selenium can interact with the application to test it.

These tests work by using the CreateHostBuilder()/CreateWebHostBuilder() methods to access the build for the application and then manually creating it (rather than using the TestServer the class usually provides) (example). The reason for re-using the WebApplicationFactory code is that there's a logic embedded within it for finding the default content root, ensuring the .deps.json files are there etc., which is a fair chunk of additional code to copy and maintain to otherwise replicate the approach with only minor tweaks on top (i.e. a real server). It also gives good code coverage of the same code that runs in production, rather than having to use an alternate code path just for the purpose of tests.

Trying this out with the changes from #33462 using a preview 6 daily build however doesn't work for this scenario. This is because in the top-level statements scenario both methods return null and the deferred implementation is private to the EnsureServer() method:

https://github.com/dotnet/aspnetcore/blob/46ef939508a2c733104726a6c744368446883dc6/src/Mvc/Mvc.Testing/src/WebApplicationFactory.cs#L142-L178

If the implementation were to be refactored in a way that supported the existing scenarios by allowing the consuming class to get access to the DeferredHostBuilder as an IHostBuilder, then I presume that the use case I have today would work if that builder was used to bootstrap an application with it instead.

Off the top of my head, maybe something like this could be a possible approach:

protected virtual IHostBuilder? CreateHostBuilder()
{
    var hostBuilder = HostFactoryResolver.ResolveHostBuilderFactory<IHostBuilder>(typeof(TEntryPoint).Assembly)?.Invoke(Array.Empty<string>());

    if (hostBuilder is null)
    {
        var deferredHostBuilder = new DeferredHostBuilder();
        var factory = HostFactoryResolver.ResolveHostFactory(
            typeof(TEntryPoint).Assembly,
            stopApplication: false,
            configureHostBuilder: deferredHostBuilder.ConfigureHostBuilder,
            entrypointCompleted: deferredHostBuilder.EntryPointCompleted);

        if (factory is not null)
        {
            deferredHostBuilder.SetHostFactory(factory);
            hostBuilder = deferredHostBuilder;
        }
    }

    hostBuilder?.UseEnvironment(Environments.Development);
    return hostBuilder;
}

The EnsureServer() method would then just consume the deferred implementation without actually having any knowledge of it, and derived classes would be able to use the deferred implementation without being aware of the actual implementation details.

I've got a GitHub repo here with a sample TodoApp using this test approach using ASP.NET Core 6 preview 5 here (it doesn't use minimal APIs yet, mainly due to this issue), and there's a branch using a preview 6 daily build with this approach that fails to run the tests due to the lack of access to a host builder.

While maybe this isn't an intended use case of WebApplicationFactory, it's been working well since ASP.NET Core 2.1 and would require a fair bit of work to move away from to leverage the new functionalities in various application codebases in order to adopt ASP.NET Core 6.

If some minimal refactoring could be done that doesn't break the intended design, which I would be happy to contribute to, that could get this sort of use case working again with ASP.NET Core 6 using top-level statements that would be appreciated.

/cc @davidfowl

To Reproduce

To reproduce, clone the preview-6 branch of my work-in-progress sample application.

Further technical details

• .NET SDK 6.0.100-preview.6.21324.1 (a daily build)
• Microsoft.AspNetCore.Mvc.Testing version 6.0.0-preview.6.21323.4
• Visual Studio 2022 17.0.0 Preview 1.1

[davidfowl]

I originally did this but it breaks applications that were using the CreateWebHost pattern, that's why it got moved to be a fallback. I can think of a sneaky way to solve this if we what you suggest above and only use the IHostBuilder if it's not the DeferredHostBuilder. That is, we would keep the precedence as it is today but CreateHostBuilder wouldn't return null.

But stepping back a bit here, it seems like we should just decouple the WebApplicationFactory from the TestServer.

[martincostello]

But stepping back a bit here, it seems like we should just decouple the WebApplicationFactory from the TestServer.

Yeah, I definitely think there would be a benefit in exposing the building blocks that are in the implementation details out so you can compose them up in different ways. I'm not sure what form they'd take, but factoring the .deps, content root etc. bits out into some sort of ""helpers"" that could be re-used, and then having WebApplicationFactory be refactored to utilise those would remove the need to sort-of user TestServer but then ignore it completely.

I feel like maybe this was discussed in a different PR or issue for WebApplicationFactory I was involved in previously, but there was some concern about doing so. I'll have a search and see if I recall correctly or if I've just imagined it 😄

[martincostello]

I feel like maybe this was discussed in a different PR or issue for WebApplicationFactory I was involved in previously, but there was some concern about doing so. I'll have a search and see if I recall correctly or if I've just imagined it 😄

It was this comment here I was thinking of: https://github.com/dotnet/aspnetcore/pull/7414#issuecomment-470059404

[ghost]

Thanks for contacting us.

We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s). If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[davidfowl]

I wonder if we can make a small change here to make the TestServer property be null if it a real server is in use. Instead typing the server here as IServer. Then change CreateClient to use the IServer to get the server addresses to create a real http client if it's not a test server. I can play with this tonight.

[martincostello]

The proposed changes would still be better as I could remove yet more code, but I had a think about this a bit more, and managed to refactor my usage to get rid of the gnarly reflection to access the deferred host builder and move my "hook" into CreateHost(). The only bit I don't like is that the real server has to be started first, otherwise you can't get the server addresses. I guess that's a consequence of the deferred host(builder) and the callback/event that gets fired by that once the application is built so the WebApplicationFactory<T> callbacks can run.

using System.Security.Cryptography.X509Certificates;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server;
using Microsoft.AspNetCore.Hosting.Server.Features;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

namespace MyApp;

public sealed class HttpServerFixture : WebApplicationFactory<MyEntrypoint>
{
    private bool _disposed;
    private IHost? _host;

    public string ServerAddress
    {
        get
        {
            EnsureServer();
            return ClientOptions.BaseAddress.ToString();
        }
    }

    public override IServiceProvider Services
    {
        get
        {
            EnsureServer();
            return _host!.Services!;
        }
    }

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        base.ConfigureWebHost(builder);

        builder.ConfigureKestrel(
            serverOptions => serverOptions.ConfigureHttpsDefaults(
                httpsOptions => httpsOptions.ServerCertificate = new X509Certificate2("localhost-dev.pfx", "Pa55w0rd!")));

        builder.UseUrls("https://127.0.0.1:0");
    }

    protected override IHost CreateHost(IHostBuilder builder)
    {
        // Create the host for TestServer now before we
        // modify the builder to use Kestrel instead.
        var testHost = builder.Build();

        // Modify the host builder to use Kestrel instead
        // of TestServer so we can listen on a real address.
        builder.ConfigureWebHost((p) => p.UseKestrel());

        // Create and start the Kestrel server before the test server,
        // otherwise due to the way the deferred host builder works
        // for minimal hosting, the server will not get "initialized
        // enough" for the address it is listening on to be available.
        _host = builder.Build();
        _host.Start();

        // Extract the selected dynamic port out of the Kestrel server
        // and assign it onto the client options for convenience so it
        // "just works" as otherwise it'll be the default http://localhost
        // URL, which won't route to the Kestrel-hosted HTTP server.
        var server = _host.Services.GetRequiredService<IServer>();
        var addresses = server.Features.Get<IServerAddressesFeature>();

        ClientOptions.BaseAddress = addresses!.Addresses
            .Select((p) => new Uri(p))
            .Last();

        // Return the host that uses TestServer, rather than the real one.
        // Otherwise the internals will complain about the host's server
        // not being an instance of the concrete type TestServer.
        testHost.Start();
        return testHost;
    }

    protected override void Dispose(bool disposing)
    {
        base.Dispose(disposing);

        if (!_disposed)
        {
            if (disposing)
            {
                _host?.Dispose();
            }

            _disposed = true;
        }
    }

    private void EnsureServer()
    {
        // This forces WebApplicationFactory to bootstrap the server
        using var _ = CreateDefaultClient();
    }
}

[ghost]

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

[achselschweisz]

The proposed changes would still be better as I could remove yet more code, but I had a think about this a bit more, and managed to refactor my usage to get rid of the gnarly reflection to access the deferred host builder and move my "hook" into CreateHost(). The only bit I don't like is that the real server has to be started first, otherwise you can't get the server addresses. I guess that's a consequence of the deferred host(builder) and the callback/event that gets fired by that once the application is built so the WebApplicationFactory<T> callbacks can run.

Slightly off-topic but I need to mention this here anyway: Thanks to this solution I finally managed to get integration tests running using Kestrel + uds (unix domain sockets). Cheers!

[ghost]

Thanks for contacting us.

We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s). If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[jepperaskdk]

When calling .Build() twice, I get: "System.InvalidOperationException : Build can only be called once." Any idea why? (.NET 6 and .NET 7)

Looks like it has been there for a long time: https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.Extensions.Hosting/src/HostBuilder.cs#L150C1-L154

[Barsonax]

The proposed changes would still be better as I could remove yet more code, but I had a think about this a bit more, and managed to refactor my usage to get rid of the gnarly reflection to access the deferred host builder and move my "hook" into CreateHost(). The only bit I don't like is that the real server has to be started first, otherwise you can't get the server addresses. I guess that's a consequence of the deferred host(builder) and the callback/event that gets fired by that once the application is built so the WebApplicationFactory<T> callbacks can run.

using System.Security.Cryptography.X509Certificates;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server;
using Microsoft.AspNetCore.Hosting.Server.Features;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

namespace MyApp;

public sealed class HttpServerFixture : WebApplicationFactory<MyEntrypoint>
{
    private bool _disposed;
    private IHost? _host;

    public string ServerAddress
    {
        get
        {
            EnsureServer();
            return ClientOptions.BaseAddress.ToString();
        }
    }

    public override IServiceProvider Services
    {
        get
        {
            EnsureServer();
            return _host!.Services!;
        }
    }

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        base.ConfigureWebHost(builder);

        builder.ConfigureKestrel(
            serverOptions => serverOptions.ConfigureHttpsDefaults(
                httpsOptions => httpsOptions.ServerCertificate = new X509Certificate2("localhost-dev.pfx", "Pa55w0rd!")));

        builder.UseUrls("https://127.0.0.1:0");
    }

    protected override IHost CreateHost(IHostBuilder builder)
    {
        // Create the host for TestServer now before we
        // modify the builder to use Kestrel instead.
        var testHost = builder.Build();

        // Modify the host builder to use Kestrel instead
        // of TestServer so we can listen on a real address.
        builder.ConfigureWebHost((p) => p.UseKestrel());

        // Create and start the Kestrel server before the test server,
        // otherwise due to the way the deferred host builder works
        // for minimal hosting, the server will not get "initialized
        // enough" for the address it is listening on to be available.
        _host = builder.Build();
        _host.Start();

        // Extract the selected dynamic port out of the Kestrel server
        // and assign it onto the client options for convenience so it
        // "just works" as otherwise it'll be the default http://localhost
        // URL, which won't route to the Kestrel-hosted HTTP server.
        var server = _host.Services.GetRequiredService<IServer>();
        var addresses = server.Features.Get<IServerAddressesFeature>();

        ClientOptions.BaseAddress = addresses!.Addresses
            .Select((p) => new Uri(p))
            .Last();

        // Return the host that uses TestServer, rather than the real one.
        // Otherwise the internals will complain about the host's server
        // not being an instance of the concrete type TestServer.
        testHost.Start();
        return testHost;
    }

    protected override void Dispose(bool disposing)
    {
        base.Dispose(disposing);

        if (!_disposed)
        {
            if (disposing)
            {
                _host?.Dispose();
            }

            _disposed = true;
        }
    }

    private void EnsureServer()
    {
        // This forces WebApplicationFactory to bootstrap the server
        using var _ = CreateDefaultClient();
    }
}

Calling CreateDefaultClient sometimes fails with System.InvalidOperationException : The server has not been started or no web application was configured.. Iam assuming here you are only calling this to call EnsureServer so in my code I have changed this to simply call var foo = base.Services which internally will call EnsureServer which seems to work. Quite hacky still and I think using reflection might actually be better here as that captures the intent in a better way.

Using this approach with playwright here: https://github.com/Barsonax/AwesomeApiTest/blob/master/Examples/Browser/AwesomeApiTest.Nunit/TestSetup/AwesomeApiTestSut.cs

Would be very nice if there was a clean way of doing this.

[KSemenenko]

is there any updates? maybe .net 9?

[martincostello]

It's too late for .NET 9 - this wouldn't be until .NET 10 at the earliest now.

[KSemenenko]

Then I push temporary solution https://github.com/managedcode/IntegrationTestBaseKit I add http liner, signalR client and playwright.

[pacoteinnov]

When calling .Build() twice, I get: "System.InvalidOperationException : Build can only be called once." Any idea why? (.NET 6 and .NET 7)

Looks like it has been there for a long time: https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.Extensions.Hosting/src/HostBuilder.cs#L150C1-L154

You probably have a public static IHostBuilder CreateHostBuilder(string[] args) somewhere that gets called.