Enable test execution in browser-wasm

idg10 commented 9 months ago

Summary

I want to be able to run tests inside a browser-hosted WASM environment. That environment behaves significantly differently from other .NET environments, so it's vital to test thoroughly, but we can't today because there seems to be no support for it.

We were hopeful when the new test runner was announced that it would address this scenario, since it deals with some other environments that used not to support running tests. But apparently not.

Background and Motivation

I maintain Rx.NET (System.Reactive) and we recently had a report of a problem that occurs in Blazor WASM.

When fixing problems I generally want to add a failing test first to verify that the fix addresses the problem, and to prevent regressions. But I can't do that here because there appears to be no way to run MSTest tests in browser-hosted WASM based .NET code. The problem described occurs only when running in the browser-wasm runtime, so for an automated test to reproduce the scenario that triggers this bug, we'd need that test to be able to execute in that runtime.

Proposed Feature

In an ideal world, we would just be able to add browser-wasm (and, since we're talking about WASM also wasi-wasm) to the <RuntimeIdentifiers> in a test project's csproj and for it to automatically run the tests in each listed runtime. But failing that, we'd like some way for browser-wasm apps to use the mechanisms that make the new test runner possible.

As far as we can see, at least two things would need to be addressed:

we'd need some mechanism to capture the output because there's no meaningful standard output in browser-wasm
the mechanism by which configuration is supplied would need to be modified

That second point is the first thing we run into when trying to do this today. Since letting the MSTest tooling generate the app entry point won't work for a browser-wasm app, we tried putting code similar to what it generates into our app:

string[] args = [];
global::Microsoft.Testing.Platform.Builder.ITestApplicationBuilder builder = await 
    global::Microsoft.Testing.Platform.Builder.TestApplication.CreateBuilderAsync(args);
Microsoft.Testing.Platform.MSBuild.TestingPlatformBuilderHook.AddExtensions(builder, args);
Microsoft.Testing.Extensions.Telemetry.TestingPlatformBuilderHook.AddExtensions(builder, args);
Task<ITestApplication> bt = builder.BuildAsync(); // Separating from await because single stepping async in blazor is weird
using (global::Microsoft.Testing.Platform.Builder.ITestApplication app = await bt)
{
    int result = await app.RunAsync();
}

This crashes inside builder.BuildAsync() call:

System.InvalidOperationException: Unexpected state in file '/_/src/Platform/Microsoft.Testing.Platform/Services/CurrentTestApplicationModuleInfo.cs' at line '64'
   at Microsoft.Testing.Platform.Helpers.ApplicationStateGuard.Ensure(Boolean condition, String path, Int32 line) in /_/src/Platform/Microsoft.Testing.Platform/Helpers/ApplicationStateGuard.cs:line 28
   at Microsoft.Testing.Platform.Services.CurrentTestApplicationModuleInfo.GetCurrentTestApplicationFullPath() in /_/src/Platform/Microsoft.Testing.Platform/Services/CurrentTestApplicationModuleInfo.cs:line 64
   at Microsoft.Testing.Platform.Configurations.JsonConfigurationSource.JsonConfigurationProvider.LoadAsync() in /_/src/Platform/Microsoft.Testing.Platform/Configurations/JsonConfigurationProvider.cs:line 32
   at Microsoft.Testing.Platform.Configurations.ConfigurationManager.BuildAsync(IFileLoggerProvider syncFileLoggerProvider) in /_/src/Platform/Microsoft.Testing.Platform/Configurations/ConfigurationManager.cs:line 40
   at Microsoft.Testing.Platform.Hosts.TestHostBuilder.BuildAsync(String[] args, ApplicationLoggingState loggingState, TestApplicationOptions testApplicationOptions, IUnhandledExceptionsHandler unhandledExceptionsHandler, DateTimeOffset createBuilderStart) in /_/src/Platform/Microsoft.Testing.Platform/Hosts/TestHostBuilder.cs:line 134
   at Microsoft.Testing.Platform.Builder.TestApplicationBuilder.BuildAsync() in /_/src/Platform/Microsoft.Testing.Platform/Builder/TestApplicationBuilder.cs:line 118
   at BlazorWasmStandaloneApp.Pages.Home.OnInitializedAsync() in C:\dev\temp\NewMsTestRunner2024Tests\BlazorWasmStandaloneApp\Pages\Home.razor:line 67
   at Microsoft.AspNetCore.Components.ComponentBase.RunInitAndSetParametersAsync()

The fundamental problem here is that the MS Test Runner makes the assumption that its JSON configuration file can be found in the filesystem nearby the main assembly. This assumption fails because with browser-wasm, the application assembly isn't actually visible on the filesystem. Both Environment.ProcessPath and Assembly.GetEntryAssembly()?.Location return null because they both represent something that doesn't really exist in browser-wasm.

The frustrating thing is that the internal design of the test runner looks like it could easily address these problems. A specialized IEnvironment implementation could return a value for the ProcessPath even though there isn't really any such file, and then a specialized IFileSystem implementation could make the necessary JSON available even though it's not really available in the way the code presumes it will be. These virtualization points look like they would provide a great way to work around browser-wasm's expectation-busting oddity. And console output is also virtualized, which would make it relatively straightforward to capture that output. (This goes on to raise some questions about how you then get those results back in to the IDE or the build output, but it's not really any different from UWP: that's an environment where you are required to launch an actual app with a window just to provide a place to run your tests; in-browser tests have very similar challenges, and a solution exists for UWP.)

But unfortunately, those features are all internal, so we can't use them.

Alternative Designs

The only alternative seems to be to write our own runner. In principle, nothing stops us from loading up a test adapter component in the same way the real runner does. But this does not seem to be a trivial endeavour.

Also, it doesn't seem to be possible in practice right now due to https://github.com/microsoft/vstest/issues/4863

This, and the error described above are unlikely to be the only problem—they're just the first thing you hit if you try this today.

AB#1950769

Evangelink commented 9 months ago

Hi @idg10,

Happy to see that the runner seems like a good opportunity for you!

In all fairness, we did some POC trying to bring support of the runner for "packaged" environments (UWP, WinUI, MAUI...) but we wanted to be able to dedicate enough time to correctly test these environments and so decided to drop it from the scope of the first release. I'll bring it back to the table on our internal discussions so we can make sure to prioritize some time for it.