How do I test my app after publishing with linker?

[ForNeVeR]

I'm sorry if this question doesn't belong to this repository, but dotnet/core repository says the feedback is welcome here, so here I am.

I have read the documentation on ILLink tasks, and it is a common recommendation to test my application after publishing: presumably, improperly configured linker could break something, e.g. delete code that's actually used in some weird reflection-driven path.

So, okay, I've read the documentation on testing after publish, and it seem to recommend me to publish the test project. But if I publish my test project, I'll get other set of binaries that are separately processed by the publish pipeline, right? I can't see the point of testing that other set of binaries: I think that I should test what I ship, and not something else. Also, it doesn't seems that unit test projects even support the publishing with linker support. Here's what the publish tells me if I try to publish an XUnit test project (created via dotnet new xunit):

$ dotnet publish --self-contained true --configuration Release --runtime win-x64 --output publish -p:PublishTrimmed=true
Microsoft (R) Build Engine version 16.5.0+d4cbfca49 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 925,97 ms for T:\Temp\xunittestproj\xunittestproj.csproj.
C:\Program Files\dotnet\sdk\3.1.201\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(136,5): error NETSDK1067: Self-contained applications are required to use the application host. Either set SelfContained to false or set UseAppHost to true. [T:\Temp\xunittestproj\xunittestproj.csproj]

So, my question is: how do I run my already written test suite against the binary published by the Mono linker? Say, I have two projects: one dotnet new console and one dotnet new xunit which is testing the former. Now, if I publish the app via

$ dotnet publish --configuration Release --runtime win-x64 --output publish -p:PublishTrimmed=true ConsoleApp

how do I run tests against this binary? Is there any best practice or at least a recommendation?

For this experiment, I use the latest stable .NET Core SDK, version 3.1.201, but I'm looking for a general answer: if this is something that will be resolved in .NET 5, that's completely fine.

[MichalStrehovsky]

Unfortunately, "testing your application after publishing" means running end to end tests on the actual application bits - so if this is e.g. a web service, testing would involve launching the service and hitting the endpoint with network requests and validating they do the right thing. If this is a e.g. a UI application, this would involve running UI automation tests (if there's any), or clicking through the entire UI to validate things work manually.

Publishing XUnit tests will produce different results - linker would see different code, prompting it to keep different things. Problems that don't exist in the final app might show up and vice versa (problems that exist in the final app don't show up for the test because linker happened to keep something).

For .NET 5, we're looking into adding ability for linker to analyze more of reflection in the app and ability of the linker to warn if there's problematic code. This should hopefully improve the experience for a subset of reflection use cases (not all, unfortunately, since reflection is impossible to statically analyze in general - in those cases linker will warn, even though the use might be safe).

More on some of the approach we're taking here: https://github.com/mono/linker/blob/master/docs/design/reflection-flow.md

[ForNeVeR]

Thanks a lot for a thorough answer, Michal!

This is okay; from now on, I'll start adding e2e tests to the plans of applications that use dotnet linker in any capacity. After all, this kind of tests is useful even without linker, and it may be even more useful if other interesting projects of .NET team will gain traction (e.g. these tests may later be used to check behavior after CoreRT or a similar AOT mechanism being developed in the future).

But I feel that in particular case of the Mono linker, the situation may be greatly improved by just running the ordinary unit tests against the processed application.

(Please correct me if I'm wrong in my assumptions below)

The application after publish should still be pure IL code, with most (or even all) the public symbols still available to be called from tests. So, if I add a reference to the published artifact from my unit test library, it should (theoretically) be possible to run the tests (almost) as usual, right? So, if my unit tests (as an exaggregated example) give me 100% code coverage, then any misbehavior of the Mono linker should be detected by a usual unit test run.

One problem with this approach would be removal of any test-exclusive public endpoints by the Mono linker, but these're exactly the cases when the linker configuration should be useful, since there shouldn't be very many of such cases, and absolutely all of them lie on the assembly public surface. So, such cases will be easily tractable, and it should be easy to add them to the XML config for the linker to consider these entry points.

Another problem is that current tooling has no options for this kind of testing (i.e. there's no easy/documented way to add "a reference to an assembly published by the Mono linker" to my unit tests — since this generally will require to use the post-linker versions of all the transitive dependencies, too), but I think this is something doable even with current tools very easily in a semi-automated way. For example, I could do something along the following lines:

1. Compile the unit test assembly
2. Publish the main application using the Mono linker
3. Publish the unit test assembly (without a linker)
4. Just copy and replace the output of step 2 to the output of step 3
5. Run the test suite as usual using the step 4 output

(it could require me to additionally modify the deps.json/runtime.json files, but I'm not too sure about that)

What do you think of this idea? Is it viable, or is there something I'm missing here?

Is it something for the team to consider improving current tools/approach?

[vitek-karas]

Sorry for missing the question here (for 2 years :-(). It's correct that currently there's no tooling to help you do this. That said we're thinking about enabling tests to be built as apps - meaning the test would actually produce an executable which when executed runs the tests. This would allow for the SDK to build the test as a normal app, meaning it can also be published - thus supporting trimming, crossgen, NativeAOT, ...

Note though that most test frameworks are not ready for trimming as they heavily rely on reflection for test discovery and execution. It's obviously fixable (source generators for example), but as far as I know it hasn't happened yet.

@MarcoRossignoli as FYI on the test scenarios

Closing this as I think the question has been answered.