Improving template tests in dotnet/installer

Background

One of the most common problems with .NET templates shipped within .NET SDK are the issues which defining grouping metadata: identity, group identity, precedence. This issue is commonly reproduced only when multiple .NET SDKs are installed. Invalid definition results in the conflict that is shown only when the template is attempted to be run and the user cannot resolve it.

During delivery of .NET 5 test templates and common templates had this issue. During delivery of .NET 6 ASP.NET templates and WPF templates had this issue. Issue with item templates in asp.net was discovered post preview 7.

The problem is amplified because the templates are delivered by 5 teams as of now, using different approaches for authoring, infrastructure and testing their templates. dotnet/installer seems to be the only common point for them.

Proposal

Since the dotnet/installer is the first and last point where the template packages are inserted, create the tests that can catch the issues described above and disallow releasing the build until they are solved. The tests will be written by template engine team, the maintenance of them may be negotiated between template engine team, dotnet/installer team or template authors.

Technical details

The tests are suggested as the separate job in existing pipeline. Running tests on a single platform is enough - should the issue occur, it is identically reproduced on all the platforms.

The tests will do the following actions:

Preparation steps:

download and install currently supported SDK versions using dotnet-install scripts (at the moment 3.1, 6.0, 7.0). This can be done as part of the build or from C# code.
unpack build install on top (artifacts\packages\<configuration>\Shipping\dotnet-sdk-<version>.zip)

Test action:

run all the available templates for all available frameworks (in case template support framework parameter), without restore - dotnet new <template> --framework <framework> --no-restore.
check if template instantiated successfully, based on output or/and exit code.

There are 2 ways how to ensure the test matrix:

prepare test data programmatically
- run dotnet new list to get all available templates
- run dotnet new <template> --help to see if template supports --framework and its values
- prepare test matrix based on discovered data
- pros: low maintenance costs, however if list of help output changes the test might need adjustments
- cons: the risk of bugs in logic, the risk of missing the cases, the risk that templates are not supporting all the frameworks they normally should
prepare test data manually
- the test matrix to be hardcoded in code
- when adding new template or supported TFM, the matrix has to be manually adjusted accordingly
- to verify that the matrix is up-to-date, the sanity test will be written failing if matrix doesn't contain all the results from dotnet new list (and dotnet new <template> --help)
- pros: eliminated risk of option 1
- cons: possibly higher maintenance costs comparing to option 1, if list of help output changes the santiy test might need adjustments

Planned number of test cases (maximum): 40 templates x 2 languages (average) x 4 TFMs = 320 cases. Note: it is a maximum number of tests as not all the templates support all the frameworks.

Predicted test run time: 320 cases * 2 sec = 640 sec ~ 11 minutes Note: doesn't include build, test preparation steps, test matrix preparation.

Other possible improvements:

Existing template tests may benefit from matrix generation / sanity tests to ensure that all the templates are being tested.

Tasks

Justification

Engineering Impact

Time savings from streamlined test execution in the installer pipelines.

dotnet / templating

SDK templates testing in dotnet/installer #6026