dotnet test - allow to specify msbuild properties

[tapika]

At the moment if you run dotnet test myexample.csproj - by default it would try to build and run that test. I however would like to pass extra msbuild parameters to project file, so I could mimic project presence in solution.

Normally you would like to set SolutionName and SolutionDir properties, but it's possible that even custom properties needs to be defined.

Command line syntax could be extended to be something like this:

dotnet test -pSolutioName=mySolution myexample.csproj

(Pure example, dev team can decide best approach)

[KalleOlaviNiemitalo]

If https://github.com/NuGet/Home/issues/11408 for dotnet list package is also implemented, I'd like these to use the same option syntax.

[tapika]

Hmm... There exists a way to walkaround missing property setting. It's possible to pass property over environment variable, but of course such things as SolutionName gets overridden by msbuild itself. One approach is just to declare custom property name if it's not defined (via environment variable).

  <PropertyGroup Condition=" '$(MySolutionName)' == '' ">
    <MySolutionName>$(SolutionName)</MySolutionName>
  </PropertyGroup>

and then continue using MySolutionName everywhere.

And override that as environment variable from build system before it, for example:

Cake: Environment.SetEnvironmentVariable("MySolutionName", ...);
Batch: set MySolutionName=...

And so on...

[nohwnd]

This already works:

dotnet test .\mstest40.csproj -p:SolutionName=mySolution.sln -bl

[KalleOlaviNiemitalo]

I don't see -p listed in dotnet test --help though. Neither the --help output nor the dotnet test documentation says that unlisted MSBuild options can be used.

[KalleOlaviNiemitalo]

I didn't find any code that explicitly forwards unrecognized options of dotnet test to MSBuild. Do the options end up in result.GetValueForArgument(TestCommandParser.SlnOrProjectArgument) here? In which case, this might be working by accident.

https://github.com/dotnet/sdk/blob/957ae5ca599fdeaee425d23928d42da711373a5e/src/Cli/dotnet/commands/dotnet-test/Program.cs#L34-L36

[nohwnd]

It does not end up there. You can provide whatever properties you want including the other msbuild parameters (see me use -bl above to get binary msbuild log). dotnet test + csproj and dotnet test + sln (but not dotnet test + dll) will translate all the provided parameters they can recognize into msbuild properties and forward the rest to msbuild.

[baronfel]

A bit of context - you should be looking at the Run method in the dotnet-test\Program.cs file. These Run methods are invoked from the System.CommandLine Command Handlers in the matching Parser classes that generally are siblings of the Programs.

The key thing you want to look at is line 82, parseResults.GetArguments(). In System.CommandLine, this returns the entire arguments array for the command in raw form, and we forward those along to VSTest directly. The Parser configuration in TestCommandParser.cs is used primarily for fast help generation and tab-completion in the SDK's CLI itself. WE aim over time to encourage teams that we integrate with in the SDK to ship System.CommandLine Commands to us directly, so that those teams control help and tab-completion in their entirety. We're currently trialling this process with the dotnet/templating team (who contribute the dotnet new templating command) and hope to do more along these lines in the future.

[baronfel]

@KalleOlaviNiemitalo what you're seeing here is that the test help output isn't exhaustive - I would encourage any contributions to bring the SDK's definition in-line with that of the raw VSTest command, but due to the separation (and not having a Command from the VSTest library) this sort of thing is not uncommon. I've fixed a few for NuGet in a recent PR, for example.

[KalleOlaviNiemitalo]

The key thing you want to look at is line 82, parseResults.GetArguments().

The result of parseResults.GetArguments() goes to the args variable, which then is used only if (ContainsBuiltTestSources(args)). Do you mean ContainsBuiltTestSources returns true in dotnet test .\mstest40.csproj -p:SolutionName=mySolution.sln -bl?

[baronfel]

Figured it out - this is actually an unanticipated interaction of the current argument configuration of the parser. Line 36, where the SlnOrProjectArgument is added to the options to forward, is where the -p argument is added. This is...unanticipated...to say the least.

[KalleOlaviNiemitalo]

Well, that explains why it is not documented.

[baronfel]

If we change the Arity of the SlnOrProjectArgument to ZeroOrOne (I believe that this matches MSBuild's syntax, but would need to check with the team), then we would also need to either:

• add a new argument for specifically unmatched tokens, then forward that argument list. or,
• on the Command set TreatUnmatchedTokensAsErrors to false and forward along any unmatched tokens (I believe this is the route we're taking in the infinite-loop fix, but I don't think it's necessarily correct either), or
• some other route I've not yet seen.

[baronfel]

Reopened this so that we can properly fix the argument-forwarding of this command.

[KalleOlaviNiemitalo]

Does BuildCommand have the same quirk? BuildCommandParser.SlnOrProjectArgument likewise has arity ZeroOrMore. dotnet build at least documents that it accepts MSBuild options.

[baronfel]

It 100% does, yes.

[nohwnd]

@baronfel Some more context: There are actually two (very distinct) paths dotnet test can take depending on whether you provided .dll OR .csproj / .sln

When you provide .dll you the arguments will be forwarded directly to vstest.console, based on the arguments we extract from parse result (and split them to args and inline settings dotnet test <args> -- <inline settings>). This is a quickfix, that someone probably did before my time to get dotnet vstest and dotnet test merged under dotnet test while keeping all the other functionality. This is unfortunate and means that dotnet test + dll and dotnet test + csproj have two very different parameter sets.

if you don't provide dll, which is the case described here, we don't go directly to vstest.console, instead all the recognized parameters are translated to MSBuild properties, and the rest is just forwarded to MSBuild. Most of those properties are just passing through to end up in our VSTest build task, which will end up calling vstest.console + the built dll path produced by MSBuild.

https://github.com/dotnet/sdk/blob/480db67d961cd7046c1d145d55f780dc164be2ad/src/Cli/dotnet/commands/dotnet-test/Program.cs#L113

---

We would definitely be interested in unifying our parameter sets for dotnet test + csproj, dotnet test + dll and vstest.console, and only add the extra options we need for dotnet test + csproj (e.g. --no-build), or possibly just consume them from the MSBuild set (if such thing is possible). We talked about doing this unification in the past multiple times, but we have a lot of legacy syntax to support, and that must not break. But on the other hand we often get issues that are caused just by parsing the commands in our code here in sdk, and we also got broken in few random places (like the current issue) when sdk moved from the previous command parser to the current command parser.

---

Are you going to fix the issue with SlnOrProjectArgument? Or is that something I need to pay attention to?

[KalleOlaviNiemitalo]

Perhaps a low-risk set of fixes would be:

1. In each command, rename SlnOrProjectArgument to OtherMSBuildArgumentOrOption to reflect how it is actually used.
2. Make dotnet build --help and dotnet test --help display a message saying that -help (with only one dash) displays the MSBuild options that can also be used.
3. Document in dotnet test that MSBuild options can be used unless <PROJECT | SOLUTION> refers to an assembly file (dll or exe).

[nohwnd]

2) you can do dotnet msbuild --help, and dotnet vstest --help to get the help for msbuild and vstest.console, I would prefer that rather than -help.

[baronfel]

@nohwnd no action needed right now - there's some churn coming in the form of #23688 that might need to settle first, and I think the SlnOrProjectArgument situation should be looked at across all of the msbuild-forwarding commands in the SDK holistically. They'll probably all need similar changes -

• Arity change to ZeroOrOne,
• TreatUnmatchedTokensAsErrors set to false
• explicitly forward along unmatched tokens to the called application

and I'd want to do that all in one pass, to minimize the changes. Sound good?

[nohwnd]

Yup, I was just making sure about who should be doing what. We will update our help in #28066 in the meantime, to mention vstest.console and msbuild.

[StefanSchoof]

I today found out that a dotnet build --property:SkipFrontendBuild=true is working, but for test I need to have a dotnet test -p:SkipFrontendBuild=true and a --property is silently ignored . Took me a moment.