Missing option to override TestRunParameters

[jvanderwoude80]

My tests will only run when a certain TestRunParameter in the runssettings file is overriden with a value generated in a pipeline (a token in this case). This option is missing from the dotnet-test-rerun tool, but is possible with dotnet test using the following syntax:

# cmd
dotnet test  -- TestRunParameters.Parameter(name=\"myParam\", value=\"value\")

This is not well documented unfortunately (and this overlooked easily ;), this page on dotnet test options: https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-test#options links (last line of the options segment) to another page: https://github.com/Microsoft/vstest-docs/blob/main/docs/RunSettingsArguments.md where the snippet above can be found.

Is it possible to add this to the dotnet-test-rerun tool? Would be much appreciated. I really need rerun functionality, I have a working powershell script for this after some work, but I think your tool would be a much better solution.

[ricardofslp]

Hi @jvanderwoude80 ,

Will try to work on this during this weekend.

[ricardofslp]

Hi @jvanderwoude80 ,

Have you tried to add this with the option "--inlineRunSettings"? I think if you had --inlineRunSettings TestRunParameters.Parameter(name=\"myParam\", value=\"value\") it should work

We did this sligthly different to have some more control

[jvanderwoude80]

Hi @joaoopereira,

Thx, didn't think of that!

However it doesn't seem to work (yet). If I try:

--inlineRunSettings TestRunParameters.Parameter(name="myParam", value="value")

The following error is output by the dotnet test command:

The test run parameter argument 'TestRunParameters.Parameter(name=myParam, value=myValue) is invalid. Please use the format below.
Format: TestRunParameters.Parameter(name=\"<name>\", value=\"<value>\")
command: dotnet test <not relevant arguments> -- TestRunParameters.Parameter(name=myParam, value=myValue)

The command is also logged and indeed the TestRunParameter is added the right way by test-rerun as you suggested, but it is missing the \" around name and value.

If I try the suggestion: --inlineRunSettings TestRunParameters.Parameter(name=\"myParam\", value=\"value\")

Same error, but the command is looking better now, however still no \ before the " in the command.

The test run parameter argument 'TestRunParameters.Parameter(name=myParam, value=myValue) is invalid. Please use the format below.
Format: TestRunParameters.Parameter(name=\"<name>\", value=\"<value>\")
command: dotnet test <not relevant arguments> -- TestRunParameters.Parameter(name="myParam", value="myValue")

I had the same problem using the dotnet task directly (without test-rerun), only way to make it work was using this syntax in the dotnet test task:

-- TestRunParameters.Parameter(name="\""keyVaultAccessToken"\"", value="\""$(ServiceTeamKvAccessToken)"\"")

Resulting in this log entries:

 ##[debug]C:\agent\_work\_tool\dotnet\dotnet.exe arg:  <not relevant arguments> -- TestRunParameters.Parameter(name="\""myParam"\"", value="\""myValue"\"")
[command]C:\agent\_work\_tool\dotnet\dotnet.exe test <not relevant arguments> -- "TestRunParameters.Parameter(name=\"myParam\"," "value=\"myValue\")"

Here the command does have the \" around param and value. I tried all possible permutations of ", \", "", "\", "\"", etc. in test-rerun command. Always the same error.

So I guess using test-rerun somewhere down the line the \ is stripped or something resulting in invalid format. Can you look into that?

[ricardofslp]

Hi @jvanderwoude80 ,

Thanks for that feedback. I will try to look into that.

[ricardofslp]

Hi @jvanderwoude80 ,

Please test that same command with version: https://www.nuget.org/packages/dotnet-test-rerun/2.0.1-alpha.1

[jvanderwoude80]

Hi @joaoopereira,

Yes, it seems to work now using this command: --inlineRunSettings TestRunParameters.Parameter(name="\""keyVaultAccessToken"\"", value="\""$(ServiceTeamKvAccessToken)"\"")

Failed tests are now rerun. Thanks!

(now I have some issues with test results not being published to Azure DevOps, but I guess that's not something the dotnet test cli command or test-rerun tool handles, but something onlyt the dotnet test Azure Devops task does, I guess I have to explicitly publish the result using the respective Azure DevOps task)

[joaoopereira]

Hi @jvanderwoude80,

Yes you need to explicitly call the PublishTestResults task. Here is an example:

- task: PublishTestResults@2
  displayName: Publish Test Results
  inputs:
    testRunner: VSTest
    testResultsFiles: test-results.trx

[ricardofslp]

Hi @jvanderwoude80 ,

Because of the need to publish the results we are supporting another dotnet tool that merges all trx files into one (the goal is not to show reruns if we got a pass at the end).

Please take a look: https://github.com/ricardofslp/dotnet-trx-merge

[jvanderwoude80]

Hi @joaoopereira,

That's interesting! I will take a look. However I'm sort of looking for the opposite.

Been experimenting with the PublishTestResults task, it seems to have the ability to merge test results files of different attempts. But then the information about reruns gets lost, and I'm actually very much interested in that, because then I can identify flaky tests (failing first time, passing second time). And even more important I want to see the original error message of the failing attempt to identify the root cause of the flakiness and be able to fix it.

This functionality is provided by the 'legacy' VSTest task, where both attempts (see screenshot below) including all error information (stack trace, standard_console_output log) are shown.

I'm trying to accomplish this with the dotnet test / publish test results task(s), but no luck yet. Seems VSTest performs some magic under water and publishes test tasks directly using the API to which attempt information can be provided, something I cannot accomplish using the PublishTestResuls task.

[ricardofslp]

Hi @jvanderwoude80 ,

Version 2.0.1 has been officially launched https://www.nuget.org/packages/dotnet-test-rerun/2.0.1