Running on ADO issue

[dgkanatsios]

For some reason, running this on ADO doesn't do test rerun.

This is the way we are using it (altering some sensitive info from project names and variables):

- task: Bash@3
      continueOnError: true
      displayName: 'Run functional tests with retries'
      inputs:
        failOnStderr: false
        targetType: 'inline'
        script: |
          # https://github.com/joaoopereira/dotnet-test-rerun
          dotnet tool install --global dotnet-test-rerun --version 1.8.0 --no-cache

          project="$(Build.SourcesDirectory)/src/project.sln"
          filter="(TestCategory=SomeCategory)"

          # build the test solution
          dotnet build $project --configuration Debug

          # run the tests
         test-rerun $project --configuration Debug --loglevel Debug --delay 3 --rerunMaxAttempts 3 --no-build --results-directory $(Agent.TempDirectory) --settings "$(Build.SourcesDirectory)/src/test.runsettings" --logger trx --verbosity detailed --filter=$filter --inlineRunSettings '"TestRunParameters.Parameter(name=\"name\", value=\"value\")"'

Detailed log output:

...

Total tests: 287
Test Run Failed.
     Passed: 255
     Failed: 29
    Skipped: 3
 Total time: 21.8892 Minutes
     6>VSTest:
         MSB4181: The "Microsoft.TestPlatform.Build.Tasks.VSTestTask" task returned false but did not log an error.
       Done executing task "Microsoft.TestPlatform.Build.Tasks.VSTestTask" -- FAILED.
     6>Done building target "VSTest" in project "project.csproj" -- FAILED.
     6>Done Building Project "/mnt/vss/_work/2/s/src/project.csproj" (VSTest target(s)) -- FAILED.
     1>Done executing task "MSBuild" -- FAILED.
     1>Done building target "VSTest" in project "project.sln" -- FAILED.
     1>Done Building Project "/mnt/vss/_work/2/s/src/project.sln" (VSTest target(s)) -- FAILED.

Build FAILED.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:21:57.57

Test Run Failed.
Test Run Failed.

Exit code 1.
command:
dotnet test /mnt/vss/_work/2/s/src/project.sln --filter "(TestCategory=SomeCategory)" --settings "/mnt/vss/_work/2/s/src/test.runsettings" --logger "trx" --no-build -c "Debug" -v "Detailed" --results-directory "/mnt/vss/_work/_temp" -- "TestRunParameters.Parameter(name=\"name\", value=\"value\")" 
   at dotnet.test.rerun.DotNetRunner.DotNetTestRunner.HandleProcessEnd() in /home/runner/work/dotnet-test-rerun/dotnet-test-rerun/src/DotNetRunner/DotNetTestRunner.cs:line 83
   at dotnet.test.rerun.DotNetRunner.DotNetTestRunner.Run(String arguments) in /home/runner/work/dotnet-test-rerun/dotnet-test-rerun/src/DotNetRunner/DotNetTestRunner.cs:line 58
   at dotnet.test.rerun.DotNetRunner.DotNetTestRunner.Test(RerunCommandConfiguration config, String resultsDirectory) in /home/runner/work/dotnet-test-rerun/dotnet-test-rerun/src/DotNetRunner/DotNetTestRunner.cs:line 38
   at dotnet.test.rerun.RerunCommand.RerunCommand.Run() in /home/runner/work/dotnet-test-rerun/dotnet-test-rerun/src/RerunCommand/RerunCommand.cs:line 51
   at dotnet.test.rerun.RerunCommand.RerunCommand.<>c__DisplayClass6_0.<<-ctor>b__0>d.MoveNext() in /home/runner/work/dotnet-test-rerun/dotnet-test-rerun/src/RerunCommand/RerunCommand.cs:line 42
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()
##[error]Bash exited with code '1'.
Finishing: Run functional tests with retries

[ricardofslp]

Hi @dgkanatsios ,

Could you please run the tool with loglevel Verbose to see if we can get some more logs to understand what may be happening?

[dgkanatsios]

thanks @ricardofslp for the response!

I modified the bash script to this and it works:

 - task: Bash@3
      continueOnError: true
      displayName: 'Run functional tests with retries'
      inputs:
        failOnStderr: false
        targetType: 'inline'
        script: |
          # https://github.com/joaoopereira/dotnet-test-rerun
          dotnet tool install --global dotnet-test-rerun --version 1.8.0 --no-cache

          project="$(Build.SourcesDirectory)/src/project.sln"
          filter="(TestCategory=SomeCategory)"

          # build the test solution
          dotnet build $project

          # run the tests
          test-rerun $project --loglevel Verbose --no-build --results-directory $(Agent.TempDirectory) --settings "$(Build.SourcesDirectory)/src/test.runsettings" --logger trx --filter=$filter --inlineRunSettings '"TestRunParameters.Parameter(name=\"name\", value=\"value\")"'

I removed the verbosity from the dotnet test tool, maybe this is the culprit? Re-running it with changes now, will post back once I have a concrete answer. Thanks again for the great library!

[dgkanatsios]

Not sure what's going on, but as soon as I add --verbosity detailed the test doesn't do retries and the output is different.

FWIW, I am running the tool with a .sln file that references .csproj files from various test projects, not sure if/how this makes a difference.

Without --verbosity detailed the output is what comes from the tool (this is --verbosity normal)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Results File: /mnt/vss/_work/_temp/_7550dd7ec000001_2023-12-05_05_08_28.trx

Passed!  - Failed:     0, Passed:     5, Skipped:     0, Total:     5, Duration: 8 s

Test retries are done successfully here!

With --verbosity detailed the output is similar to running dotnet test

Total tests: 287
Test Run Failed.
     Passed: 248
     Failed: 36
    Skipped: 3
 Total time: 22.2292 Minutes
     6>VSTest:
         MSB4181: The "Microsoft.TestPlatform.Build.Tasks.VSTestTask" task returned false but did not log an error.
       Done executing task "Microsoft.TestPlatform.Build.Tasks.VSTestTask" -- FAILED.
     6>Done building target "VSTest" in project "test.csproj" -- FAILED.
     6>Done Building Project "/mnt/vss/_work/2/s/src/test.csproj" (VSTest target(s)) -- FAILED.
     1>Done executing task "MSBuild" -- FAILED.
     1>Done building target "VSTest" in project "test.sln" -- FAILED.
     1>Done Building Project "/mnt/vss/_work/2/s/test.sln" (VSTest target(s)) -- FAILED.

Build FAILED.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:22:17.63

No test retries with this verbosity!

[ricardofslp]

Hi @dgkanatsios,

Regarding the way you are running should not make a difference. It seems the issue could be related to the verbosity level.

If you run with other verbosity levels it also fails?

[dgkanatsios]

hey @ricardofslp just edited my post to make it more visible, apologies for the confusion.

With --verbosity detailed, tool doesn't do retries. With --verbosity normal, retries work just fine. Not sure what's up with other verbosity levels.

[ricardofslp]

Hi @dgkanatsios ,

Can you please test again with the latest version (https://www.nuget.org/packages/dotnet-test-rerun/2.0.1)?