Test runner is failing to publish the results due to unknown test runner

[chadrickman]

This is running on the newest updates from this morning.

Pool: Azure Pipelines Image: macOS-10.14 Agent: Hosted Agent

Async Command Start: Publish test results
Async Command End: Publish test results
##[error]Unknown Test Runner.
##[debug]System.ArgumentException: Unknown Test Runner.
   at Microsoft.VisualStudio.Services.Agent.Worker.LegacyTestResults.LegacyTestRunDataPublisher.GetTestResultReader(String testRunner, Boolean publishRunLevelAttachments)
   at Microsoft.VisualStudio.Services.Agent.Worker.LegacyTestResults.LegacyTestRunDataPublisher.InitializePublisher(IExecutionContext context, String projectName, VssConnection connection, String testRunner, Boolean publishRunLevelAttachments)
   at Microsoft.VisualStudio.Services.Agent.Worker.TestResults.PublishTestResultsCommand.PublishTestRunDataAsync(String teamProject, TestRunContext testRunContext)
   at Microsoft.VisualStudio.Services.Agent.Worker.AsyncCommandContext.WaitAsync()
   at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
Finishing: Run tests

[aloisdeniel]

Thanks for reporting.

I switched back to JUnit test publisher in a new published version, but since the Devops package API have changed and isn't documented at all ... I hope it is still working.

Let me know!

[chadrickman]

@aloisdeniel This does appear to still be an issue. I'll run a build later tonight that has more logging to get more details. It's the same error though.

[ghost]

Still an issue here, same error as described above.

[skela]

Same here

[alexaniko88]

What is the status of this? We have the same problem here, all tests are passed but build fails because of Unknown Test Runner issue

[davidfranquet]

Same here

[jasperamorgan]

As a work around, you can run the tests without publishing the results. To do this, comment out the Flutter test task and replace with this:

- task: CmdLine@2
        displayName: 'Flutter Test'
        inputs:
          script: '$(FlutterToolPath)/flutter test'
          workingDirectory: '$(Build.SourcesDirectory)'

##instead of##
# - task: aloisdeniel.flutter.flutter-test.FlutterTest@0
        # displayName: 'Flutter Test'
        # inputs:
          # projectDirectory: '$(Build.SourcesDirectory)'

[skela]

Is there a similar workaround, that works without having to modify the yaml? We can't use yml directly in devops, because we have an external git, only github, bitbucket cloud and devops git supports yml to configure pipelines for some annoying reason :(

[ezzabuzaid]

Same here, any updates

[andre-gravato]

I'm also getting this:

[error]Unable to process command '##vso[results.publish type=flutter;mergeResults=false;publishRunAttachments=true;resultFiles=C:\azureagentx64_work\2\s\junit.xml;testRunSystem=Flutter;]' successfully. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=817296)

[error]Unknown Test Runner.

[skela]

:facepalm: in case anyone cannot modify the yml to match what jaspermorgans workaround. It does work, just take notice that its not the same task, basically just running the unit test via command line. :facepalm:

[hey24sheep]

I have added a check for publish results in my package.

[josepmg]

I have added a check for publish results in my package.

@hey24sheep could you give an example of how you did that? I'm a Azure DevOps Pipelines beginner hehe

[hey24sheep]

@josepmg I have highlighted the checkbox also the add button. Read this to get start easily. After that you could add "flutter tasks extension" and add "flutter test task". Check/Uncheck publish checkbox according to your need.

[josepmg]

@josepmg I have highlighted the checkbox also the add button. Read this to get start easily. After that you could add "flutter tasks extension" and add "flutter test task". Check/Uncheck publish checkbox according to your need.

Thanks, @hey24sheep I'll try it right away

[Tregan]

Temporary workaround (which also immediately improves the test report in the process) for me is to replace the Flutter Test task with these 3 tasks:

• PowerShell task that adds Flutter folders to PATH

  Write-Host "##vso[task.setvariable variable=PATH;]${env:PATH};$(FLUTTERTOOLPATH);$(FLUTTERTOOLPATH)\cache\dart-sdk\bin;$(FLUTTERTOOLPATH)\..\.pub-cache\bin";

• Followed by a cmd task that activates junitreport and generates the test report

  flutter pub get && flutter pub global activate junitreport && flutter test --machine | tojunit --output TEST-report.xml

• Publish Test Results task with search folder $(System.DefaultWorkingDirectory)

Even better: once https://github.com/flutter/flutter/issues/56675 is fixed it will even be possible to generate both a test report and coverage data at the same time by using --machine --coverage

This will especially be useful since I'm once again having issues making generation of the coverage data reliable, so I closed the PR for that.

[rspilker]

junitreport version 1.3, that has just been released, is capable of handling the output of flutter test --machine --coverage; it ignores the malformatted lines, generated by the extra --coverage.