search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.47k stars 2.6k forks source link

PublishTestResults@2 fails to publish test results to Azure DevOps Server 2020 #13791

Closed ljani closed 3 years ago

ljani commented 3 years ago

Required Information

Question, Bug, or Feature? Bug Type: PublishTestResults@2 fails to publish test results to Azure DevOps Server 2020 Enter Task Name: PublishTestResults

Environment

Issue Description

I have the following task:

    - task: PublishTestResults@2
      displayName: 'Publish test results'
      condition: succeededOrFailed()
      inputs:
        testResultsFormat: 'JUnit'
        testResultsFiles: '$(ui_directory)junit.xml'
        failTaskOnFailedTests: true
        mergeTestResults: true
        testRunTitle: 'UI tests'

junit.xml is generated by jest-junit.

This used to work with Azure DevOps Server 2019.

Task logs

Important lines:

##[warning]Failed to publish test results: One or more errors occurred.
API resource location 364538f9-8062-4ce0-b024-75a0fb463f0d is not registered on https://tfs.mycompany.com/Collection/..

All:

    ##[debug]testRunSystem: VSTS - PTR
    ##[debug]pullrequesttargetbranch: 
    ##[debug]stageName: Build
    ##[debug]stageAttempt: 1
    ##[debug]phaseName: UI
    ##[debug]phaseAttempt: 1
    ##[debug]jobName: __default
    ##[debug]jobAttempt: 1
    ##[debug]Object of JUnitResultParser created.
    ##[debug]Entering ParseTestResultFiles
    ##[debug]runContext.Platform is null
    ##[debug]runContext.Configuration is null
    ##[debug]runContext.ReleaseURI is null
    ##[debug]runContext.ReleaseEnvironmentUri is null
    ##[debug]Reading test results from file 'C:\opt\azure-devops-agent\_work\20\s\oee-ui\junit.xml'.
    ##[debug]Total test results: 24.
    ##[debug]Leaving ParseTestResultFiles
    ##[debug]TestManagement.Server.ParserAndPublisherTelemetryEnabledFF feature flag is off
    No build level attachments to publish.
    ##[debug]Object of TestLogStore created.
    Result Attachments will be stored in FileService
    ##[debug]Object of TestResultsPublisher created.
    ##[debug]Object of TestLogStore created.
    Run Attachments will be stored in FileService
    ##[debug]Object of TestRunPublisher created.
    ##[debug]runContext.Platform is null
    ##[debug]runContext.Configuration is null
    ##[debug]runContext.ReleaseURI is null
    ##[debug]runContext.ReleaseEnvironmentUri is null
    ##[debug]Entering PublishAllTestResultsToSingleTestRunAsync
    ##[debug]Entering CreateTestRunAsync
    ##[debug]Leaving CreateTestRunAsync
    ##[warning]Failed to publish test results: One or more errors occurred.
    ##[debug]Processed: ##vso[task.logissue type=warning;]Failed to publish test results: One or more errors occurred.
    API resource location 364538f9-8062-4ce0-b024-75a0fb463f0d is not registered on https://tfs.mycompany.com/Collection/..
    ##[debug]Leaving PublishAllTestResultsToSingleTestRunAsync
    ##[debug]TestManagement.PTR.CalculateTestRunSummary feature flag is on
    ##[debug]failTaskOnFailedTests: true
    ##[debug]Stored test run summary {"Total":24,"Failed":0,"Passed":24,"Skipped":0}, in PTR_TEST_RUNSUMMARY.json
    ##[debug]Finished: Execute TestResultsPublisher.exe
    ##[debug]Exit code 0 received from tool 'C:\opt\azure-devops-agent\_work\_tasks\PublishTestResults_0b0f01ed-7dde-43ff-9cbb-e48954daf9b1\2.170.0\modules\TestResultsPublisher.exe'
    ##[debug]STDIO streams have closed for tool 'C:\opt\azure-devops-agent\_work\_tasks\PublishTestResults_0b0f01ed-7dde-43ff-9cbb-e48954daf9b1\2.170.0\modules\TestResultsPublisher.exe'
    ##[debug]Exit code of TestResultsPublisher: 0
    ##[debug]Agent.Version=2.170.1
    ##[debug]Agent.TempDirectory=C:\opt\azure-devops-agent\_work\_temp
    ##[debug]RESOURCE_URIS:
    ##[debug]Setting task variable METADATA_e8643cc6-2db3-4f65-a138-0443fbd0dee5: {"name":"7b9dd092-3742-455e-8407-a5237ce90539","resourceUris":[],"metadata":{"description":"Test Results published from Publish Test Results tool","relatedUrls":[{"url":"https://tfs.mycompany.com/Collection/Project/_build/results?buildId=2129","label":"pipeline-url"}],"humanReadableName":"Test Results from Publish Test Results utility","serializedPayload":"{\"testId\":\"PublishTestResults\",\"testTool\":\"JUnit\",\"testResultAttestation\":{\"total\":24,\"failed\":0,\"passed\":24,\"skipped\":0},\"testDurationSeconds\":0.0,\"testPassPercentage\":\"100\",\"relatedUrls\":[{\"url\":\"https://tfs.mycompany.com/Collection/Project/_build/results?buildId=2129\",\"label\":\"pipeline-url\"}]}"}} 
    ##[debug]Processed: ##vso[results.publishtoevidencestore name=PublishTestResults;testrunner=JUnit;testrunsummary={"Total":24,"Failed":0,"Passed":24,"Skipped":0};description=Test Results published from Publish Test Results tool;]
    ##[debug]task result: Succeeded
    ##[debug]Processed: ##vso[task.complete result=Succeeded;]
    ##[debug]Release.ReleaseUri=undefined
    ##[debug]Release.ReleaseId=undefined
    ##[debug]Build.BuildUri=vstfs:///Build/Build/2129
    ##[debug]Build.Buildid=2129
    ##[debug]Processed: ##vso[telemetry.publish area=TestExecution;feature=PublishTestResultsTask]{"builduri":"vstfs:///Build/Build/2129","buildid":"2129","osType":"Windows_NT","testRunner":"JUnit","failTaskOnFailedTests":"true","mergeResultsUserPreference":"true","config":null,"platform":null,"testResultsFilesCount":1,"dotnetVersion":"3.1.403","subFeature":"publishTestResultsTaskConsolidatedCiEvent"}
    Async Command Start: PublishTestResultsToEvidenceStore
    Async Command End: PublishTestResultsToEvidenceStore
    Finishing: Publish test results
ljani commented 3 years ago

Interesting enough, this has fixed itself even if I didn't do anything. Any idea what might have been the problem?

EDIT: Well, I added a test to the project. Could that have fixed the issue?

suyash691 commented 3 years ago

@ljani You need to have test plans active on onprem to get the test service active

ljani commented 3 years ago

Yeah, it's off. Is that requirement mentioned somewhere? Test reporting from CI pipeline worked in version 2019 with "Test Plans" turned off, and now it works in version 2020 after adding a test even if "Test Plans" is turned off.

EDIT: Also "Test Plans" is described "Structured manual testing at any scale for teams of all sizes." Does it really have something to do with automated testing?

suyash691 commented 3 years ago

That is not a requirement but some of the customers facing similar issues said that this fixed their problem so I just suggested that as a workaround. (sorry for the confusion caused)

Also I tried reproducing this issue on a fresh onprem setup without test plans and I was unable to. (PTR task and the respective test APIs worked) I would need more information from you regarding the onprem setup. Was it a fresh onprem install? or was it an upgraded server? If so from which version was the upgrade carried out from?

ljani commented 3 years ago

Was it a fresh onprem install? or was it an upgraded server? If so from which version was the upgrade carried out from?

It was updated in-place from Azure DevOps Server 2019 Update 1.1 Patch 5 to Azure DevOps Server 2020.

But as said, my problem was somehow fixed, feel free to close this issue, if I'm the only one who experienced this.

suyash691 commented 3 years ago

Ok. Thanks for confirming