search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.5k stars 2.61k forks source link

[BUG]: PublishCodeCoverageResultsV2 fails due to NullReferenceException. #18026

Open EMI0101 opened 1 year ago

EMI0101 commented 1 year ago

Task name

PublishCodeCoverageResultsV2

Task version

2.218.0

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows 11

Task log

Starting: PublishCodeCoverageResults
==============================================================================
Task         : Publish code coverage results v2
Description  : Publish any of the code coverage results from a build
Version      : 2.218.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/publish-code-coverage-results
==============================================================================
##[debug]Using node path: C:\Agent\externals\node16\bin\node.exe
##[debug]agent.TempDirectory=C:\Agent\_work\_temp
##[debug]check path : C:\Agent\_work\_tasks\PublishCodeCoverageResults_2a7ebc54-c13e-490e-81a5-d7561ab7cd97\2.218.0\task.json
##[debug]adding resource file: C:\Agent\_work\_tasks\PublishCodeCoverageResults_2a7ebc54-c13e-490e-81a5-d7561ab7cd97\2.218.0\task.json
##[debug]system.culture=en-US
##[debug]summaryFileLocation=C:\Agent\_work\1\s\Results/s-opencov/EditMode/TestCoverageResults*.xml
##[debug]failIfCoverageEmpty=false
##[debug]System.DefaultWorkingDirectory=C:\Agent\_work\1\s
##[debug]pathToSources=undefined
##[debug]defaultRoot: 'C:\Agent\_work\1\s'
##[debug]findOptions.allowBrokenSymbolicLinks: 'false'
##[debug]findOptions.followSpecifiedSymbolicLink: 'false'
##[debug]findOptions.followSymbolicLinks: 'false'
##[debug]findOptions.skipMissingFiles: 'undefined'
##[debug]matchOptions.debug: 'false'
##[debug]matchOptions.nobrace: 'true'
##[debug]matchOptions.noglobstar: 'false'
##[debug]matchOptions.dot: 'true'
##[debug]matchOptions.noext: 'false'
##[debug]matchOptions.nocase: 'true'
##[debug]matchOptions.nonull: 'false'
##[debug]matchOptions.matchBase: 'false'
##[debug]matchOptions.nocomment: 'false'
##[debug]matchOptions.nonegate: 'false'
##[debug]matchOptions.flipNegate: 'false'
##[debug]pattern: 'C:\Agent\_work\1\s\Results/s-opencov/EditMode/TestCoverageResults*.xml'
##[debug]findPath: 'C:\Agent\_work\1\s\Results\s-opencov\EditMode'
##[debug]statOnly: 'false'
##[debug]findPath: 'C:\Agent\_work\1\s\Results\s-opencov\EditMode'
##[debug]findOptions.allowBrokenSymbolicLinks: 'false'
##[debug]findOptions.followSpecifiedSymbolicLink: 'false'
##[debug]findOptions.followSymbolicLinks: 'false'
##[debug]findOptions.skipMissingFiles: 'undefined'
##[debug]  C:\Agent\_work\1\s\Results\s-opencov\EditMode (directory)
##[debug]  C:\Agent\_work\1\s\Results\s-opencov\EditMode\TestCoverageResults_0000.xml (file)
##[debug]2 results
##[debug]found 2 paths
##[debug]applying include pattern
##[debug]1 matches
##[debug]1 final results
Found 1 result(s) matching pattern: C:\Agent\_work\1\s\Results/s-opencov/EditMode/TestCoverageResults*.xml
C:\Agent\_work\1\s\Results\s-opencov\EditMode\TestCoverageResults_0000.xml
##[debug]Agent.Version=2.218.1
##[debug]Agent.TempDirectory=C:\Agent\_work\_temp
##[debug]which 'dotnet'
##[debug]found: 'C:\Program Files\dotnet\dotnet.exe'
##[debug]which 'C:\Program Files\dotnet\dotnet.exe'
##[debug]found: 'C:\Program Files\dotnet\dotnet.exe'
##[debug]C:\Program Files\dotnet\dotnet.exe arg: C:\Agent\_work\_tasks\PublishCodeCoverageResults_2a7ebc54-c13e-490e-81a5-d7561ab7cd97\2.218.0\node_modules\coveragepublisher\CoveragePublisher\CoveragePublisher.Console.dll
##[debug]C:\Program Files\dotnet\dotnet.exe arg: "C:\Agent\_work\1\s\Results\s-opencov\EditMode\TestCoverageResults_0000.xml"
##[debug]C:\Program Files\dotnet\dotnet.exe arg: --reportDirectory C:\Agent\_work\_temp\4d9e285e-62e1-474b-a827-ccd9093f6082
##[debug]SystemVssConnection auth param AccessToken = ***
##[debug]System.TeamFoundationCollectionUri=[REDACTED]
##[debug]Build.BuildId=5968
##[debug]Build.ContainerId=26176260
##[debug]Agent.TempPath=undefined
##[debug]System.TeamProjectId=[REDACTED]
##[debug]PIPELINES_COVERAGEPUBLISHER_DEBUG=undefined
##[debug]exec tool: C:\Program Files\dotnet\dotnet.exe
##[debug]arguments:
##[debug]   C:\Agent\_work\_tasks\PublishCodeCoverageResults_2a7ebc54-c13e-490e-81a5-d7561ab7cd97\2.218.0\node_modules\coveragepublisher\CoveragePublisher\CoveragePublisher.Console.dll
##[debug]   "C:\Agent\_work\1\s\Results\s-opencov\EditMode\TestCoverageResults_0000.xml"
##[debug]   --reportDirectory C:\Agent\_work\_temp\4d9e285e-62e1-474b-a827-ccd9093f6082
"C:\Program Files\dotnet\dotnet.exe" C:\Agent\_work\_tasks\PublishCodeCoverageResults_2a7ebc54-c13e-490e-81a5-d7561ab7cd97\2.218.0\node_modules\coveragepublisher\CoveragePublisher\CoveragePublisher.Console.dll "C:\Agent\_work\1\s\Results\s-opencov\EditMode\TestCoverageResults_0000.xml" --reportDirectory C:\Agent\_work\_temp\4d9e285e-62e1-474b-a827-ccd9093f6082
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.DotNet.Cli.Utils.EnvironmentProvider.get_ExecutableExtensions()
   at Microsoft.DotNet.Cli.Utils.EnvironmentProvider.GetCommandPathFromRootPath(String rootPath, String commandName, String[] extensions)
   at Microsoft.DotNet.Cli.Utils.AppBaseCommandResolver.ResolveCommandPath(CommandResolverArguments commandResolverArguments)
   at Microsoft.DotNet.Cli.Utils.AbstractPathBasedCommandResolver.Resolve(CommandResolverArguments commandResolverArguments)
   at Microsoft.DotNet.Cli.Utils.CompositeCommandResolver.Resolve(CommandResolverArguments commandResolverArguments)
   at Microsoft.DotNet.Cli.Utils.CommandResolver.TryResolveCommandSpec(ICommandResolverPolicy commandResolverPolicy, String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration, String outputPath, String applicationName)
   at Microsoft.DotNet.Cli.Utils.Command.Create(ICommandResolverPolicy commandResolverPolicy, String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration, String outputPath, String applicationName)
   at Microsoft.DotNet.Cli.Utils.Command.Create(String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration, String outputPath, String applicationName)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)

##[debug]task result: Failed
##[debug]Processed: ##vso[task.complete result=Failed;]
##[debug]Exit code 1 received from tool 'C:\Program Files\dotnet\dotnet.exe'
##[debug]STDIO streams have closed for tool 'C:\Program Files\dotnet\dotnet.exe'
Finishing: PublishCodeCoverageResults

Relevant log output

System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.DotNet.Cli.Utils.EnvironmentProvider.get_ExecutableExtensions()
   at Microsoft.DotNet.Cli.Utils.EnvironmentProvider.GetCommandPathFromRootPath(String rootPath, String commandName, String[] extensions)
   at Microsoft.DotNet.Cli.Utils.AppBaseCommandResolver.ResolveCommandPath(CommandResolverArguments commandResolverArguments)
   at Microsoft.DotNet.Cli.Utils.AbstractPathBasedCommandResolver.Resolve(CommandResolverArguments commandResolverArguments)
   at Microsoft.DotNet.Cli.Utils.CompositeCommandResolver.Resolve(CommandResolverArguments commandResolverArguments)
   at Microsoft.DotNet.Cli.Utils.CommandResolver.TryResolveCommandSpec(ICommandResolverPolicy commandResolverPolicy, String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration, String outputPath, String applicationName)
   at Microsoft.DotNet.Cli.Utils.Command.Create(ICommandResolverPolicy commandResolverPolicy, String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration, String outputPath, String applicationName)
   at Microsoft.DotNet.Cli.Utils.Command.Create(String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration, String outputPath, String applicationName)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)

Aditional info

Some context:
- Using .Net Core 2.1.818
- Its a unity project, the pipeline runs tests and the unity editor exports a coverage report.
- The task worked flawlessly for about 1-2 months.
- Tried updating the agents, but did not fix issue.
MisinformedDNA commented 1 year ago

I'm trying to switch to V2 and have this same issue. It looks like it is trying to access a directory that doesn't exist.

--reportDirectory was a parameter in V1 but does not exist in V2. The task is adding the --reportDirectory parameter to a directory that doesn't exist.

MisinformedDNA commented 1 year ago

Looks like a dup of #17756.

EMI0101 commented 1 year ago

Agents on 3.218.0 and switched them to .net 7 identical issue persists.

How is this not fixed, Azure just doesn't have working coverage for months ?

MisinformedDNA commented 1 year ago

Everyone still uses the old version? Baffling.

davayd commented 1 year ago

Same issue

Starting: PublishCodeCoverageResults
==============================================================================
Task         : Publish code coverage results v2
Description  : Publish any of the code coverage results from a build
Version      : 2.218.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/publish-code-coverage-results
==============================================================================
Found 1 result(s) matching pattern: **/code-coverage.xml
D:\a\1\s\AgilityPlanning.UI\ClientApp\testresults\coverage\code-coverage.xml
"C:\Program Files\dotnet\dotnet.exe" D:\a\_tasks\PublishCodeCoverageResults_2a7ebc54-c13e-490e-81a5-d7561ab7cd97\2.218.0\node_modules\coveragepublisher\CoveragePublisher\CoveragePublisher.Console.dll "D:\a\1\s\AgilityPlanning.UI\ClientApp\testresults\coverage\code-coverage.xml" --reportDirectory D:\a\_temp\87bea6ad-fe11-43e3-bd08-bd55bbde32bd

Welcome to .NET 7.0!
---------------------
SDK Version: 7.0.203

Telemetry
---------
The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.

Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.DotNet.Cli.Utils.EnvironmentProvider.get_ExecutableExtensions()
   at Microsoft.DotNet.Cli.Utils.EnvironmentProvider.GetCommandPathFromRootPath(String rootPath, String commandName, String[] extensions)
   at Microsoft.DotNet.CommandFactory.AbstractPathBasedCommandResolver.Resolve(CommandResolverArguments commandResolverArguments)
   at Microsoft.DotNet.CommandFactory.CompositeCommandResolver.Resolve(CommandResolverArguments commandResolverArguments)
   at Microsoft.DotNet.CommandFactory.CommandFactoryUsingResolver.Create(ICommandResolverPolicy commandResolverPolicy, String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration, String outputPath, String applicationName)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)

Finishing: PublishCodeCoverageResults
EMI0101 commented 1 year ago

Still broken.

jbatmis commented 1 year ago

I am experiencing this same issue.

Using V2 of the publish code coverage results task results in the following:

Starting: Publish code coverage results
==============================================================================
Task         : Publish code coverage results v2
Description  : Publish any of the code coverage results from a build
Version      : 2.221.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/publish-code-coverage-results
==============================================================================
Found 1 result(s) matching pattern: MyPath\coverage\*.xml
C:\agent\_work\3\s\MayPath\coverage\cobertura-coverage.xml
"C:\Program Files\dotnet\dotnet.exe" C:\agent\_work\_tasks\PublishCodeCoverageResults_2a7ebc54-c13e-490e-81a5-d7561ab7cd97\2.221.0\node_modules\coveragepublisher\CoveragePublisher\CoveragePublisher.Console.dll "C:\agent\_work\3\s\ActiveH\ActiveH Web\ActiveHWeb.Angular\coverage\cobertura-coverage.xml" --reportDirectory C:\agent\_work\_temp\4f772f7e-0ea6-4942-afc0-654650be06bd --sourceDirectory C:\agent\_work\3\s\ActiveH\ActiveH Web\ActiveHWeb.Angular
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.DotNet.Cli.Utils.EnvironmentProvider.get_ExecutableExtensions()
   at Microsoft.DotNet.Cli.Utils.EnvironmentProvider.GetCommandPathFromRootPath(String rootPath, String commandName, String[] extensions)
   at Microsoft.DotNet.CommandFactory.AppBaseCommandResolver.ResolveCommandPath(CommandResolverArguments commandResolverArguments)
   at Microsoft.DotNet.CommandFactory.AbstractPathBasedCommandResolver.Resolve(CommandResolverArguments commandResolverArguments)
   at Microsoft.DotNet.CommandFactory.CompositeCommandResolver.Resolve(CommandResolverArguments commandResolverArguments)
   at Microsoft.DotNet.CommandFactory.CommandResolver.TryResolveCommandSpec(ICommandResolverPolicy commandResolverPolicy, String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration, String outputPath, String applicationName)
   at Microsoft.DotNet.CommandFactory.CommandFactoryUsingResolver.Create(ICommandResolverPolicy commandResolverPolicy, String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration, String outputPath, String applicationName)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)

Finishing: Publish code coverage results

This is in a classic build pipeline (has to be classic, as we are still using TFVC) trying to publish cobertura coverage results produced by a previous npm (karma) step in the pipeline.

Running Azure DevOps Services with a locally hosted agent.

bgp1 commented 1 year ago

We also experience this issue with our builds with a similar signature:

Found 1 result(s) matching pattern: **/coverage/cobertura-coverage.xml
C:\agent\_work\7\s\coverage\cobertura-coverage.xml
"C:\Program Files\dotnet\dotnet.exe" C:\agent\_work\_tasks\PublishCodeCoverageResults_2a7ebc54-c13e-490e-81a5-d7561ab7cd97\2.221.0\node_modules\coveragepublisher\CoveragePublisher\CoveragePublisher.Console.dll "C:\agent\_work\7\s\coverage\cobertura-coverage.xml" --reportDirectory C:\agent\_work\_temp\1b094c04-0148-4f7d-a08f-1a9fed482402
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.DotNet.Cli.Utils.EnvironmentProvider.get_ExecutableExtensions()
   at Microsoft.DotNet.Cli.Utils.EnvironmentProvider.GetCommandPathFromRootPath(String rootPath, String commandName, String[] extensions)
   at Microsoft.DotNet.CommandFactory.AbstractPathBasedCommandResolver.Resolve(CommandResolverArguments commandResolverArguments)
   at Microsoft.DotNet.CommandFactory.CompositeCommandResolver.Resolve(CommandResolverArguments commandResolverArguments)
   at Microsoft.DotNet.CommandFactory.CommandFactoryUsingResolver.Create(ICommandResolverPolicy commandResolverPolicy, String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration, String outputPath, String applicationName)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)
jbatmis commented 1 year ago

I've found a resolution (or a workaround depending on your point of view) to our issue:

The V2 of the publish code coverage results task (classic pipeline) has two path fields:

1) Path to summary files (summaryFileLocation) 2) Path to source files (pathToSources)

If the path to source files is left blank, then the System.NullReferenceException occurs. If the path to source files is explicitly specified, then the code coverage results are published correctly. In our case it is now set to:

$(System.DefaultWorkingDirectory)\MyPath

It appears that the task is not able to interpret the following root path from the cobertura-coverage.xml file correctly:

  <sources>
    <source>C:\agent\_work\3\s\MyPath</source>
  </sources>

To be fair, the info provided for this field does clearly state:

"Path to source files is required when coverage XML reports do not contain absolute path to source files. For e.g., JaCoCo reports do not use absolute paths and when publishing JaCoCo coverage for Java apps, the pattern would be similar to $(System.DefaultWorkingDirectory)/MyApp/src/main/java/"

Mentions nothing about cobertura though...

The proper fix for MS would be to either support the element and relative paths correctly, or at least provide a proper error message.

One other word of warning: The Path to source files cannot contain a space - this throws things out completely. We have to rename the problem directory first, publish the results, and then rename back. Other 'logical' workarounds like creating a symbolic link or using the 8.3 path name don't work.

github-actions[bot] commented 9 months ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

EMI0101 commented 9 months ago

Not fixed, using jbatmis workaround.