Warnings statistics report shows warnings as from "unknown file"

[mdwhitten]

I would like to be able to see the individual warnings per file, which seems to be possible based on the screenshot in WarningsPolicy.md. However, all of my warnings appear as "Unknown File":

The log is being published to VSTS using the syntax##vso[task.logissue type=warning;sourcepath=$messageFilePath;]$outputMessage". Hence, the file path is being published as a part of the warning (see below), but doesn't seem that it is being picked up by the task.

I've attached the log for more context: log_21_154154.zip

Thanks!

[mdwhitten]

I recognize that this is probably what is meant by the statement in the documentation:

Note: Statistics currently only work for MSBuild builds (i.e., the Visual Studio Build and MSBuild tasks) and only on Team Foundation Server 2017 or later and Azure DevOps Services. If you need support for additional tasks, please let us know and preferably send us a sample log of the corresponding build task to psgerextsupport@microsoft.com.

However, I had hoped that if I at least formatted my warning in the proper way, it would be picked up and reported in the same way as the MSBuild tasks.

[ReneSchumacher]

Hi Michael,

sorry for letting you down, but we only run the proper analysis on logs created by one of the MSBuild-based tasks (VS Build, MSBuild) or the StyleCop task. Thus, it doesn't matter how you format your log output, the analysis logic will never pick up those log lines.

However, it looks like you're using a standard tool (TestStand from National Instruments) to run some kind of static analysis (I haven't spent too much time skimming through their website). I assume there is no direct support for Azure DevOps from their end, right? I'm wondering if it might be worth investing a bit in creating a "standard" task for running this analysis that properly logs warnings and errors as real build issues (this would get rid of the need of custom warning filters to pick up those warnings) and then adding support for the resulting log file to the Build Quality Checks task. Especially if you'd be willing to share your parsing script and information about how to properly run the analysis. :)

René

[mdwhitten]

Hi @ReneSchumacher,

Thanks for the quick response, and sorry for the delay on my end. Allow me to give some background here:

TestStand is a test sequencing application for engineering test automation workflows. A test sequence will consist of a variety of discrete code modules (possibly in different languages), expressions invoking the TestStand API, and evaluations of results. You rightly understood the report above as something of a static analysis, outputting errors, warnings, and information (as close to compiler warnings as is sensible for something that isn't being compiled). The output of this report is an XML file that, as you guessed, has no direct support for Azure DevOps.

With that background, allow me to clarify a few points:

I'm wondering if it might be worth investing a bit in creating a "standard" task for running this analysis... Especially if you'd be willing to share your parsing script and information about how to properly run the analysis. :)

I agree, which is why I've created a PowerShell script that consumes this XML report and generates the log visible above. I've attached the PowerShell script and an example XML file. You can call it with ParseTSAReport.ps1 -Path "Path_to_XML_File" or ParseTSAReport.ps1 -Path "Path_to_XML_File" -Verbose ParseTSAReport.zip

that properly logs warnings and errors as real build issues (this would get rid of the need of custom warning filters to pick up those warnings) and then adding support for the resulting log file to the Build Quality Checks task.

Because of the above, I am not using any warning filters. Rather, I am using the Azure Pipelines Logging Commands to log the errors and warnings so that they show up as warnings and errors in the build summary in Azure. Is this what you mean by properly log[ing] warnings and errors as real build issues"?

As best as I can tell, MSBuild and other tasks utilize the exact same command syntax to output their build warnings to the console, specifying which file rendered the warning using the sourcepath flag. I am doing the same thing in my logging above. Hence, this is the reason that I was surprised that the task does not pickup the file included with the warnings, since I think I am logging these the proper way.

What are your thoughts? Thanks again for your support!

[gerwinjansen]

I have the same issue. My expectation is:

1. BuildQualityChecks takes the stdout and stderr from the previous steps (up to the first one or the preceding BuildQualityChecks)
2. It looks for task.logissue lines.
3. It parses all these lines and collects the warnings and errors.

The lines contain the context info. Example warning that is written by an arbitrary app: ##vso[task.logissue type=warning;sourcepath=consoleapp/main.cs;linenumber=1;columnnumber=1;code=100;]Found something that could be a problem.

It shows in Azure DevOps job run output as: ##[warning]consoleapp/main.cs(1,1): warning 100: Found something that could be a problem.

What I see is that the warnings are collected, but without parsing the context like sourcepath, linenumber, columnnumber and code.

we only run the proper analysis on logs created by one of the MSBuild-based tasks (VS Build, MSBuild) or the StyleCop task. Thus, it doesn't matter how you format your log output, the analysis logic will never pick up those log lines.

What is 'logs'? Is that the stdout/stderr or separate files? Can you explain how BuildQualityChecks works? Why can it detect warnings-including-sourcepaths for MSBuild-based tasks but only raw warnings for every other task? The raw-warning contains all the information that is needed. And even for MSBuild-based tasks it does not display the linenumber and columnnumber.

[ReneSchumacher]

Hi @gerwinjansen,

Build Quality Checks is not designed to display warnings created by other tasks. It's purpose is to simply count warnings and compare the current count with defined thresholds (either fixed value or the warning count from a previous build) to decide whether or not to break the build.

Every issues (warnings and errors) that is correctly logged by a task with the logging command you mentioned is displayed in the build summary automatically. That is the purpose of those logging commands. Thus, there's no need to repeat that information again which is the reason why we don't display any line or column information.

I believe that you are talking about the warning statistics. The purpose of those is to give a summary of warning changes in specific code files (or arbitrary files when using the warning file evaluation). So instead of listing all details for all warnings in a file, we just show files that have either more or less warnings compared to the previous build.

What are you expecting to see?

Btw: we only run statistics analysis for MSBuild and StyleCop since we parse the actual warnings from the tool itself, not the logging commands. This is because in the past logging commands were not written to the log file so there was no easy way to grab them. We could update our task to run a logging command analysis, but we'll have to think about backwards compatibility.

[ReneSchumacher]

Hi @mdwhitten,

first, I have to apologize that it took us so long to work on your feedback. It was sitting on our backlog but there was always something else to do. Good news is that we were able to create a generic solution now that works for all tasks or scripts that use our logging commands similar to what you do in your PowerShell script. Your files should now properly show up in the statistics section of the extension summary.

As this issue is resolved now I'm going to close it. Nevertheless, please feel free to add additional comments or open new issues if you still have problems getting statistics from your pipeline.

Happy building, René

[mdwhitten]

That's great! Thank you for getting back to me and for creating the generic solution. I will happily use this in my pipelines now.