Code Coverage: exclusions in .runsettings file are ignored

berndku commented 6 years ago

Description

after upgrading SonarQube Server from 5.6 to 6.7.4.38452 and C# plugin to 7.2 (build 5463), Code Coverage is still imported from VSTS build, but provided exclusions in .runsettings file for VSTest are ignored. With SonarQube 5.6 the .runsettings file exclusions were correctly applied.

Repro steps

VSTS SonarQube Build tasks used. VSTest task has Code Coverage enabled and a .runsettings file specified in the property "Settings file".

Expected behavior

Code Coverage in SonarQube should recognize exlusions from VSTest .runsettings file.

Actual behavior

Code Coverage is imported into SonarQube, but exclusions are not recognized.

Known workarounds

Related information

SonarC# Version : 7.2
Visual Studio Version : 15.7.3
Scanner for MSBuild: VSTS Tasks Version 3

rliberoff commented 6 years ago

We are having exactly the same issue.

Please fix it!!!!!!

millusdk commented 5 years ago

Are there any updates on this? We are experiencing the same issue in 6.7

ghost commented 5 years ago

Has anyone looked into where this change would need to be made? I would like to see if I can get this fixed/added, but I can't seem to locate where.

Evangelink commented 5 years ago

Hi there,

I finally managed to spend some time looking at this issue.

Starting with the version 6.6 of SonarC#, we are now processing a new metric called "executable lines of code" which is used in addition to the coverage info to track lines covered/uncovered. As we don't process (and we never did) the .runsettings file, all files excluded in this configuration won't have any coverage info BUT using the executable lines of code will appear as un-covered on SonarQube.

As a short-term answer, you will need to provide a new parameter (sonar.coverage.exclusion=LIST_OF_FILES) to the scanner for msbuild (or the prepare task of TFS/VSTS) OR configure your project on the SonarQube/SonarCloud project administration page to exclude all files you have excluded in your .runsettings file. Note that you can only exclude a file or a group of files BUT not something with a lower granularity.

As a mid-term/long-term solution, we will investigate whether we should automatically exclude all files excluded in the .runsettings file.

puddlewitt commented 5 years ago

Closed?

andrei-epure-sonarsource commented 5 years ago

Before moving forward with this, we'll have to check the impact of recently handling of .gitignore, and then come back with a plan

joshardt commented 4 years ago

@andrei-epure-sonarsource How far is the plan?

StingyJack commented 3 years ago

@andrei-epure-sonarsource - Would this apply to SonarCloud as well?

andrei-epure-sonarsource commented 3 years ago

I'm coming back on this topic after a long pause (sorry for that!).

@berndku @rliberoff @millusdk @puddlewitt @JFMG @StingyJack For clarification of your usecase(s): are you looking for excluding projects / files just for code coverage or from analysis in general? In other words:

would your expectation be for us to read the runsettings.xml file and exclude those files completely from analysis (like the behavior @Evangelink mentioned here and in the docs, but out of the box based on what's excluded in the runsettings file)
or would you expect these files to be analyzed by our analyzers, but excluded only from code coverage?

Thank you 🙏

Would this apply to SonarCloud as well?

@StingyJack - if this were implemented, yes, it would apply for both SonarQube and SonarCloud.

berndku commented 3 years ago

Analyzed, but excluded from code coverage.

StingyJack commented 3 years ago

Just excluded from code coverage analysis, which is what that runsettings file declares for the VS enterprise code coverage tooling.

joshardt commented 3 years ago

I agree with the previous comments. The runsettings file should just do what it says in the official documentation https://docs.microsoft.com/en-US/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file That means just exclude everything what is described in the runsettings file from code coverage.

StingyJack commented 3 years ago

At the bottom of that docs page @JFMG shared is a link to this page that details the types that can be included or excluded for coverage analysis.

In the past I've needed three of those filters. ModulePath - to either whitelist the assemblies to check or to exclude unit tests from being analyzed for coverage Attribute - to filter out the ones mentioned in the example runsettings Source - to exclude code gen files by extension (*.g.cs)

andrei-epure-sonarsource commented 3 years ago

Thanks for the replies, @berndku , @JFMG , @StingyJack.

I'm thinking out loud: if we were to implement a partial solution (we like to deliver value incrementally) , for example starting with the Source filter only:

would you find it more confusing that the Source filter is taken into consideration, and the ModulePath and Attribute (and others) are not?
would you check the build logs in case the behavior from SonarQube/SonarCloud is different than what you'd expect (we'd mention in the build logs that only filter X is supported, and the rest are ignored)?

Another question: in my understanding, there would be only one .runsettings file per analysis, right?

mack0196 commented 2 years ago

What is the status of respecting the runsettings in sonar analysis?

andrei-epure-sonarsource commented 2 years ago

Hi @mack0196 - this ticket is still open. I've asked a few questions here which I didn't get reply to.

We don't have this on our near-future roadmap.

(cc @tom-howlett-sonarsource )

StingyJack commented 2 years ago

@andrei-epure-sonarsource to answer your questions (didnt see them, sorry)

The Attribute filter and module path are usually the thing I would use the most, because historically I've had to exclude unit test projects themselves from having coverage reported on them (which is silly but true), and add attributes like for generated code (and exclude from code coverage). The Source is only when I need to exclude a generated file that doesnt use attributes to denote that its generated (like grpc.net used to but that should be fixed now).

I would expect there to be logs somewhere about what was included or not, at least when diagnostics or system.debug pipeline variable was enabled.

If we are doing it right, there is only one runsettings for all the projects in the solution (and repo). There shouldnt be a great need to test projects in the same repo differently that couldnt be handled by accommodating for it tin the runsettings, but there may be a case for it somewhere.

mack0196 commented 12 months ago

My team and I are still interested if you could have sonar respect the exclusions in our runsettings file.

wilkovanderveen commented 8 months ago

Any other options than waiting for this change?

nathfy commented 8 months ago

@andrei-epure-sonarsource Any update on when this will be picked up by sonar? At the moment we need to add exclusion to both CodeCoverage.runsettings and in sonar cloud > General settings > B. Code Coverage Exclusions

meyerjg commented 5 months ago

Having the same issue/behavior, would like to see an update for this

almolo1 commented 1 month ago

This is still relevant in 2024, .runsettings file should be respected

pavel-mikula-sonarsource commented 1 month ago

cc @denis-troller to raise awareness

SonarSource / sonar-dotnet