OpenCover fails on .NET Framework 4.8

DirkFEng commented 5 years ago

Please provide the following information when submitting an issue.

My Framework

[X ] .NET 4.8

My Environment

[X ] Windows Server 2012
[X ] repeated the problem using the latest stable release of OpenCover.
[X] reviewed the usage guide and usage document.
[X] have looked at the opencover output xml file in an attempt to resolve the issue.
[X] reviewed the current issues to check that the issue isn't already known.

My issue is related to (check only those which apply):

[X] no coverage being recorded
[X] feature request

Expected Behavior

Create report

Actual Behavior

Committing... 17:32:42 No results, this could be for a number of reasons. The most common reasons are: 17:32:42 1) missing PDBs for the assemblies that match the filter please review the 17:32:42 output file and refer to the Usage guide (Usage.rtf) about filters. 17:32:42 2) the profiler may not be registered correctly, please refer to the Usage 17:32:42 guide and the -register switch. 17:32:42 Der Befehl ""C:\Jenkins\workspace\MyApp\OpenCover.Console.exe" -register:user -target:"C:\Jenkins\workspace\MyApp\nunit\nunit3-console.exe" -filter:+[]myfilter. -output:"C:\Jenkins\workspace\MyApp\TestCoverage.xml" -returntargetcode -targetargs:"--agents=1 --labels=On --result=TestResult.xml;format=nunit2 myapp.Tests.x64\bin\Release\Tests64.dll "" wurde mit dem Code -5 beendet

Steps to reproduce the problem:

works fine with .NET 4.6.x and OpenCover 4.6.519 but after update to .NET 4.8 I got the result above (tried also the actual Version of OpenCover 4.7.922. Yes also I read the release notes where I saw support for 4.7.x was integrated but not 4.8 - so maybe this is the answer.

So my questions are ...

is this true that 4.8 is not working ? (Property full PDB support is checked for all projects in csproj)
is support for 4.8 is planned and maybe when ?
is there another switch ?

Thanks for your help. Regards Dirk

P.S. First question here so sorry for mistakes.

Birgos commented 5 years ago

During this week and last week, all of our build servers have received Windows updates, including an update to the .NET Framework 4.8.03761.

Since then, CI Server (TeamCity) can no longer generate OpenCover reports, whether they are MS tests or NUnit tests. The command is called from an MSBuild script that is part of a larger build process.

Test Run Summary
  Overall result: Passed
  Test Count: 953, Passed: 953, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2019-08-15 14:48:43Z
    End time: 2019-08-15 14:55:16Z
    Duration: 393.267 seconds

Results (nunit2) saved as TestResult.xml
Committing...
No results, this could be for a number of reasons. The most common reasons are:
    1) missing PDBs for the assemblies that match the filter please review the
    output file and refer to the Usage guide (Usage.rtf) about filters.
    2) the profiler may not be registered correctly, please refer to the Usage
    guide and the -register switch.

But when I surprisingly open a Developer Command Prompt (elevated as TeamCity User) on the server and explicitly call the command (e.g.): ".\packages\OpenCover.4.7.922\tools\OpenCover.Console.exe" -register:user -target:".\packages\NUnit.ConsoleRunner.3.9.0\tools\nunit3-console.exe" -targetargs:"\".\src\ProjectName.Web.Tests\bin\Release\ProjectName.Web.Tests.dll\". \".\src\ProjectName.Core.Tests\bin\Release\ProjectName.Core.Tests.dll\" \".\src\ProjectName.Web.Tests.Integration\bin\Release\ProjectName.Web.Tests.Integration.dll\" \".\src\ProjectName.Core.Tests.Integration\bin\Release\ProjectName.Core.Tests.Integration.dll\" --process=Single --encoding=utf-8 --teamcity --result=TestResult.xml;format=nunit2" -filter: "+[*ProjectName*]* -[*.Tests]* -[*.Test]* -[*.Tests.Integration]* -[*.Test.Integration]* +[*ProjectName*]* -[*.Tests]* -[*.Test]* -[*.Test.Integration]* -[*.Test.Integration]* -[*]*BundleConfig -[*]*NinjectWebCommon -[*]*RouteConfig -[*]*MvcApplication -[*]*Startup" output:".\build\output\build\unitTests_NUnit_OpenCover\OpenCoverReport.xml" -skipautoprops -excludebyattribute: "*ExcludeFromCodeCoverageAttribute;*CompilerGeneratedAttribute".

the reports are generated without problems.

It looks like no more reports can be generated if our build script is executed by the CI server (TeamCity) since the framework updates were installed...

Birgos commented 5 years ago

I can darkly remember that we already had a similar problem last year. The calculation didn't work then either, because the profiler didn't seem to be able to register anymore.

After days of research, I came across this Knowledge Base article at Microsoft: https://support.microsoft.com/en-us/help/4346406/update-for-net-framework-4-6-4-6-1-4-6-2-4-7-4-7-1-and-4-7-2-on-window

This update addresses an issue that was introduced in the July 2018 .NET Framework updates. Applications that rely on COM components were failing to load or run correctly because of "access denied," "class not registered," or "internal failure occurred for unknown reasons" errors.

After installation of the described update from the KB-Article (windows8.1-kb4346406-x64_03d960444b42b5838683d4791c9f782b2c242593.msu) the bug was fixed and OpenCover worked fine again.

According to Microsoft, the update is also said to have reached the normal update process: https://support.microsoft.com/en-us/help/4345913/access-denied-errors-after-installing-july-2018-security-rollup-update

sawilde commented 5 years ago

Not currently tested against 4.8 - usually though the change of frameworks doesn't break OpenCover

DirkFEng commented 5 years ago

We just tried it from VS-Dev - command line (after installing the patch kb4346406) - same result - we will check this later again in more detail

Birgos commented 5 years ago

To be clear: The patch KB4346406 fixed this issue the last time - after updating .NET framework to 4.6-4.7.2.

I guess the probability that this patch will help again is rather low, because meanwhile, according to Microsoft, this patch should already be included and Opencover does not function anymore when started through our CI-System and after the framework was updated to 4.8.0.

SteveGilham commented 5 years ago

It is likely that using --register:Path32 or --register:Path64 (depending on context) will provide a work-round. The behaviour reported here and in #915 are very similar, with an interactive process working, but the same hardware failing with the build client.

My gut feel is that the non-interactive build process is somehow being sandboxed sufficiently that it can't perform registration, even as a user.

splatteredbits commented 4 years ago

We also just updated our build servers to 4.8 and OpenCover now complains about no results. Here's how we're calling OpenCover (new lines added to show each argument):

"D:\Jenkins_Home\********\workspace\packages\OpenCover.4.7.922\tools\OpenCover.Console.exe" 
    "-target:D:\Jenkins_Home\********\workspace\packages\NUnit.Runners.2.6.4\tools\nunit-console.exe" 
    "-targetargs:\"D:\Jenkins_Home\********\workspace\WebMD.********.Tests\bin\Debug\WebMD.********.Tests.dll\" \"D:\Jenkins_Home\********\workspace\WebMD.********.Tests\bin\Debug\WebMD.********.Tests.dll\" /noshadow /framework=4.0 /xml=\"D:\Jenkins_Home\********\workspace\.output\nunit2+2tryvj0r.ddj.xml\"  
    " -filter:+[WebMD*]* 
    -register:user 
    "-output:D:\Jenkins_Home\********\workspace\.output\opencover\openCover.xml" 
    -returntargetcode

splatteredbits commented 4 years ago

Looks like -register:Path64 fixes the issue. We were using -register:user.

sawilde commented 4 years ago

.NET framework core doesn't use the registry which the switches -register and -register:user use.

I suspect 4.8 has taken the same path - may need to update instructions if we can identify the framework/runtime being used.

carlos-quintero commented 4 years ago

@sawilde There is no such thing as ".NET framework core". There is ".NET Framework" (latest version 4.8) and there is ".NET Core" (latest version 3.0). Within a year there will be ".NET 5" (which will be actually .NET Core, removing the "Core" moniker)

I guess that you refer to ".NET Core", which is cross-platform, and therefore cannot use Windows-specific things such as the Registry.

But ".NET Framework 4.8" is Windows-only and can use the Windows Registry as usual.

For the record:

On Windows 10, -register:user works correctly with .NET Framework 4.8 with both User Account Control (UAC) activated or deactivated (Control Panel, User Accounts).
On Windows 2012 R2 Server, -register:user works correctly with .NET Framework 4.8... if UAC is activated. It's only when UAC is deactivated when it fails and, even worse, it does it silently. With .NET Framework 4.7 it works with UAC activated or deactivated.

It happens that for some (likely stupid) reason most IT admins turn off UAC on Windows Server (some of them turn it off on Windows client for end users too, more stupidly) but I digress...

I don't know yet why per-user COM registration doesn't work on Windowes Server with .NET Framework 4.8, admin user accounts and UAC deactivated (and I don't have much time for further investigation) but I suspect it's a bug of .NET Framework 4.8 with Windows Server. Some pointers:

I am sure that the "server part" of the per-user COM registration (-register:user in OpenCover) works fine and Registry entries are created correctly; it's the "client part" what fails to "see" the per-user COM registration Registry entries. When talking about per-user COM registration we must talk always about the UAC feature introduced in Windows Vista:

UAC provides two kinds of user accounts: standard user accounts (no admin rights ever) and admin user accounts (admin rights depending on UAC):

With UAC activated, admin user accounts don't have admin rights by default, they are like standard user accounts most of the time. Only when admin rights are required, you get a "consent prompt" to "elevate" the process to adquire admin rights.
With UAC deactivated, admin user accounts have admin rights all the time (like pre-UAC operating systems such as Windows XP or Windows 2003 Server).

Per-user COM registration and UAC are somewhat tricky:

1) With UAC activated:

1.1) The admin user account, which runs processes actually without admin rights (not elevated), can "see" per-user COM registration. For this reason, "-register:user" works correctly on Windows Server 2012 and Windows 10 with an admin user account and .NET Framework 4.8.

1.2) If the admin user account "elevates" a process (for example, right-clicking "Run as administrator" to launch the process), then the elevated process cannot see per-user COM registration. This is a security safeguard to prevent a process running with admin rights to use a COM DLL that didn't require admin rights to be registered (per-user COM registration writes to HKEY_CURRENT_USER\Software\Classes rather than to HKEY_LOCAL_MACHINE\Software\Classes, being HKEY_CLASSES_ROOT a fusion of both)

2) But what happens with an admin user account when UAC is deactivated?:

2.1) On the original Windows Vista, an admin user account could not "see" per-user COM registration. Since this decision didn't make much sense (with UAC deactivated the admin user account runs "unprotected" anyway so there is no point on protecting it from rogue per-user COM DLLs) Microsoft reverted it on Windows Vista SP1.

2.2) On Windows Vista SP1 and higher, an admin user account should "see" per-user COM registration...unless something breaks this behavior unintentionally. This breaking actor can be a security patch (that must be reverted or fixed) or maybe .NET Framework 4.8.

Bottom line for users hitting this issue: avoid per-user COM-registration (-register:user) as much as possible and use either per-machine COM-registration (-register) or, better yet, no COM-registration (-register:Path64, -register:Path32)

For OpenCover: at the very least, it should not fail silently when the client part of per-user COM registration doesn't work. Rather than the generic "Committing...No results, this could be for a number of reasons..." message, it should show the specific reason:

When trying to instantiate a per-user COM object (using CreateObject() and similar APIs) you get an error/exception that should not be silenced.
When reading directly with the .NET API Registry.ClassesRoot.OpenSubKey(...), an error should be shown if the code doesn't "see" the key.

That would save long hours to users hitting this problem.

Further investigation should isolate this problem to simple code and report it to Microsoft for triage and fixing.

sawilde commented 4 years ago

@carlos-quintero some good points and it would be worth carding up into specific requirements, however fixing it is not going to happen in the short term due to lack of available time.

anderssonjohan commented 4 years ago

Same error here, build server was upgraded to dotnet 4.8 by windows updates and since then the same commits (re-running jobs in our CI Jenkins manually) does not provide code coverage results.

Jenkins is running non-interactively as a service, starting opencover from a build step but when run the same command (same job workspace that Jenkins created) from a powershell prompt via an RDP session it works fine.

Jenkins windows service is running with its own service account.

sawilde commented 3 years ago

when -register fails it returns the following error messages (this was included in release 4.7.922)

Failed to register(user:False,register:True,is64:False):5 the profiler assembly; you may want to look into permissions or using the -register:user option instead. C:\WINDOWS\system32\regsvr32.exe /s   "C:\prj\opencover\main\bin\Debug\x86\OpenCover.Profiler.dll"
If you are unable to resolve the issue please contact the OpenCover development team

The 4.8 issues could be due to PDB changes - which now load due to an upgrade to the latest mono.cecil assembly

OpenCover / opencover