Testhost process for source(s) '...\__Instrumented_UnitTests\UnitTests.dll' exited with error: Unhandled exception. System.MethodAccessException: Attempt by method 'Microsoft.VisualStudio.TestPlatform.Execution.UiLanguageOverride..ctor()' to access method 'Microsoft.VisualStudio.TestPlatform.CoreUtilities.Helpers.EnvironmentVariableHelper..ctor()' failed.

jstedfast commented 10 months ago

I'm trying to switch MimeKit from OpenCover to AltCover and I'm running into the following issue:

       AltCoverRunPreparation:
         Instrumenting files from D:\src\MimeKit\UnitTests\bin\Debug\net6.0\
         Writing files to D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\
            => D:\src\MimeKit\UnitTests\bin\Debug\net6.0\AltCover.Monitor.dll
            => D:\src\MimeKit\UnitTests\bin\Debug\net6.0\Microsoft.TestPlatform.CommunicationUtilities.dll
            => D:\src\MimeKit\UnitTests\bin\Debug\net6.0\Microsoft.TestPlatform.CoreUtilities.dll
            => D:\src\MimeKit\UnitTests\bin\Debug\net6.0\Microsoft.TestPlatform.CrossPlatEngine.dll
            => D:\src\MimeKit\UnitTests\bin\Debug\net6.0\Microsoft.TestPlatform.PlatformAbstractions.dll
            => D:\src\MimeKit\UnitTests\bin\Debug\net6.0\Microsoft.TestPlatform.Utilities.dll
            => D:\src\MimeKit\UnitTests\bin\Debug\net6.0\Microsoft.VisualStudio.TestPlatform.Common.dll
            => D:\src\MimeKit\UnitTests\bin\Debug\net6.0\Microsoft.VisualStudio.TestPlatform.ObjectModel.dll
            => D:\src\MimeKit\UnitTests\bin\Debug\net6.0\MimeKit.dll
            => D:\src\MimeKit\UnitTests\bin\Debug\net6.0\NUnit3.TestAdapter.dll
            => D:\src\MimeKit\UnitTests\bin\Debug\net6.0\testhost.dll
            => D:\src\MimeKit\UnitTests\bin\Debug\net6.0\UnitTests.dll

         Coverage Report: D:\src\MimeKit\UnitTests\coverage.xml

     3>CopyFilesToOutputDirectory:
         Copying file from "D:\src\MimeKit\Benchmarks\obj\Debug\net6.0\Benchmarks.dll" to "D:\src\MimeKit\Benchmarks\bi
         n\Debug\net6.0\Benchmarks.dll".
         Copying reference assembly from "obj\Debug\net6.0\refint\Benchmarks.dll" to "D:\src\MimeKit\Benchmarks\obj\Deb
         ug\net6.0\ref\Benchmarks.dll".
         Benchmarks -> D:\src\MimeKit\Benchmarks\bin\Debug\net6.0\Benchmarks.dll
         Copying file from "D:\src\MimeKit\Benchmarks\obj\Debug\net6.0\Benchmarks.pdb" to "D:\src\MimeKit\Benchmarks\bi
         n\Debug\net6.0\Benchmarks.pdb".
     3>Done Building Project "D:\src\MimeKit\Benchmarks\Benchmarks.csproj" (Build target(s)).
     3>BuildProject:
         Build completed.

     3>Done Building Project "D:\src\MimeKit\Benchmarks\Benchmarks.csproj" (VSTest target(s)).
     4>AltCoverRunPreparation:
             D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\MimeKit.dll
                         <=  MimeKit, Version=4.1.0.0, Culture=neutral, PublicKeyToken=null
             D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\Microsoft.TestPlatform.PlatformAbstract
         ions.dll
                         <=  Microsoft.TestPlatform.PlatformAbstractions, Version=15.0.0.0, Culture=neutral, PublicKeyT
         oken=null
             D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\AltCover.Monitor.dll
                         <=  AltCover.Monitor, Version=8.6.0.0, Culture=neutral, PublicKeyToken=null
             D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\Microsoft.TestPlatform.CoreUtilities.dl
         l
                         <=  Microsoft.TestPlatform.CoreUtilities, Version=15.0.0.0, Culture=neutral, PublicKeyToken=nu
         ll
             D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\Microsoft.VisualStudio.TestPlatform.Obj
         ectModel.dll
                         <=  Microsoft.VisualStudio.TestPlatform.ObjectModel, Version=15.0.0.0, Culture=neutral, Public
         KeyToken=null
             D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\UnitTests.dll
                         <=  UnitTests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
             D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\NUnit3.TestAdapter.dll
                         <=  NUnit3.TestAdapter, Version=4.5.0.0, Culture=neutral, PublicKeyToken=null
             D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\Microsoft.TestPlatform.Utilities.dll
                         <=  Microsoft.TestPlatform.Utilities, Version=15.0.0.0, Culture=neutral, PublicKeyToken=null
             D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\Microsoft.VisualStudio.TestPlatform.Com
         mon.dll
                         <=  Microsoft.VisualStudio.TestPlatform.Common, Version=15.0.0.0, Culture=neutral, PublicKeyTo
         ken=null
             D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\Microsoft.TestPlatform.CommunicationUti
         lities.dll
                         <=  Microsoft.TestPlatform.CommunicationUtilities, Version=15.0.0.0, Culture=neutral, PublicKe
         yToken=null
             D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\Microsoft.TestPlatform.CrossPlatEngine.
         dll
                         <=  Microsoft.TestPlatform.CrossPlatEngine, Version=15.0.0.0, Culture=neutral, PublicKeyToken=
         null
             D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\testhost.dll
                         <=  testhost, Version=15.0.0.0, Culture=neutral, PublicKeyToken=null
       AltCoverVSTestCore:
         Settings Before:
         Settings After: C:\Users\jestedfa\AppData\Local\Temp\tmplgyva1.altcover.runsettings
Test run for D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\UnitTests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.7.0-preview-23317-01+919ec8358820228cc5fa77ef000051c1d6875399 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Testhost process for source(s) 'D:\src\MimeKit\UnitTests\bin\Debug\net6.0\__Instrumented_UnitTests\UnitTests.dll' exited with error: Unhandled exception. System.MethodAccessException: Attempt by method 'Microsoft.VisualStudio.TestPlatform.Execution.UiLanguageOverride..ctor()' to access method 'Microsoft.VisualStudio.TestPlatform.CoreUtilities.Helpers.EnvironmentVariableHelper..ctor()' failed.
   at Microsoft.VisualStudio.TestPlatform.Execution.UiLanguageOverride..ctor() in /_/src/Microsoft.TestPlatform.Execution.Shared/UILanguageOverride.cs:line 22
   at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Run(String[] args) in /_/src/testhost.x86/Program.cs:line 54
   at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Main(String[] args) in /_/src/testhost.x86/Program.cs:line 37
. Please check the diagnostic logs for more information.

Test Run Aborted.
         MSB4181: The "Microsoft.TestPlatform.Build.Tasks.VSTestTask" task returned false but did not log an error.
       AltCoverGenerateCoverageResult:
         ... D:\src\MimeKit\UnitTests\coverage.xml.0.acv (b)
     4>D:\NuGet\altcover\8.6.68\build\netstandard2.0\AltCover.targets(203,5): warning : A total of 0 visits recorded [D
       :\src\MimeKit\UnitTests\UnitTests.csproj]
         Coverage statistics flushing took 2.06 seconds
         Visited Classes 0 of 1685 (0)
         Visited Methods 0 of 10412 (0)
         Visited Points 0 of 82541 (0)
         Visited Branches 0 of 32393 (0)
         Maximum CRAP score 8556

         ==== Alternative Results (includes all methods including those without corresponding source) ====
         Alternative Visited Classes 0 of 1785 (0)
         Alternative Visited Methods 0 of 12132 (0)
         Alternative maximum CRAP score 8556
         Deleting file "C:\Users\jestedfa\AppData\Local\Temp\tmplgyva1.altcover.runsettings".
     4>Done Building Project "D:\src\MimeKit\UnitTests\UnitTests.csproj" (VSTest target(s)) -- FAILED.
     5>Done Building Project "D:\src\MimeKit\UnitTests\UnitTests.csproj.metaproj" (VSTest target(s)) -- FAILED.
     1>Done Building Project "D:\src\MimeKit\MimeKit.sln" (VSTest target(s)) -- FAILED.

Build FAILED.

       "D:\src\MimeKit\MimeKit.sln" (VSTest target) (1:2) ->
       "D:\src\MimeKit\UnitTests\UnitTests.csproj.metaproj" (VSTest target) (5) ->
       "D:\src\MimeKit\UnitTests\UnitTests.csproj" (VSTest target) (4:6) ->
       (AltCoverGenerateCoverageResult target) ->
         D:\NuGet\altcover\8.6.68\build\netstandard2.0\AltCover.targets(203,5): warning : A total of 0 visits recorded
       [D:\src\MimeKit\UnitTests\UnitTests.csproj]

    1 Warning(s)
    0 Error(s)

Time Elapsed 00:00:25.79

Any idea what I'm doing wrong or what I could try to resolve this?

This is the change I made: https://github.com/jstedfast/MimeKit/commit/3928b53b70bc803f910b1a76690f6162b3986322

Also, I'm wondering if I could simplify the AssemblyExcludeFilter down to "?MimeKit.dll" so that only MimeKit.dll gets code coverage data.

Thanks.

jstedfast commented 10 months ago

I should probably mention that dotnet test UnitTests\UnitTests.csproj by itself works (which is why I switched to that method), but once I add in /p:AltCover=true, it fails.

When I was using OpenCover, I was running the OpenCover tool directly with the NUnit.ConsoleRunner command-line arguments. I may have to do something similar with AltCover. I was just hoping the simplified dotnet test approach would work to simplify things a ton :-)

jstedfast commented 10 months ago

I may have figured out how to do things properly:

dotnet AltCover -i=$OutputDir --inplace -s="System.*" -s="Microsoft.*" -s="Org.BouncyCastle.*" -s="Mono.*"
    -s="NUnit*" -s="AltCover.*" -s="testhost" -s="UnitTests"

dotnet AltCover Runner --recorderDirectory=$OutputDir --executable=$NUnitConsoleRunner --summary=O --
    --domain:single $UnitTestsAssembly

Question:

Can I just invoke the NUnut Console Runner instead of doing dotnet AltCover Runner? It looked like a coverage.xml was generated, but I'm not sure if that contained everything or not.

SteveGilham commented 10 months ago

The latest testhost release does cause problems like that; excluding the whole third-party stack from instrumentation as you have done - by whatever filtering method - is the thing to do.

As to the question - yes, you can run the tests as a separate stage between instrumenting the code and collecting the coverage data. The option to run tests as a sub-process is there purely as a convenience, if that should suit existing build & test processes.

jstedfast commented 10 months ago

FWIW, I've got AltCover working and reporting everything to coveralls successfully and here's how I managed to do it:

dotnet AltCover -i="bin\Debug\net6.0" --inplace -s="System.*" -s="Microsoft.*" -s="BouncyCastle.*" -s="NUnit*" -s="AltCover.*" -s="testhost" -s="UnitTests"
nunit3-console.exe --domain:single "bin\Debug\net6.0\UnitTests.dll"

As far as I'm concerned, this can be closed

SteveGilham commented 10 months ago

That is in fact how the earliest implementation (in 2010) would have worked, and is good because you're directly running tests with NUnit.

The whole Runner appendage and intermediate files is entirely a workround for the behaviour of vstest.exe (aka dotnet test) in swiftly terminating end of execution handlers, such as altcover uses to write the in-memory coverage data into the XML report when the test process terminates. These days, alas, dotnet test is the majority case, so has to be catered for.

SteveGilham / altcover