search

nunit / nunit3-vs-adapter

NUnit 3.0 Visual Studio test adapter for use under VS 2012 or later
https://nunit.org
MIT License
203 stars 105 forks source link

Operation could destabilize the runtime #519

Open NikitaEgorov opened 6 years ago

NikitaEgorov commented 6 years ago
OsirisTerje commented 6 years ago

Can you please upload a repro project? Make it as small as you can, it should compile "as is".

NikitaEgorov commented 6 years ago

It is internal project

siprbaum commented 6 years ago

Perhaps try reducing the internal project, e.g. remove code so long as it still reproduces. If you have a minimal reproducer with your production code, the next step would be to "rewrite" it so you can publish it here. Otherwise I'd guess it is really hard to analyse the issue without further information.

OsirisTerje commented 6 years ago

We need some more information on this, the normal way is that you provide a similar working repro project. We dont need to see anything from your internal project, but we need to know what you are trying to do. Running a normal test project with VSTS (which is ahead of TFS dev 16) works as it should, using the ordinary VSTest task.

(Thanks @siprbaum , you beat me to it :-) )

NikitaEgorov commented 6 years ago

I reproduce it use empty nunit project

packages.config

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="NUnit" version="3.10.1" targetFramework="net472" />
  <package id="NUnit3TestAdapter" version="3.10.0" targetFramework="net472" />
</packages>

UnitTest1.cs

using System;

using NUnit.Framework;

namespace UnitTestProject1
{
    [TestFixture]
    public class UnitTest1
    {
        [Test]
        public void TestMethod1()
        {
            Console.WriteLine("test");
        }
    }
}

b7b3e470-73cd-11e8-ba0f-2dd83a1f0155.runsettings

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <ModulePaths>
              <Include>
                <ModulePath>.*.dll$</ModulePath>
                <ModulePath>.*.exe$</ModulePath>
              </Include>
              <Exclude>
                <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
              </Exclude>
            </ModulePaths>
            <Functions>
              <Exclude>
                <Function>^Fabrikam.UnitTest..*</Function>
                <Function>^std::.*</Function>
                <Function>^ATL::.*</Function>
                <Function>.*::__GetTestMethodInfo.*</Function>
                <Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
                <Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
              </Exclude>
            </Functions>
            <Attributes>
              <Exclude>
                <Attribute>^System.Diagnostics.DebuggerHiddenAttribute$</Attribute>
                <Attribute>^System.Diagnostics.DebuggerNonUserCodeAttribute$</Attribute>
                <Attribute>^System.Runtime.CompilerServices.CompilerGeneratedAttribute$</Attribute>
                <Attribute>^System.CodeDom.Compiler.GeneratedCodeAttribute$</Attribute>
                <Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
              </Exclude>
            </Attributes>
            <Sources>
              <Exclude>
                <Source>.*\atlmfc\.*</Source>
                <Source>.*\vctools\.*</Source>
                <Source>.*\public\sdk\.*</Source>
                <Source>.*\microsoft sdks\.*</Source>
                <Source>.*\vc\include\.*</Source>
              </Exclude>
            </Sources>
            <CompanyNames>
              <Exclude>
                <CompanyName>.*microsoft.*</CompanyName>
              </Exclude>
            </CompanyNames>
            <PublicKeyTokens>
              <Exclude>
                <PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
                <PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
                <PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
                <PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
                <PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
                <PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
                <PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
              </Exclude>
            </PublicKeyTokens>
            <UseVerifiableInstrumentation>False</UseVerifiableInstrumentation>
            <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
            <CollectFromChildProcesses>True</CollectFromChildProcesses>
            <CollectAspDotNet>False</CollectAspDotNet>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

run.bat

e:\15.8.0-preview-20180610-02\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "UnitTestProject1.dll" /logger:"trx" /Settings:"b7b3e470-73cd-11e8-ba0f-2dd83a1f0155.runsettings" /TestAdapterPath:"e:\Test\Debug\"
NikitaEgorov commented 6 years ago 
Microsoft (R) Test Execution Command Line Tool Version 15.8.0-preview-20180610-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
NUnit Adapter 3.10.0.21: Test execution started
Running all tests in e:\Test\Debug\UnitTestProject1.dll
Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests in e:\Test\Debug\UnitTestProject1.dll
The NUnit 3 driver encountered an error while executing reflected code.
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary`2 settings)
   at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
   at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
   at NUnit.Engine.Runners.TestDomainRunner.LoadPackage()
   at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
   at NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter)
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, TestFilter filter) in D:\repos\nunit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 229
Innerexception: System.Security.VerificationException: Operation could destabilize the runtime.
   at NUnit.Framework.Api.FrameworkController..ctor(String assemblyNameOrPath, String idPrefix, IDictionary settings) in C:\src\nunit\nunit\src\NUnitFramework\framework\Api\FrameworkController.cs:line 68
NUnit Adapter 3.10.0.21: Test execution complete
No test is available in e:\Test\Debug\UnitTestProject1.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
OsirisTerje commented 6 years ago
  1. Can you zip together the project and upload that, the code above should work.
  2. Does it also fail when you don't use the runsettings file?
  3. Does it also fail when you change the .net framework target to e.g. 4.7.1 or 4.6.1 ?
  4. Does it also fail within Visual Studio ?
OsirisTerje commented 6 years ago

Ok, I can confirm it now. It is caused by the runsettings file. If you change your runsettings file to use InProcDataCollectors instead it will work.
This needs to be properly documented. I'll add a seperate issue for that.

image

NikitaEgorov commented 6 years ago

file b7b3e470-73cd-11e8-ba0f-2dd83a1f0155.runsettings is generated by VSTest task I can not change it manual

OsirisTerje commented 6 years ago

Yes, you can change it:
image

(MS also has noted the current behaviour as a bug, see links from @pvlakshm above. )

ShreyasRmsft commented 6 years ago

@OsirisTerje

The UseVerifiableInstrumentation is set to false by the VsTest task when the platform is acquired via the vstestplatform tool installer (gets the nuget package) as the profiler may not be available in the GAC and needs to be resolved using certain environment variables set by the VsTest task.

This is a must for code coverage to work when there is no VS installed on the target machine.

This will have to be fixed otherwise code coverage will not work for NUnit tests when run using the VsTest Platform Nuget package (No VS).

OsirisTerje commented 6 years ago

Thanks @ShreyasRmsft

With "have to be fixed", do you mean in the VSTest task or will the project need its own runsettings file?

NikitaEgorov commented 6 years ago

I added custom runsettings with True </ UseVerifiableInstrumentation> Nothing changes

NikitaEgorov commented 6 years ago

All works fine with MSTest :(

ShreyasRmsft commented 6 years ago

@OsirisTerje is the NUnit adapter throwing this exception? Is the adapter reading the data collector node in the runsettings?

OsirisTerje commented 6 years ago

@ShreyasRmsft It is an exception coming from the NUnit Engine, so it is not the adapter itself, but the embedded NUnit engine. I am unsure about how this collector is intended to work, one of you guys did these changes earlier, and it seems that something here causes the engine to crash. However, when I changed it to InProcDatacollector my repro worked.

And yes, the adapter reads the node and then sets some other properties: bilde

@NikitaEgorov Did you change the runsettings as I described above? (not only setting the flag to true, but replacing the DataCollector with InProcDataCollector?

ShreyasRmsft commented 6 years ago

@OsirisTerje changing it to inproc is not the solution. Ideally in this case the engine should not be reading these settings at all as they are specific to code coverage.

OsirisTerje commented 6 years ago

@ShreyasRmsft The engine don't read MS properties, only NUnit properties. The adapter does the conversion as shown above. Anything else being done must happen on the datacollector side. And this code above is from MS, so I assume you are setting whatever is needed correctly here ? The exception surfacing in the Engine doesn't seem to have anything to do with the Engine itself, but with whatever it is trying to load: bilde

ShreyasRmsft commented 6 years ago

@OsirisTerje I do not have a lot of context on the change that was done.

But yes if the change that was made by one of us is trying to read this settings then it needs to be fixed. I haven't had time yet to see where exactly in the code this is happening. But Nunit should ideally ignore anything present in the code coverage node. Will get back to you on the fix.

OsirisTerje commented 6 years ago

@ShreyasRmsft I believe the code involved were written by @navin22, i think this PR #391 was the essential one. I also see a later modification by @drognanar to "Use InProcDataCollector setting to enable sequential run", PR #510, but this is not released yet, it's for 3.11.

MoiMoiDG commented 5 years ago

Hi there,

I have been told that this issue affects another issue (which I submitted). I am quite new to this repo, so I cannot get what is the current status. Is anyone working on a solution to this issue?

Thanks!

OsirisTerje commented 5 years ago

@MoiMoiDG MS themselves are working on it on their side, but I'll check up on it.

MoiMoiDG commented 5 years ago

@OsirisTerje: Thanks! Is there any update on this?

OsirisTerje commented 5 years ago

@ShreyasRmsft Can you mail me on what I need to do with the adapter to have this fixed? (terje at hermit.no)

OsirisTerje commented 5 years ago

@ShreyasRmsft

Since we use sourcelink ( and many more do the same), using the pdb’s as a indicator for the “dll’s under test” is likely to cause more such issues. And the test framework dll's should not in any case be part of the profiling.

Do you call the profiler before or after you call the adapter? Can we add the nunit dll’s to the exclude filter in the adapter ?

ThomasPiskol commented 5 years ago

Is there any update on this issue?

mayankbansal018 commented 5 years ago

Sorry for such delay on this.

Issue Explanation: When we instrument a dll which contains methods/classes marked with certain Security Attributes e.g. SecurityCritical or few more, then during code coverage these methods are likely to throw exception, since these attributes prohibit certain scenarios, e.g run-time modification of existing code, or writing to raw memory, etc.

Solutions: We do have fix for such scenarios, where we delegate responsibility of writing to raw memory, to another dll, which we call Shim dll, which gets installed in GAC when VS is present, however in XCopyable scenario this fails.

We are planning to include Shim dll as part of our XCopy package, & also we plan to release code coverage nugets, which will allow users to take explicit dependency on Shim dll. I'll update the thread once we have done either.

For now only solution available to user would be use code coverage runsettings which will exclude NUnit dlls from getting instrumented.

MoiMoiDG commented 5 years ago

Hi @mayankbansal018,

Thanks for explaining the problem!

For now only solution available to user would be use code coverage runsettings which will exclude NUnit dlls from getting instrumented.

Could you please provide an example of such runsettings file where it works? I know it is a workaround, but it would be helpful.

Thanks!

mayankbansal018 commented 5 years ago

@MoiMoiDG the following runsettings should help you

`<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

.*.dll$ .*.exe$ .*CPPUnitTestFramework.* .*nunit.* ^Fabrikam.UnitTest..* ^std::.* ^ATL::.* .*::__GetTestMethodInfo.* ^Microsoft::VisualStudio::CppCodeCoverageFramework::.* ^Microsoft::VisualStudio::CppUnitTestFramework::.* ^System.Diagnostics.DebuggerHiddenAttribute$ ^System.Diagnostics.DebuggerNonUserCodeAttribute$ ^System.Runtime.CompilerServices.CompilerGeneratedAttribute$ ^System.CodeDom.Compiler.GeneratedCodeAttribute$ ^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$ .*\atlmfc\.* .*\vctools\.* .*\public\sdk\.* .*\microsoft sdks\.* .*\vc\include\.* .*microsoft.* ^B77A5C561934E089$ ^B03F5F7F11D50A3A$ ^31BF3856AD364E35$ ^89845DCD8080CC91$ ^71E9BCE111E9429C$ ^8F50407C4E9E73B6$ ^E361AF139669C375$ False True True False `
MoiMoiDG commented 5 years ago

Hi @mayankbansal018,

Thank you!

However, I still get the error in attachment. I attached the full solution too, for your convenience (it is just a small sample) ... :( 7_VsTest - testAssemblies.zip ESXD-NUTests.zip

OsirisTerje commented 5 years ago

@MoiMoiDG @mayankbansal018 : The project works when in Visual Studio, using the runsettings file, and failed without. After a bit hazzling it started working in both cases. Not really sure why. But in your log it says the runsettings ARE loaded, and I see you still get the same error.
@mayankbansal018 : This is a very annoying issue, and I there seems to be some caching issues involved. The issue is not stable.

mayankbansal018 commented 5 years ago

@OsirisTerje the caching of file issue can be only with Visual Studio, though we have not seen it too often, adding @AbhitejJohn, @ManishJayaswal to see if they have more info.

@MoiMoiDG can you confirm if this ever fails for you in CLI when used with above runsettings?

MoiMoiDG commented 5 years ago

Hi everyone,

Thank you for the support!

@mayankbansal018: I am not sure that I understood your question. The log comes from a VM attached to TFS 2018.3 and "Test platform version = Installed by Tools Installer".

samneefs commented 5 years ago

The following seems to be related: https://github.com/Microsoft/vstest/issues/1950 I had the error after switching to Newtonsoft.Json version 12. Turning off code coverage or using another json serializer are workarounds.

mayankbansal018 commented 5 years ago

Fixed with https://github.com/Microsoft/vstest/pull/1997, we have also released latest nuget package (16.0.2 preview) of test platform with above change, please upgrade to it.

OsirisTerje commented 5 years ago

@mayankbansal018 Do you refer to the VSTest task or to which nuget package?

mayankbansal018 commented 5 years ago

@OsirisTerje I'm referring to nuget that gets installed via Test Platform nuget installer task in pipeline. If users facing this issue upgrade to latest preview, then it should be resolved for them

sjblack commented 5 years ago

The latest version of the console (16.0.2 preview) did not resolve the problem for me.

DeadlyDragon commented 5 years ago

Is there an update on this? Running 16.2.0 and still hitting this issue. We need code coverage from the nunit test results in order to achieve our targets.

ShreyasRmsft commented 5 years ago

Please take a look at the comments in https://github.com/microsoft/vstest/pull/1997. I have provided details on the fix. Make sure you have followed all the instructions and even after that you are facing issues please provide the full set of debug logs for the pipeline run and we will take a look.

samneefs commented 5 years ago

The following seems to be related: microsoft/vstest#1950 I had the error after switching to Newtonsoft.Json version 12. Turning off code coverage or using another json serializer are workarounds.

Seems to be fixed with the combination of: Latest Newtonsoft.Json 12.0.2 (doesn't work with 12.0.1) Latest stable Test platform: 16.2 (should be >16.0.2) Latest VSTest task 2.156.2 (should be >2.153.0)