Calls to CallContext.LogicalSetData made in external libraries during tests error out in VS IDE

smadala commented 7 years ago

From @ekumlin on April 20, 2017 17:55

Description

When calling CallContext.LogicalSetData in an external library during a test method, the VS IDE will return the following error:

An exception occurred while invoking executor 'executor://mstestadapter/v2': Unable to find assembly 'External.Library.Name, Version=2.0.5.0, Culture=neutral, PublicKeyToken=null'.

I expect this issue would happen with calls other than LogicalSetData, but this is the reliable repro we found first.

Notes

This is a regression from Visual Studio 2015.
This only happens in the VS IDE. If I run vstest.console.exe on the DLLs, there is no error and the test passes.

Steps to reproduce

(Note: I had some issues getting a .NET Core sample set up. My sample projects all use the net452 framework.)

Create a solution with a Class Library (.NET Standard) called External.Library.Name.

Add a class called Foo with the following code:

namespace External.Library.Name
{
    using System;
    using System.Runtime.Remoting.Messaging;

    [Serializable]
    public class Foo
    {
        private static readonly string ValueKey = typeof(Foo).FullName;

        public static Foo B()
        {
            return CallContext.LogicalGetData(ValueKey) as Foo ?? new Foo();
        }

        public static void A(Foo value)
        {
            CallContext.LogicalSetData(ValueKey, value);
        }
    }
}

Build and package the project as a NuGet: msbuild /t:pack External.Library.Name.csproj
Move the packaged NuGet to a local NuGet feed (that is referenced by Visual Studio).
Create a solution with a Console App (.NET Core) called FooAppX.
Add a Class Library (.NET Standard) called Test.FooAppX.
1. Via the NuGet Package Manager UI, add the appropriate Microsoft.NET.Test.Sdk, MSTest.TestAdapter, and MSTest.TestFramework packages. (We have tried with latest 15.0.0 and 1.1.14 versions.)
2. Via the NuGet Package Manager UI, add the local External.Library.Name package.

Add a test class called Tests with the following code:

namespace Test.FooAppX
{
    using External.Library.Name;
    using Microsoft.VisualStudio.TestTools.UnitTesting;

    [TestClass]
    public class Tests
    {
        [TestMethod]
        public void Test()
        {
            Foo.A(Foo.B());
        }
    }
}

Run the test.

Sample project

ExternalLibraryVsTestExecutorError.zip

Includes:

FooAppX project
External.Library.Name project
External.Library.Name nupkg file

Expected behavior

The test should pass with no errors.

Actual behavior

The test does not run. The error below is shown in the output window.

An exception occurred while invoking executor 'executor://mstestadapter/v2': Unable to find assembly 'External.Library.Name, Version=2.0.5.0, Culture=neutral, PublicKeyToken=null'.

The test will also halt execution; that is, if other tests are available in the app after this test, they will not run either.

Environment

Windows 10 x64
Visual Studio 15.1 (26403.7); the bug was first seen in 26403.3

Copied from original issue: Microsoft/vstest#744

smadala commented 7 years ago

From @sbaid on April 26, 2017 3:55

/cc @AbhitejJohn

smadala commented 7 years ago

From @ekumlin on May 9, 2017 20:42

Just following up on this after a couple weeks.

Is this issue being actively triaged or worked on? Currently our only workaround for this is to run tests from the console instead of the IDE, which is a significant workflow change.

Thanks.

smadala commented 7 years ago

This is bug in MSTestV2, External.Library.Name.Foo is not available in parent app domain causing failure.

Available workarounds to run tests from VS IDE: 1) Downgrade to MSTestV1 for test project which depends on External.Library.Name. 2) Copy External.Library.Name.dll to C:\Users\samadala\AppData\Local\Temp\VisualStudioTestExplorerExtensions\<MSTestVersion>\build\_common (not recommended).

Moving this to https://github.com/Microsoft/testfx

smadala commented 7 years ago

@ekumlin

Downgrade to MSTestV1 for test project which depends on External.Library.Name.

Sorry there was patching issue, this won't work and I have observed same test app not working in VS 2015 too.

Can you please provide vstest.executionengine TPTrace logs and process dump when you are running in debug mode in VS 2015?

Another workaround, You can use runsettings options to pass test adapter path. Example:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <RunConfiguration>
    <TestAdaptersPaths>C:\Users\samadala\Downloads\ExternalLibraryVsTestExecutorError\FooAppX\Test.FooAppX\bin\Debug\net452</TestAdaptersPaths>
  </RunConfiguration>
</RunSettings>

In host process that runs tests, there are usually two app domains:

Parent app domain that loads adapter and other infrastructure. App base for this is global adapter cache i.e %LOCALAPPDATA%VisualStudioTestExplorerExtensions folder.
Child app domain that loads single test source(test dll) that runs/discovers the tests. App base for this is test dll directory(bin/debug).

Child app domain is need to isolation and it gets unloaded as soon as tests completed.

When we call CallContext.LogicalSetData it appears to be recreating object Foo in parent app domain. this triggers assembly load failure for External.Library.Name which it can't find in parent app domain app base. By design test dll dependencies should not load in parent app domain for isolation.

ek68794998 commented 7 years ago

Can you please provide vstest.executionengine TPTrace logs and process dump when you are running in debug mode in VS 2015?

This might take some time as I have uninstalled VS 2015. I will work on this and get back to you.

When we call CallContext.LogicalSetData it appears to be recreating object Foo in parent app domain. this triggers assembly load failure for External.Library.Name which it can't find in parent app domain app base.

By design test dll dependencies should not load in parent app domain for isolation.

Then it's odd, though, that it would work properly from the command line, since that runs in isolation mode as well. I tend to agree that this description matches up, but the fact that the IDE has different behavior in this case than the command line, seems to be undesired behavior.

smadala commented 7 years ago

Behavior is depends on /Testadapterpath, Parent app domain has event AssemblyResolve which search for assemblies in test adapter path. From IDE the test adapter path is %LOCALAPPDATA%VisualStudioTestExplorerExtensions. In command line you might giving test dll directory path(by default), If you give .nuget cache path then it will fail.

ek68794998 commented 7 years ago

Why was this closed? Has it been fixed?

AbhitejJohn commented 7 years ago

@ekumlin: Sorry I missed adding a comment. From the discussion above this looks By-Design to me. Hence the closure. However I am puzzled as to how this was working in VS 2015. Do feel free to re-open this when you get the chance to get a repro.

ek68794998 commented 7 years ago

I'll do a repro when I can; however, I'm not sure why this would be considered by-design since the behavior differs between IDE and command line. The IDE should be able to pick up the correct DLL paths and find the desired DLL, same as the command line does.

AbhitejJohn commented 7 years ago

Well it actually depends on what mode in the CLI we are comparing this against. If CLI is run only against one test source then yes this is a disparity. However when that is scaled up to multiple test sources then you would find that this scenario fails there. Now IDE by default functions as the later case in CLI because we want to fire up one discovery/execution request for all the test sources(that can be hosted in the same runtime) for performance reasons. That said, this currently also invades into the parent app domain, that isn't ideal for isolation and could cause in-determinate behavior in a test session for other sources. Would be more problematic in IDE scenarios where a host process can be kept alive as long as that VS instance is alive.

vinishiru commented 4 years ago

Same issue here in MSTest V2 - VS 2019 16.6.0.0.

An exception occurred while invoking executor ‘executor://mstestadapter/v2’: Type is not resolved for member ‘log4net.Util.PropertiesDictionary,log4net, Version=2.0.8.0, Culture=neutral, PublicKeyToken=669e0ddf0bb1aa2a’.

I'm using log4net.LogicalThreadContext, which internally calls CallContext.LogicalSetData, so I think it's the same problem as reported.

Have resolved this using the registering log4net into GAC workaround on ours agent server.

microsoft / testfx