Exception thrown while loading tests if In-Proc VSTest DataCollector is used

[pkrukp]

Hello,

When a custom VSTest In-Proc DataCollector is used, NUnit Test Adapter cannot find nunit.framework.

Part of output:

NUnit Adapter 3.9.0.0: Test execution started Running all tests in C:\Users\pkruk\Documents\Visual Studio 2017\Projects\VSTestIntegration\SampleLibrary.NUnitTests\bin\Debug\SampleLibrary.NUnitTests.dll Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests in C:\Users\pkruk\Documents\Visual Studio 2017\Projects\VSTestIntegration\SampleLibrary.NUnitTests\bin\Debug\SampleLibrary.NUnitTests.dll An exception occurred in the driver while loading tests. at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage) [...] Innerexception: System.IO.FileNotFoundException: Could not load file or assembly 'nunit.framework' or one of its dependencies. The system cannot find the file specified. File name: 'nunit.framework'

Attaching: nunit.output.txt InProcTestDataCollector.zip

This happens with NUnit3TestAdapter.3.9.0 I also used vstest built locally from recent master but also vstest from VS 2017 (15.3.0-preview-20170628-02).

At first I thought this might be a problem with VSTest itself, but I have tested xUnit and MSTest and they both work correctly with In-Proc data collectors. Also, I have quickly checked there is some special code in NUnit Test Adapter which checks for presence of InProcDataCollector in run settings, and in that case behaves differently than normally (while I could not find any similar code in xUnit) which hints at issue in NUnit Test Adapter code.

[pkrukp]

More info: it looks like assembly search paths are different. When I put nunit.framework.dll next to test adapter dll, everything worked correctly.

Note that when in-proc data collector is not used, everything works correctly - nunit.framework.dll which is next to tested assembly is used.

[OsirisTerje]

Can you please upload (or point to a git repo containing it) a complete repro project? Make it as small as you can, it should compile "as is".

[pkrukp]

Hi @OsirisTerje

I'm attaching updated InProcTestDataCollector.zip. I have added Microsoft.TestPlatform.ObjectModel nuget package reference so it should now build out of the box without errors.

Please note that after extracting it you MUST update inproc.runsettings file and specify correct codebase (depends where you extracted it to).

I'm also attaching a test project with NUnit tests: SampleLibrary.zip

Please let me know if something is still not working.

Here is how I tested/reproduced this issue:

1. extract SampleLibrary.zip to d:\dv.pr\github_nunit3-vs-adapter\426\SampleLibrary\
2. build SampleLibrary.sln
3. extract InProcTestDataCollector.zip to d:\dv.pr\github_nunit3-vs-adapter\426\InProcTestDataCollector\
4. build CustomInProcTestDataCollector.sln
5. edit inproc.runsettings - update codebase property to point to D:\dv.pr\github_nunit3-vs-adapter\426\InProcTestDataCollector\bin\Debug\CustomInProcTestDataCollector.dll
6. download vstest console runner https://www.nuget.org/packages/Microsoft.TestPlatform/15.6.0-preview-20171211-02
7. extract microsoft.testplatform.15.6.0-preview-20171211-02.nupkg to d:\dv.pr\github_nunit3-vs-adapter\426\microsoft.testplatform.15.6.0-preview-20171211-02\
8. download NUnit3TestAdapter https://www.nuget.org/packages/NUnit3TestAdapter/
9. extract nunit3testadapter.3.9.0.nupkg to d:\dv.pr\github_nunit3-vs-adapter\426\nunit3testadapter.3.9.0\

a) run without in-proc data collector:

"D:\dv.pr\github_nunit3-vs-adapter\426\microsoft.testplatform.15.6.0-preview-20171211-02\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" "D:\dv.pr\github_nunit3-vs-adapter\426\SampleLibrary\SampleLibrary.NUnitTestsV3\bin\Debug\SampleLibrary.NUnitTests.dll" "/TestAdapterPath:D:\dv.pr\github_nunit3-vs-adapter\426\nunit3testadapter.3.9.0\build\net35"

Result: tests are executed correctly

b) run with in-proc data collector:

"D:\dv.pr\github_nunit3-vs-adapter\426\microsoft.testplatform.15.6.0-preview-20171211-02\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" "D:\dv.pr\github_nunit3-vs-adapter\426\SampleLibrary\SampleLibrary.NUnitTestsV3\bin\Debug\SampleLibrary.NUnitTests.dll" "/TestAdapterPath:D:\dv.pr\github_nunit3-vs-adapter\426\nunit3testadapter.3.9.0\build\net35" "/settings:D:\dv.pr\github_nunit3-vs-adapter\426\InProcTestDataCollector\inproc.runsettings"

Result: exception is thrown saying it can't find nunit.framework or one of its dependencies.

c) copy nunit.framework.dll to d:\dv.pr\github_nunit3-vs-adapter\426\nunit3testadapter.3.9.0\build\net35\ and run with in-proc data collector again

Result: tests are executed correctly

[pkrukp]

Hi,

vstest team looked at this problem from their side (https://github.com/Microsoft/vstest/issues/1331) and and seems it's an issue in nunit test adapter code. Please see the referenced issue for details.

[CharliePoole]

I looked at the issue and it doesn't make sense entirely. NUnit doesn't "look" anywhere to find assemblies that are referenced. It merely allows .NET to find things in the normal way. Of course, it does create an AppDomain in which to run the tests but the user has the ability to control both the base and the private probing path used for loading. If there are situations in which the adapter should be setting up the appdomain in a non-standard way, then we would need to know exactly what that is.

[ShreyasRmsft]

Basically when the new appdomain is created Nunit is probably setting the base/private probing path to the test dlls directory, but the behavior changes when an in proc data collector is used.

[CharliePoole]

No probably about it! That's what nunit does. By default it sets the base automatically to the location of the rest assembly but the calling program can override that behavior.

In this case, the adapter is the calling program. All we need is for someone to spell out what changes to the base / probing path are needed and in what conditions.

Just saying "It's NUnit's problem" as I read the MS issue doesn't really help.

[ShreyasRmsft]

@CharliePoole, terribly sorry I called it an Nunit problem. It's just that in the case there's an inproc data collector the base/probing path probably still needs to remain the test assembly directory to maintain parity with the scenario there's no inproc datacollector. Can you please confirm what the base/probing path is when an inproc data collector is present?

[CharliePoole]

@ShreyasRmsft Ah... didn't notice the msft in your handle before. 😄 No offense taken, I was mostly commenting on the language that talked about NUnit "looking for" assemblies, which is a loose way of speaking I get from users a lot. FWIW, many test frameworks in the past did actually do their own loading.

Anyway, I looked at the code to refresh my memory. I had forgotten that we accepted a MS contribution, which disables creation of an AppDomain when an inproc data collector is active. I was dubious, frankly, but we were assured that everything needed to execute would be available in the AppDomain under which we were called.

@OsirisTerje If my memory is right, you got assurance that the contributor or his group would fix anything about this that got broken. Is that right?

We originally objected to this idea because NUnit itself really doesn't have full support for the --domain=None setting, which we intended to remove in 3.0 but kept for a minority of users. Users are told that they can only use it if they copy all the necessary files - including NUnit assemblies - into a common directory.

Unfortunately, we were told it's the only way that an inproc data collector can work.

[OsirisTerje]

@CharliePoole Yes, there is not much we know about this part. I am also a bit confused about this issue, and wonder if this is somehow related to #328 ? @ShreyasRmsft ?
Does @acesiddhu know anything about this part?

[CharliePoole]

It seems to me that life would be a lot simpler if the explorer didn't use .runsettings to pass us settings but instead provided parameters programmatically in an API. As it is we use the same file for both user settings and VS overrides.

[ShreyasRmsft]

@CharliePoole The NUnit adapter Nuget package can modify the csproj so that the adapter dlls get copied to the out directory on building. Both Xunit and MsTest do this I believe. Might save folks a lot of trouble copying the dll there themselves when using an inproc datacollector.

@OsirisTerje No this issue is not related to #328. It only occurs in the case of inproc datacollectors and like @CharliePoole pointed out its occurring because a separate appdomain isn't created.

[CharliePoole]

That's the sort of thing we were told would be done before we were called if VS passed in the setting that told us to use --domain=None. That is, everything would be in the output directory that was needed in order to run the tests.

Of course, it's up to @OsirisTerje if he wants to do as you suggest but it seems we are being led a merry dance here by Microsoft.

[pkrukp]

I understand there was some misunderstanding between nunit and vstest teams (I didn't want my original report to be seen as blaming anyone, sorry if you read like that), but could you tell me if there's a plan for fixing this issue, and if yes, when would the fix be available?

I understand this is not a high priority issue, as it works under visual studio (test adapter is copied to project output directory, so nunit.framework.dll is there too), but this issue makes vstest.console.exe or VsTestConsoleWrapper quite unusable.

Do you think copying nunit.framework.dll into test adapter directory would be viable workaround? I assume it would have to be the same version of the framework dll as the one referenced in test project...

[thomashoef]

@CharliePoole Is there solution for this issue? Whenever I run tests locally, they work, but on Team Foundation Server, they fail saying nunit.framework could not be found (FileNotFoundException)

2018-05-04T11:20:56.4737305Z Die Vstest-Diagnose wird in der Datei E:\agent_work_temp\357c6d10-4f8d-11e8-b1c2-eb05188701ef.txt protokolliert. 2018-05-04T11:20:56.8487239Z NUnit Adapter 3.10.0.21: Test execution started 2018-05-04T11:20:56.8643421Z Running all tests in E:\agent_work\14\s\d\App.Tests\obj\Debug\App.Tests.dll 2018-05-04T11:20:57.2706299Z Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests in E:\agent_work\14\s\d\App.Tests\obj\Debug\App.Tests.dll 2018-05-04T11:20:57.2706299Z An exception occurred in the driver while loading tests. 2018-05-04T11:20:57.2862129Z bei NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage) 2018-05-04T11:20:57.2862129Z bei NUnit.Engine.Runners.DirectTestRunner.LoadPackage() 2018-05-04T11:20:57.2862129Z bei NUnit.Engine.Runners.TestDomainRunner.LoadPackage() 2018-05-04T11:20:57.2862129Z bei NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded() 2018-05-04T11:20:57.2862129Z bei NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter) 2018-05-04T11:20:57.2862129Z bei NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter) 2018-05-04T11:20:57.2862129Z bei NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, TestFilter filter) 2018-05-04T11:20:57.2862129Z Innerexception: System.IO.FileNotFoundException: Die Datei oder Assembly "nunit.framework" oder eine Abhängigkeit davon wurde nicht gefunden. Das System kann die angegebene Datei nicht finden. 2018-05-04T11:20:57.2862129Z Dateiname: "nunit.framework"

Thanks in advance! Thomas

[CharliePoole]

@thomashoef I'm no longer an active member of this team so @OsirisTerje will have to help you on this.

@pkruk2 (Just noticed your comment) I don't think there's disagreement. It's simply that the VSTest team contributed the code for this feature to the adapter and agreed to maintain it if problems arose. AFAIK, we're waiting for them to pick up this issue. @OsirisTerje please correct me if I'm wrong there!

[OsirisTerje]

@CharliePoole Right you are. I am waiting for their response, pinged today.

@thomashoef Thanks for "upvoting" it, it helps clarify what people need.

[thomashoef]

hi @OsirisTerje would you have an idea how to work around this? I set a test adapter folder to $(Build.SourcesDirectory)\Testing\NUnit3TestAdapter.3.10.0\build\net35 and made sure to copy the nunit.framework.dll into it -- result: does not change a thing. It does not work with either v1 or v2 of the task...

[OsirisTerje]

Been told they are looking at it now, and we should get a response very soon.

[cltshivash]

The issue looks to be caused by the commit : https://github.com/nunit/nunit3-vs-adapter/commit/2bb463d0ebe95ba8d6ddb568d5dd26ce13359e55

https://github.com/nunit/nunit3-vs-adapter/commit/2bb463d0ebe95ba8d6ddb568d5dd26ce13359e55#diff-aaf79f0666f0f2147d49023c76eea659R146

Will update once we confirm on the fix needed.

[cltshivash]

@drognanar Could you please followup on this issue ? (I don't see a reason to have DomainUsage=None when InprocCollectors are present).

[thomashoef]

@drognanar @cltshivash Do you perhaps have a workaround for me so I can still run the tests on TFS?

[drognanar]

With our InProcDataCollector we wanted to determine to which test the collected data belongs (run tests one at a time). Whilst disabling the app domain creation is not necessary for our collector to work, it does remove the overhead of creating and talking between app domains.

My initial PR has exposed these as separate options, but in the end was changed to rely on the presence of InProcCollectors to avoid having adapter users be able to change this option freely. I guess that this issue shows that an explicit option is needed in order to enable to single test run mode rather than implicit option (are InProcCollectors present). The alternatives I can think of:

1. Add options for <DisableAppDomain> <RunTestsOneAtATime> <RunMessagePumpSynchronously>. This gives the flexibility to control how tests are run, however it brings back the concern that the NUnit team originally had.
2. Add a single option <CollectDataForEachTestSeparately>. This option does not allow users to change the three underlying options one at a time, thus limits the possibility of misuse and still allows VS/TestPlatform to enable more thorough data collection.
3. Allow an InProcDataCollector to specify the new option RunTestsSynchronously=True, that if set would enable the single test run mode. However it might be confusing that the setting on one InProcDataCollector will affect all others.
4. Have an explicit check for the LiveUnitTesting InProcDataCollector. Whilst possible I would prefer not having hardcoded behaviour in the adapter.

I personally would prefer implementing the second option, however I'm open to suggestions.

@thomashoef I cannot think of a workaround from the top of my head.

[OsirisTerje]

The 2nd option also seems the most user friendly to me. Will you add a PR for this?

[drognanar]

Certainly, I should have a PR ready by the end of today

[thomashoef]

@drognanar do you know how I could install your fix on TFS?

[OsirisTerje]

@thomashoef You can get a pre-release package from our myget feed. See http://hermit.no/get-nunit3testadapter-pre-release-packages/ for details