Unexpected dependency location when running on VSTS

[TroyWalshProf]

Description

I have multiple tests that depend on a common utility library. The utility library is designed to pull in data from a file and that file is relative to the test dll. The problem is that if I have more than one test dll that depends on this utility VSTS test will always choose to use the first instance of the dll it found. This is unfortunate when I expect a test file in a relative path.

Example: Structure:

• SLN

  • UtilityProj -bin/debug Utility.dll (original)

  • Test1Proj testDataFile.txt -bin/debug Test1.dll Utility.dll (copy 1) testDataFile.txt

  • Test2Proj testDataFile.txt -bin/debug Test2.dll Utility.dll (copy 2) testDataFile.txt

Behavior: Running locally (or with 1.3.0-beta 2 or below): - The good When you run tests from Test1.dll the Utility.dll "copy 1" is used When you run tests from Test2.dll the Utility.dll "copy 2" is used

Running on VSTS with 1.3.0 or above: - The bad When you run tests from Test1.dll the Utility.dll "copy 1" is used When you run tests from Test2.dll the Utility.dll "copy 1" is used

Steps to reproduce

0. Use this code - https://github.com/Magenic/QATDemo/tree/master/MSTestBug or follow steps 1-4
1. Create a solution with 2 unit test project and 1 utility dll
2. Have the utility find or open a file using a relative path
3. Have the tests use the utility
4. Add test files to the test projects
5. Run locally and note no errors occurred
6. Create a CI build on VSTS and note that tests for one of the dll fail because it cannot find or finds the work test file

Expected behavior

If a dll is found in the same path as the test dll than it should be used at execution time

Actual behavior

If a dll is found in multiple test dll directories the first instance is always used at execution time

Environment

Build server: VSTS Build task: Visual Studio Test V 2.* NuGet packages:

[jayaranigarg]

@TroyWalshProf : Acknowledging this to be a bug.

[TroyWalshProf]

Any ETA on when we could see this bug fix?

[ShreyasRmsft]

@TroyWalshProf,

There's a workaround for this

<RunConfiguration>
    <DisableAppDomain>True</DisableAppDomain>
</RunConfiguration>

add the above snip to your runsettings.

[ShreyasRmsft]

Also @TroyWalshProf

I ran locally with 1.2.0 The tests still failed.

Are you sure this was passing with older versions of the adapter?

[ShreyasRmsft]

The reason why you are facing problems is that by default mstest copies all the test binaries into a single folder and runs the test. Any files with the same name will be overridden.

<RunConfiguration>
    <DisableAppDomain>True</DisableAppDomain>
</RunConfiguration>

  <RunConfiguration>
    <MaxCpuCount>2</MaxCpuCount>
  </RunConfiguration>

Adding either of the above to you run settings ensures the tests are run from their respective bin folders and your scenario will work fine. And this is the behavior we want to make default.

This is the scenario as of today and yes we do need to document this more clearly from our side. We are also considering reworking the entire deployment mechanism.

I am marking this issue as an enhancement as the work that is planned is more of an enhancement than a bug (the behavior i mentioned has been there since the beginning of mstestv2, its not something that we regressed in 1.3.0 or anything).

[TroyWalshProf]

Thanks, that was very helpful. Totally agree that the settings files need better documentation.
*Better docs would be a great Christmas, Hanukkah, Kwanzaa, Saturnalia, Yule, Solstice, Hogswatch present :)

[ghost]

We have seen this in our VSTS onprem agent as well, with tests that were loading components from other assemblies via MEF. A check of the filesystem indicated the assemblies were being correctly positioned for the load (into the bin directory of the test projects' build output) but cardinality exceptions were still being thrown.

While the DisableAppDomain=True workaround solves our case as well, this was working correctly for us against v1.2.1 of the adaptor/framework with assembly-level parallelism, and broke when moving to >= v1.3.0.

[jayaranigarg]

@PST2: Can you give us little more insights as to what broke for you after moving to 1.3.0? Consider filing a separate issue if the issue faced by you is not same as the one being talked about on this thread.

[ghost]

We're on a closed source codebase, I'll see if I can construct a repro scenario for what we've observed.

[Evangelink]

Given there is no activity here and no upvotes, I will move forward by closing this issue. Please feel free to comment here if you are still facing this issue.