TestCentric fails to load .nunit project file that contains parent relative path

[maettu-this]

Describe the bug Loading tests of an .nunit project file containing child relative paths works fine. Loading tests of an .nunit project file containing parent relative path fails.

To Reproduce Steps to reproduce the behavior:

1. Download and extract the attached solution
2. Start TextCentric
3. Open ProjectWithChildRelativePathsWorksFine.Debug.nunit and run all tests -> works fine
4. Open TestMain\ProjectWithParentRelativePathFailsToLoad.Debug.nunit -> fails to load tests

Issue1077.zip

Expected behavior Parent relative paths shall be properly resolved by TestCentric (if that is the culprit) or NUnit (if that is the culprit), TestMain\ProjectWithParentRelativePathFailsToLoad.Debug.nunit shall work fine as well.

Screenshots

Environment:

• TestCentric Version 1.6.4
• OS: Win10
• Target Runtime for your test assembly: .NET Framework 4.8

[CharliePoole]

@maettu-this What version of nunit-project-loader are you using?

[maettu-this]

@maettu-this What version of nunit-project-loader are you using?

3.7.1

[CharliePoole]

Strange! The properties dialog shows the correct location for the dll and the package settings are correct. Yet it cannot be loaded!

[CharliePoole]

OK, this was a bit obscure but makes sense now...

You can use your project from a subdirectory with the following changes:

<NUnitProject>
  <Settings activeconfig="Debug" appbase=".." />
  <Config name="Debug" binpathtype="Auto">
    <assembly path="TestLib\bin\Debug\TestLib.dll" />
    <assembly path="TestMain\bin\Debug\TestMain.dll" />
  </Config>
</NUnitProject>

It turned out it was failing because it couldn't find the nunit.framework assembly, even though it was in the same directory as the test lib. .NET 4.x looks for dependencies in the appbase and the probing path. The appbase, in this case is the location of the .nunit file. The probing path directories (called binpath in the project settings) must be under the appbase and that was not the case here. I changed it to specify .. as the appbase and modified paths accordingly.

The use of binpathtype="Auto" tells the project loader "You figure it out" but that wasn't possible since the referenced assemblies were not under the appbase. This is explained fairly well here: https://docs.nunit.org/articles/nunit/technical-notes/usage/NUnit-Project-XML-Format.html

[maettu-this]

I understand. While https://docs.nunit.org/articles/nunit/technical-notes/usage/NUnit-Project-XML-Format.html indeed documents this, I neither find the behavior obvious nor convenient.

Could there be a way to automatically determine the appbase? E.g. if all paths are ..\TestXY the NUnit project loader could figure out the appbase probably is ... But not knowing the details behind, I don't know whether there is any way to make this more convenient.

And just for curiosity, would such feature had to be requested at https://github.com/nunit/nunit-project-loader or is it rather the way TestCentric interprets the data provided by the project loader?

[CharliePoole]

@maettu-this

The appbase has long been documented to be the location of the project file, with any relative paths taken relative to that location. The format is too old for a breaking change like that.

The real issue is the probing path. As you may know, .NET follows some simple rules for looking in other directories. I wrote the code we use to figure it out automatically based on the assembly paths in a config but it was almost 20 years ago and I can't remember it very well. I think I looked at the common partial path containing all assemblies, which doesn't work for your example.

The loader gives testcentric info in the package. TC gives .NET info from the package. .NET does the loading. For it to work any other way would be a complete change of architecture, so not likely unless we write a new framework and runner. One possible mitigation would be to give a warning or error when a path contains ... That's legal if the assembly has no dependencies in the directory, but it might save trouble to advise against it.

[maettu-this]

Having TC giving a comprehensive warning sounds like a good approach.