Test DLL are ran with incorrect AppContext.BaseDirectory in netcore 3.1.

[Kuinox]

Hello, I tried to run our netcore 3.1 tests, a lot of them passes, but the ones relying on working directory AppContext.BaseDirectory path fails. These tests pass using dotnet test.

Edited issue and issue title: I'm not using the working directory but the AppContext.BaseDirectory.

[CharliePoole]

There's a lot to say here, so

TLDR;

• You appear to have broken an NUnit rule about the working directory.
• Using dotnet test probably got you into the habit.
• Maybe we can fix. A lot depends on how many other people run into this issue, so if you're reading this and running into the same thing please comment!
• There are workarounds.

CONFIRMATION

It looks as if you are starting with the current directory, either explicitly or by use of a relative path and then trying parent folders to see where the solution is located. Can you confirm that? Can you supply the simplest possible test case we can add to the engine tests to catch this error?

THE "RULE"

In NUnitV2, the working directory (current directory) was always set to the directory containing the each test assembly. This was very convenient for users, of course, so they got into the habit of relying on it.

With NUnit 3, we introduced parallel execution of test assemblies as well as parallel tests within each assembly. Because it was possible to run multiple assemblies in the same process, we could no longer guarantee this. We might set the working directory when the first assembly began execution and then change it when a second assembly started, while the first one was still running!

So we established the policy that NUnit would never change the working directory in its code. That seems like a good policy, in fact, and I'd prefer to stick with it if possible. I don't think applications should change the CD while they run.

It took a while for NUnit V2 users to adjust to this change in NUnit 3. We gave them the TestDirectory property of TestContext so they could retrieve the location of the current test directory within a test and build an absolute path relative to that directory. Lots of people had to change code but we arrived finally at a point where the engine and console runner - and now the testcentric GUI - are pretty independent of the working directory. You can run tests from a completely unrelated directory if you need to, for example, and it just works.

MEANWHILE IN THE DOTNET WORLD

... people got used to running one test assembly at a time from the directory containing that assembly. Since NUnit doesn't change that directory it would just work. The same thing happened if you used nuinitlite to run a single assembly.

SO WHAT BROKE?

Although NUnit never changes the current directory, it does launch an agent process, which ends up with it's own current directory. That's what you are seeing. I need a test case to verify this, but I believe the same thing would happen if your tests targeted the .NET Framework. As soon as you give me a reproducible case, I'll try that out.

POSSIBLE FIXES

Ordered from no change to biggeest change...

1. We could tell @Kuinox and anay other users impacted by this to bite the bullet and use TestDirectory to find out where their assembly is located.

2. The engine could explicitly set the current directory for each agent to its own current directory. That effectively keeps our promise not to "change" the current directory.

3. We could break compatibility and set the current directory when launching an agent with only one assembly or with multiple assemblies in the same directory. For consistency, I think we would need to do this for all runtimes, not just .NET Core.

@TestCentric/gui-team What do you think?

@ChrisMaddock @rprouse @jnm2 This same issue is bound to come up eventually in the console runner, so what do you guys think?

WORKAROUNDS

1. For each test or fixture, use TestDirectory to get the directory containing the test assembly.

2. Same as 1, but do it in a base class if you already have one for your tests.

3. Use an assembly-level SetUpFixtur to set the working directory based on TestAssembly. If you use this one, you can't run multiple assemblies in parallel, however.

FINALLY

In spite of all the words, I still need a simple test case. In fact, when I get it, I could easily change my mind about some of the above. :smiley:

[Kuinox]

Thanks for the detailed answer, Sorry, I made an error because i didn't know well our TestHelper . It doesn't use the Working Directory but the AppContext.BaseDirectory. For all the runners we are using and have used until now, the assembly is... where it is... It seems that TestCentric moves it before executing it...

I'll edit the issue to avoid more confusion.

[CharliePoole]

Ouch! All those wasted words. :smiley:

TestCentric doesn't move your assembly. Could you explain why it seems like it does? Could it be Shadow Copy that's biting you?

[immeraufdemhund]

🔵 LEAVE MY CURRENT DIRECTORY ALONE!!! 🔴 I've grown so accustomed to it and running in parallel that if you changed my working directory mid test that would break me.

😄

[CharliePoole]

Well, based on @Kuinox 's last comment, this may not actually be about WorkingDirectory, except as the BaseDirectory may be influenced by it.

Nevertheless, I'm going to start a separate issue about setting the WorkingDirectory for the agent, like this...

startInfo.WorkingDirectory = Environment.CurrentDirectory

Why? Because by not doing this, we are sort of breaking our promise not to mess with it. As the engine now works, a test run in an agent has a different working directory from one run in process! We should carry the same working directory from process to process.

Would that change mess you up?

[immeraufdemhund]

I explicitly set my directory when I need to to whatever TestContext.WorkingDirectory is when I need too

[CharliePoole]

Should be OK then. We would still not rely on the directory, but I think having the agent run in the same current directory as the user started with is probably reasonable.

IIRC, if you set the directory in one of your tests, NUnit sets it back to what it was after the test has run. But anyone who does that should avoid parallel execution.

[CharliePoole]

@Kuinox AppContext.BaseDirectory is doing exactly what it is supposed to do: giving the base directory of the main executable. In this case, the main executable is .../tools/agents/netcoreapp3.1/testcentric-agent.dll.

We may be able to compensate for this in the agent code. Still waiting for a code sample though!

[Kuinox]

This test is green in VSTest using NUnit3TestAdapter but red in TestCentric:

   [TestFixture]
public class RootTests
{
    static string ThisDirectory( [System.Runtime.CompilerServices.CallerFilePath]string file = null ) => Path.GetDirectoryName( file );
    [Test]
    public void tests_should_be_ran_where_they_are()
    {
        Assert.That( AppContext.BaseDirectory.StartsWith( ThisDirectory() + Path.DirectorySeparatorChar ) );
    }
}

[CharliePoole]

OK I can work with that. I'll try to make it work, but no guarantees.

The limitation is that you're using AppContext..BaseDirectory in a way that it's not intended to be used. It definitely is supposed to show where the base directory for the "App" is, but in this case that App is actually the test-agent. The reason it worked with dotnet test is that everything is in the same directory in that environment.

Have you checked whether the shadow copy option makes a diff?

BTW, with the fix I did to issue #577, you could switch to using CurrentDirectory and it should work if you started the GUI in the directory with the test assembly.

[CharliePoole]

Insofar as I can determine, we have no way to influence the value of AppContext.BaseDirectory. It will always be set to the running executable, which in this case is testcentric-agent.dll.

As I commented earlier, it's usable by you when running the dotnet CLI or vstest because all the different components end up in the same directory. That's also true when running NUnitLite but not when running the GUI.

I'm closing this issue but if you comment on it further we will still see and respond to your comments.

[CharliePoole]

@Kuinox Sorry it didn't work out but thanks for identifying the current directory bug, which is now fixed!

[Kuinox]

I really wanted to use the test centric, due to the VSTest GUI being unreliable.
Could the testcentric copy the agent next to the dll it test ?

[CharliePoole]

@Kuinox Needing to copy things into the directory containing your tests is kind of a hack and I'd rather not get into it. It took a lot of effort to make .NET Core tests run using runners located elsewhere and I don't want to reverse that. The tests run fine and the only problem is that your helper code makes an assumption that the BaseDirectory will always be where the tests are located.

Perhaps you could consider using NUnit's TestContext.CurrentContext.TestDirectory as a base for looking for the parent directory. It always designates the directory containing the test assembly.