Missing possibility to retrieve whether a test is running in parallel (formerly ...retrieve a test's resulting `ParallelScope`)

maettu-this commented 4 months ago

Retrieving a test's resulting ParallelScope combined from assembly, fixture and test level Parallelizable and NonParallelizable attributes missing?

I have a test which shall slightly change its behavior depending on whether it is executing in parallel with other tests. Pseudo code:

if (executing in parallel)
    use dynamic ephemeral port to run local socket test
else
    use fixed default port to run local socket test

Rationale:

CI/Standard use case: Running multiple socket tests in parallel requires servers to use different local ports.
Devel/Debug use case: Running single socket test is easier to debug when using fixed port.

As Parallelizable and NonParallelizable can be applied to assembly, fixture or test level, I have tried finding a way to retrieve the resulting ParallelScope:

Searched documentation and issues.
Tried to retrieve from TestContext as well as as internal TestExecutionContext.

See attached project which trace outputs the ParallelScope based on the test's Properties. However, only the test's attributes are replicated in the Properties, not the resulting ParallelScope combined from assembly, fixture and test level. Is this information available elsewhere?

RetrieveParallelScope.zip

grafik

maettu-this commented 4 months ago

Also tried TestExecutionContext.CurrentContext.ParallelScope but that returns ParallelScope.Default in all cases expect TestFixtureParallelizable.TestParallelizable where ParallelScope.Children is returned.

And there is no TestExecutionContext.CurrentContext.CurrentText.ParallelScope that could hold the value for the current test.

OsirisTerje commented 2 months ago

@manfred-brands Comments? Could we add this?

OsirisTerje commented 2 months ago

@maettu-this Have you tried walking up the tree and check the ParallelScope of each level ?

Also, what you "can" retrieve is the wanted ParallelScope, but that says what you wants, not actually what happens. What happens depends on more factors than the ParallelScope alone. (Sure you know this, but just mentioning it.)

PS: Added your repro here: https://github.com/nunit/nunit.issues/tree/main/Issue4707

OsirisTerje commented 2 months ago

I updated the repro, and noticed that we have the parent property just now, hopefully it will make it into 4.2.0

Anyway, I used an alpha version alpha.33, it is in the repro project now. The result then is as follows (added some tracelines for the parents, two levels up)

And, the same alpha also has a method for listing all values of a given property over the whole tree, look it up AllPropertyValues

maettu-this commented 2 months ago

@OsirisTerje,

depends on more factors than the ParallelScope alone

is exactly what I am struggling with. The intention of NUnitEx.CurrentTestIsExecutingParallel at https://github.com/nunit/nunit.issues/blob/main/Issue4707/NUnitEx.cs is to return the effective way a test is being executed.

I guess I should have called this issue "Missing possibility to retrieve whether a test is running in parallel" rather than "Missing possibility to retrieve a test's resulting ParallelScope". I'll fix this.

OsirisTerje commented 2 months ago

Please take into the account that the dotnet testhost sends the adapter one assembly at a time, and then it may (or may not) send those in parallel. Each will live in its own process, and there is no way these can know about it each other, so one test assembly don't know if it runs in parallel with another. The testhost however, should know that.

Also, NUnit may try to start multiple threads, but that again is limited, so a test may be waiting for a free thread to run.

Feel free to dig into the code in NUnit that handles this, but again, you can only know what happens within a single assembly. Look up ParallelQueue and ParallelSTAQueue. If your test case is in one of these, they probably will run in parallel. It might be possible to add something around this code to the context.

maettu-this commented 2 months ago

I see, indeed not that easy, especially given in my case even multiple assemblies need to use a dynamic ephemeral port if executed in parallel.

Maybe there's an alternative approach, if it is possible to let a test "know" whether it is the only test that got started by a runner, I could do this:

if (just a single test has been started)
    use fixed default port to run local socket test
else
    use dynamic ephemeral port to run local socket test

This would cover my use case well-enough.

OsirisTerje commented 1 month ago

@maettu-this In the static TestContext class, there is a property WorkerId?. If this is non-null you see the thread worker id for the current test, which means it [probably] runs in parallel, whereas if it is null, it is not running in parallel. Can you try to test that?

maettu-this commented 1 month ago

@OsirisTerje, I have tried this with the repro project. Findings:

The worker ID, containing e.g. "NonParallel", can be used to determine whether a test is potentially running in parallel.
But it cannot be used to determine whether a test indeed is running in parallel with others:

grafik

Even when running just a single test, the worker ID still is named the same as if the test was running in parallel.

RetrieveParallelization.zip

OsirisTerje commented 1 month ago

Do you see any cases where it is null => non-parallel?

maettu-this commented 1 month ago

Nope, the worker always has a name, either NonParallelWorker or ParallelWorker#I where is I is 1, 2, 3,...

The previsouly attached repro project is .NET 4.8 Framework and NUnit 3.14.0. I have now also verified this with .NET 6 and NUnit 4.1.0:

NUnit Adapter 4.6.0.0: Test execution started
Running selected tests in <Workspace>\bin\Debug\net6.0\RetrieveParallelization.dll
   NUnit3TestExecutor discovered 6 of 6 NUnit test cases using Current Discovery mode, Non-Explicit run
This is RetrieveParallelization.TestFixture.TestNonParallelizable:
    Worker ID = NonParallelWorker
    Worker is non-parallel
This is RetrieveParallelization.TestFixture.TestParallelizable:
    Worker ID = ParallelWorker#1
    Worker is parallel
This is RetrieveParallelization.TestFixtureNonParallelizable.TestNonParallelizable:
    Worker ID = NonParallelWorker
    Worker is non-parallel
This is RetrieveParallelization.TestFixtureNonParallelizable.TestParallelizable:
    Worker ID = ParallelWorker#16
    Worker is parallel
This is RetrieveParallelization.TestFixtureParallelizable.TestNonParallelizable:
    Worker ID = NonParallelWorker
    Worker is non-parallel
This is RetrieveParallelization.TestFixtureParallelizable.TestParallelizable:
    Worker ID = ParallelWorker#5
    Worker is parallel
NUnit Adapter 4.6.0.0: Test execution complete

Running just a single test:

NUnit Adapter 4.6.0.0: Test execution started
Running selected tests in <Workspace>\bin\Debug\net6.0\RetrieveParallelization.dll
   NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Non-Explicit run
This is RetrieveParallelization.TestFixtureParallelizable.TestParallelizable:
    Worker ID = ParallelWorker#15
    Worker is parallel
NUnit Adapter 4.6.0.0: Test execution complete

RetrieveParallelization[.NET6].zip

nunit / nunit

Missing possibility to retrieve whether a test is running in parallel (formerly ...retrieve a test's resulting `ParallelScope`) #4707