Missing possibility to retrieve whether a test is running in parallel (formerly ...retrieve a test's resulting `ParallelScope`)

maettu-this commented 6 months ago

Retrieving a test's resulting ParallelScope combined from assembly, fixture and test level Parallelizable and NonParallelizable attributes missing?

I have a test which shall slightly change its behavior depending on whether it is executing in parallel with other tests. Pseudo code:

if (executing in parallel)
    use dynamic ephemeral port to run local socket test
else
    use fixed default port to run local socket test

Rationale:

CI/Standard use case: Running multiple socket tests in parallel requires servers to use different local ports.
Devel/Debug use case: Running single socket test is easier to debug when using fixed port.

As Parallelizable and NonParallelizable can be applied to assembly, fixture or test level, I have tried finding a way to retrieve the resulting ParallelScope:

Searched documentation and issues.
Tried to retrieve from TestContext as well as as internal TestExecutionContext.

See attached project which trace outputs the ParallelScope based on the test's Properties. However, only the test's attributes are replicated in the Properties, not the resulting ParallelScope combined from assembly, fixture and test level. Is this information available elsewhere?

RetrieveParallelScope.zip

grafik

maettu-this commented 6 months ago

Also tried TestExecutionContext.CurrentContext.ParallelScope but that returns ParallelScope.Default in all cases expect TestFixtureParallelizable.TestParallelizable where ParallelScope.Children is returned.

And there is no TestExecutionContext.CurrentContext.CurrentText.ParallelScope that could hold the value for the current test.

OsirisTerje commented 4 months ago

@manfred-brands Comments? Could we add this?

OsirisTerje commented 4 months ago

@maettu-this Have you tried walking up the tree and check the ParallelScope of each level ?

Also, what you "can" retrieve is the wanted ParallelScope, but that says what you wants, not actually what happens. What happens depends on more factors than the ParallelScope alone. (Sure you know this, but just mentioning it.)

PS: Added your repro here: https://github.com/nunit/nunit.issues/tree/main/Issue4707

OsirisTerje commented 4 months ago

I updated the repro, and noticed that we have the parent property just now, hopefully it will make it into 4.2.0

Anyway, I used an alpha version alpha.33, it is in the repro project now. The result then is as follows (added some tracelines for the parents, two levels up)

And, the same alpha also has a method for listing all values of a given property over the whole tree, look it up AllPropertyValues

maettu-this commented 4 months ago

@OsirisTerje,

depends on more factors than the ParallelScope alone

is exactly what I am struggling with. The intention of NUnitEx.CurrentTestIsExecutingParallel at https://github.com/nunit/nunit.issues/blob/main/Issue4707/NUnitEx.cs is to return the effective way a test is being executed.

I guess I should have called this issue "Missing possibility to retrieve whether a test is running in parallel" rather than "Missing possibility to retrieve a test's resulting ParallelScope". I'll fix this.

OsirisTerje commented 4 months ago

Please take into the account that the dotnet testhost sends the adapter one assembly at a time, and then it may (or may not) send those in parallel. Each will live in its own process, and there is no way these can know about it each other, so one test assembly don't know if it runs in parallel with another. The testhost however, should know that.

Also, NUnit may try to start multiple threads, but that again is limited, so a test may be waiting for a free thread to run.

Feel free to dig into the code in NUnit that handles this, but again, you can only know what happens within a single assembly. Look up ParallelQueue and ParallelSTAQueue. If your test case is in one of these, they probably will run in parallel. It might be possible to add something around this code to the context.

maettu-this commented 3 months ago

I see, indeed not that easy, especially given in my case even multiple assemblies need to use a dynamic ephemeral port if executed in parallel.

Maybe there's an alternative approach, if it is possible to let a test "know" whether it is the only test that got started by a runner, I could do this:

if (just a single test has been started)
    use fixed default port to run local socket test
else
    use dynamic ephemeral port to run local socket test

This would cover my use case well-enough.

OsirisTerje commented 3 months ago

@maettu-this In the static TestContext class, there is a property WorkerId?. If this is non-null you see the thread worker id for the current test, which means it [probably] runs in parallel, whereas if it is null, it is not running in parallel. Can you try to test that?

maettu-this commented 3 months ago

@OsirisTerje, I have tried this with the repro project. Findings:

The worker ID, containing e.g. "NonParallel", can be used to determine whether a test is potentially running in parallel.
But it cannot be used to determine whether a test indeed is running in parallel with others:

grafik

Even when running just a single test, the worker ID still is named the same as if the test was running in parallel.

RetrieveParallelization.zip

OsirisTerje commented 3 months ago

Do you see any cases where it is null => non-parallel?

maettu-this commented 3 months ago

Nope, the worker always has a name, either NonParallelWorker or ParallelWorker#I where is I is 1, 2, 3,...

The previsouly attached repro project is .NET 4.8 Framework and NUnit 3.14.0. I have now also verified this with .NET 6 and NUnit 4.1.0:

NUnit Adapter 4.6.0.0: Test execution started
Running selected tests in <Workspace>\bin\Debug\net6.0\RetrieveParallelization.dll
   NUnit3TestExecutor discovered 6 of 6 NUnit test cases using Current Discovery mode, Non-Explicit run
This is RetrieveParallelization.TestFixture.TestNonParallelizable:
    Worker ID = NonParallelWorker
    Worker is non-parallel
This is RetrieveParallelization.TestFixture.TestParallelizable:
    Worker ID = ParallelWorker#1
    Worker is parallel
This is RetrieveParallelization.TestFixtureNonParallelizable.TestNonParallelizable:
    Worker ID = NonParallelWorker
    Worker is non-parallel
This is RetrieveParallelization.TestFixtureNonParallelizable.TestParallelizable:
    Worker ID = ParallelWorker#16
    Worker is parallel
This is RetrieveParallelization.TestFixtureParallelizable.TestNonParallelizable:
    Worker ID = NonParallelWorker
    Worker is non-parallel
This is RetrieveParallelization.TestFixtureParallelizable.TestParallelizable:
    Worker ID = ParallelWorker#5
    Worker is parallel
NUnit Adapter 4.6.0.0: Test execution complete

Running just a single test:

NUnit Adapter 4.6.0.0: Test execution started
Running selected tests in <Workspace>\bin\Debug\net6.0\RetrieveParallelization.dll
   NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Non-Explicit run
This is RetrieveParallelization.TestFixtureParallelizable.TestParallelizable:
    Worker ID = ParallelWorker#15
    Worker is parallel
NUnit Adapter 4.6.0.0: Test execution complete

RetrieveParallelization[.NET6].zip

puagarwa commented 2 days ago

Do we have any way to just log or print the parallelism degree used while executing test suite ?

OsirisTerje commented 2 days ago

@puagarwa Doesn't any of the methods above do this for you?

puagarwa commented 2 days ago

@puagarwa Doesn't any of the methods above do this for you?

I went through the thread but my requirement is a bit different. I just want to know how many workers are running when i run my test run.

I could try to play around with WorkerId as suggested above, but not sure if this can give me what i am looking for.

nunit / nunit

Missing possibility to retrieve whether a test is running in parallel (formerly ...retrieve a test's resulting `ParallelScope`) #4707