--Explore Does Not Appear To Respect --Agents

[aolszowka]

UPDATE (10/3/2024) This quite old bug report has never been addressed. I'm calling it an idea for a new feature, since that's actually what it is.

Reproduce with the following:

nunit3-console.exe @TestAssemblies.txt

Where TestAssemblies.txt is:

--explore=Tests.xml;format=nunit3
--agents=8
TestAssembly1.dll
TestAssembly2.dll
TestAssembly3.dll
.
.
.
.
TestAssembly400.dll

This probably could be reproduce with a smaller number of assemblies, but a small test case seems to "hide" this error.

Looking at this in Process Explorer will show an Agent be spun up for each of the 400 test assemblies:

Eventually the agents will consume all available system resources and die with a resource related exception (in this case it died with a System.IO.FileLoadException due to the paging file being too small for the operation to complete).

Currently I am attempting to evaluate several hundred NUnit Tests (all of the above 400 are NUnit Test Assemblies) to filter to a particular category, to figure out which of the 400 assemblies contain the tests I am interested in.

[ChrisMaddock]

Good find @aolszowka, this clearly isn't practical.

Verified by looking at the code in AggregatingTestRunner - we need an equivalent of RunTestsInParallel for Explore. The only additional complexity over RunTestsInParallel is that I think we should try and keep the results in order, for exploring. (I think TestPackage SubPackages are ordered, from memory.)

A PR would be welcome for this, if you're interested? 🙂

[ChrisMaddock]

@aolszowka Just reaslied we've never offered a workaround - adding the --inprocess flag should do the job, and I don't think will have any impact on performance either.