cs-util-com / cscore

cscore is a minimal-footprint library providing commonly used helpers & patterns for your C# projects. It can be used in both pure C# and Unity projects.
https://cs-util-com.github.io/cscore/
Apache License 2.0
195 stars 32 forks source link

Make sure the xUnit tests all run in parallel without problems #55

Closed cs-util closed 11 months ago

cs-util commented 3 years ago

Currently when running all xUnit tests at once not all tests run through successfully

Some tests fail since they depend on internal singleton constructs, testing generic interfaces that internally use injection and because of this are affected by the other running tests. If the interface of the components is not modified the only way to manage to run all tests at once successfully is to separate the tests into fully independent processes without any shared memory, I guess/hope this is possible in xUnit but did not invest time into figuring out how to tell xUnit to run all tests fully separate in memory

Another type of test are the performance tests which fail because they expect the CPU to not be busy with running hundreds of other tests in parallel, maybe there is something in xUnit that allows specifying that a test should only be executed if no other tests are running at the same time? So queuing all performance tests to execute them linearly at the very end of a test run?

One bonus if this is fixed and all tests run reliably through each time they are executed at once is that the Stryker mutation test system can be used again

cs-util commented 3 years ago

should be a little easier to finish now, most of the tests run now successfully when executed together

The maim problem still are the ones that use external dependencies like APIs, use internal singleton systems or persist data to a non test specific location

cs-util commented 11 months ago

Now moved all tests that have external dependencies or state that is shared between multiple tests into an integrationTest namespace and excluded them from being used by Stryker so that Stryker only uses the real unit tests that always pass