Asynchronous tests concurrency problems.

[sergey-bulavskiy]

Hello, first of all i want to thank you for this great project, this is a brilliant idea and it works incredibely fast.

Maybe i'm using Respawn incorrectly, but i'm currently having problems with concurrent xunit tests, as there are no way to specify maximum number of parallel tests in xunit (only number of threads), tests are running asynchronously.

I get the problem that: Test X:

1. Creates entity in table A
2. .... doing some stuff
3. Check that entity A exist in the database.

Test Y:

1. Creates entity in table A
2. ... doing some other stuff.
3. Check that entity A exist in the database.

Sometimes tests failing due to the reason that one test cleaned up the database earlier than second test reached the assertion.

I've tried to manage it by introducing multiplie databases (3 for example) and a semaphore to block new tests from receiving connection string to a database that's currently being used by another test, but still i get some inconsistency in some cases, specifically when tests are in separate test projects. I'm just wandering, maybe i'm missing something, or if i assume correctly that each time ResetAsync resets database to a state when respawner was created - we can possibly clean up data for other concurrent tests so no parallel tests execution possible with the Respawn.

Can somebody clarify my thoughts regarding this? Thanks in advance.

[keithn]

you can stop xunit running things in parallel by putting a collection attribute on the test class [Collection("Anything with this name won't be run in parallel")]

[sergey-bulavskiy]

@keithn so it is intended way to use Respawn, only with sequential tests execution?

[keithn]

I can't see how you'd expect tests against a single state store to work?

[sergey-bulavskiy]

@keithn indeed, i thought that somebody faced same problems previously and managed it somehow, but i'll stick to sequential runs, thank you.

[jbogard]

Other ways you can address this is with Dev Containers or separate Local DB instances. But using this library in those cases has diminishing returns of course.

[Insire]

I've a similar use case for Respawn, although the setup seems to be a bit more involved than yours @sergey-bulavskiy .

I create an inmemory pool of databases, that i also manage in a separate ms sql server db(TestManagementDb). Each test run creates its own set of up to 16 databases. Each time, a test needs a database, i ask my pool for an instance. Each instance knows if there is a corresponding db already created(raw sql), whether migrations(EF Core) have been run on it and whether it has been seeded with default data(EF Core).

If anything of that is still missing, i'll fix that before returning the database as EF Core dbcontext to my test. Then i run my test logic as usual and then dispose it via IAsyncDisposable, which in turn uses Respawn to clean up the data in the database and also update its state in my TestManagementDb until finally i return the database back to the pool.

Then the next test can fetch an instance from the pool. I also use Semaphore slim here.

This approach is incredibly fast, provided you can figure out the locking around the pool and how to deal with failing and cancelled tests. In my case, when all tests are done, i drop all of my 16 databases and remove the corresponding entries from my TestManagementDb aswell.

[sergey-bulavskiy]

Hello, @Insire! Thanks for you experience, can you please clarify a couple of moments? What test framework are you using? Are your tests all in one project? I was trying to achieve concurrency via semaphore slim and database pool via xunit fixtures but failed miserably because of xunit tricky parallel execution of the tests, and xunit fixtures being project-wide at maximum. To correctly support pool for multiple projects we'd need to use Assembly-level fixture which is not supported yet.

[Insire]

Hello, @Insire! Thanks for you experience, can you please clarify a couple of moments? What test framework are you using? Are your tests all in one project? I was trying to achieve concurrency via semaphore slim and database pool via xunit fixtures but failed miserably because of xunit tricky parallel execution of the tests, and xunit fixtures being project-wide at maximum. To correctly support pool for multiple projects we'd need to use Assembly-level fixture which is not supported yet.

I am using x-unit with https://github.com/JDCain/Xunit.Extensions.AssemblyFixture which does add the assembly level fixture you mentioned.

I have multple projects

• unit tests (no database involved here)
• integration tests (the majority of our tests use ms sql server as database here)
• performance tests (a form of integration test)
• smoke tests (a form of integration test)
• bdd tests (a form of integration test)
• ui tests (a form of integration test)
• bdd-ui tests (a form of integration test)

[sergey-bulavskiy]

Thank you very much, @Insire, will have a look on this assembly fixture!