How to clone a Container to avoid initialization cost for each test case

[erdalsivri]

We create/initialize a separate Container for each test case to be able to completely isolate tests. We've tried using a single shared container but Singleton registrations cause leaks between tests (we were able to use a shared Container for some of the tests and these tests did execute faster).

What we need is a very fast Container "clone" operation so that we can keep using Container per test but still have fast tests. This is probably not feasible but I still wanted to create this bug because maybe you'll have another suggestion for us. Maybe there is a better way to isolate tests and still use a single container. Note that we do run each test case in a separate AsyncScope but as I mentioned above, Singleton objects cause issues.

[dotnetjunkie]

We create/initialize a separate Container for each test case to be able to completely isolate tests.

Agreed, having a single container instance to maximize isolation is the best approach to take.

Maybe there is a better way to isolate tests and still use a single container.

No, having a single container instance per test makes the test suite more reliable, which is what you should strive for.

What we need is a very fast Container "clone" operation so that we can keep using Container per test but still have fast tests.

Simple Injector's registration process is typically pretty fast, because there's not a lot of verification and checking going on during that time. That means that a container-provided 'clone' operation wouldn't be much faster compared to creating a new container again.

What will, however, slow down the performance of your tests tremendously is if you leave auto verification enabled during testing. Auto verification is enabled by default and ensures that the complete configuration is verified upon the first performed resolve. But verification is very expensive -especially if its performed per test on a big container instance- as it causes the generation of expression trees, compilation of IL, and jitting for all registrations in the container.

That is why, when it comes to unit and integration testing, I advise switching this feature off. This can be done as follows:

var container = new Container();
container.Options.EnableAutoVerification = false;

// This is the same as 'clone'
var container = MainAppBootstrapper.InitializeContainer(container);

// optionally replace some dependencies for testing
container.Options.AllowOverridingRegistrations = true;
container.RegisterInstance<IUserContext>(new FakeUserContext());

// NOTE: do NOT call container.Verify() in your test suite, because that's basically the same
// as when leaving Options.EnableAutoVerification  = true.

return container;

I hope this helps

[erdalsivri]

Thanks Steven for the prompt response! We do disable AutoVerification. During my profiling sessions, I've seen some SimpleInjector frames so assumed it could be due to registration but I guess I should dig deeper. We have around 7000 test cases, which take almost 7 minutes on a 16 core machine. We identified some EFCore-related issues and optimized those but our tests are still a lot slower than they should be. Anyways, thanks for eliminating container initialization from our list of things that might be causing slowing.

[dotnetjunkie]

We have around 7000 test cases, which take almost 7 minutes on a 16 core machine.

That's about 1 test per second per core. That is really slow. I can imagine that causing problems, especially when you're dealing with unit tests, which should be very fast. But if they're all integration tests and often hit the database, 1 second might be reasonable (although I'd likely want to see better performance on them).

The performance issues you are having could be Simple Injector related, especially when the you have hundreds or thousands of registrations, but this should be straightforward for you to test. And it's best to do this without any profiling tools. These can easily get in the way and measure the wrong thing. Don't get me wrong, I think those profiling tools are tremendously useful when it comes to pinpointing hotspots, but they can also take too much overhead skewing the results.

So instead, wrap the code that creates the new container in a var sw = Stopwatch.StartNew() and sw.Elapsed calls. Perhaps you can sum the total amount spent for all created container instances for all tests in your suite. After 7 minutes you'll know the total amount of time spent creating new container instances. Divided by 7,000 and you'll know whether or not this part is an issue or not.

Let me know what you find. If this proves to be problematic, we'll go back to the drawing board and find a solution that is more optimized; for instance by constructing a clearly constructed container instance that can be reused by tests.

[erdalsivri]

Our unit tests do not hit an external database. We use in-memory sqlite in tests. However, we do have thousands of registrations for what we call handlers in our codebase. I am not sure if it matters but these are generic classes and have decorators too. Here is a screenshot from a profiling session:

This screenshot shows the cost breakdown for the constructor of the test, which does SI registrations for all the handlers. This particular test has about 30 test cases so 5.5s is for 30 calls. This is not a huge cost but we create and destroy containers so it adds up.

I am wondering if there is a way to cache these expression (BuildExpressionInternal) or something else to a similar effect.

I will get add some manual instrumentation with Stopwatch but wanted to share this profiling session in the meantime.

[dotnetjunkie]

I added an Integration Testing Guide to the Simple Injector documentation. It talks about unit vs integration testing and performance considerations when using a DI Container, and shows a way to reuse the same container instance for the entire test suite. I considered adding information about using a container pool, but that solution is likely only more complex compared to using a single container, while being slower, and providing no additional benefits.

Let me know whether that information is helpful.

[erdalsivri]

Thanks Steven, it is definitely helpful!