Improve test performance

[robjwells]

Couple of changes to conftest.py to make the pytest run faster:

• Use an in-memory SQLite database for testing, rather than one saved in a temporary file.
• Cache the Flask application object per unique configuration dictionary.

The SQLite change seems to be a straightfoward performance win.

The Flask object caching I was a bit hesitant with, as my concern is that it could cause interference between tests. However, the app context is still pushed before each test and popped afterwards, so this change doesn’t appear to affect the correctness of the tests.

The dependency on the pyrsistent library is because the built-in lru_cache decorator only caches hashable objects, which the built-in dict is not. Caching in this way extends the performance benefit to all tests that parametrize on test_config. Initially I used the singleton pattern and cached the app the first time it was created, and any following test that had a different config would receive a completely new instance. But that approach means you’re relying on the first test to have a “common” config, whereas the first test could quite easily be the only test using a particular config, throwing away any performance benefit.

Here are some numbers, the result of running time pytest test 10 times consecutively on my laptop, so your mileage may vary somewhat.

With the times reported by pytest itself:

Scenario	Timed by	Mean (seconds)	% change on master	Median (seconds)	% change on master
Current master (d5a3e8b)	pytest	48.388		49.305
SQLite in-memory	pytest	28.269	-42%	27.810	-44%
SQLite in-memory & app caching	pytest	22.006	-54%	21.735	-56%

And from time, which suggests a slightly smaller improvement from the SQLite change:

Scenario	Timed by	Mean (seconds)	% change on master	Median (seconds)	% change on master
Current master (d5a3e8b)	time	53.241		54.195
SQLite in-memory	time	32.249	-39%	31.998	-41%
SQLite in-memory & app caching	time	25.177	-53%	24.509	-55%

The advantage of each change varies with the characteristics of a particular test. For example, I'm working on a branch at the moment with a lot of short tests with little database interaction, so for those most of the benefit comes from caching the Flask object.

[Polsaker]

Ohh those are some pretty good optimizations, good job!

[robjwells]

Thanks!