RFC: Performance Benchmarks and Tests

Discussions about how to benchmark and what to benchmark and which tools to use.

The more I look at it, the more I come to the conclusion that there cannot be one catchall tool for benchmarking such a complex beast as Neos. Just as with tests and code quality performance must be checked on different levels of integration and ideally with specialised tools.

I have identified the following "areas" for now that I would at least in mind treat differently:

database queries

Benchmarking databases is a highly specialized topic and I don't think code based benchmarks will do this too well unless it's really low level. Additionally DB performance is dependent on concurrency and some problems only come to light at higher concurrency, so this benchmark must include parallel execution, so either we create a thin wrapper around something like paratest or reactphp or rely on more low level tools for this. mysqlslap is quite nice and comes with a typical mysql installation (also mariadb) and can laod everything from files as well. So we could generate testcases (files) from php and then execute them via mysqlslap to get numbers in concurrent execution.

simple PHP code

I call it simple, meaning anything that can run by just having an autoloader and putting classes together. This applies to the CR Core as well as to a bunch of low level Flow code (think PositionalArraySorter). A benchmark for the NodeTypeManager would be a good example here I guess. This is probably were phpbench shines and with a little wrapper to load composer autoload this is good to go. Very similar to unit tests.

complex PHP code

This is anything that more or less needs Flow as a framework booted. Injections, ObjectManager, Configuration. From here it gets complicated. PHPBench is probably still our best bet, but just as for functional tests we need a big wrapper around it, to prepare everything and cleanup afterwards. Persistence is another topic on top.

full stack

We could consider starting the php buildin server and benchmarking requests against (test) controllers? The question is if this will give us any sensible information at all. I would wait with this even if on some level it's probably easiest to implement.

related: #4388

neos / neos-development-collection