Improve benchmark fidelity

[hbgl]

There are a couple of issues that I found while scanning through the benchmark code. I do not have time to create a pull request right now, but, I can give you some improvement suggestions in this issue.

1. Isolate test cases All benchmarks are run in a single process one after another. The problem with this approach is that the test cases effect each other's performance. For example, it makes a difference if a test is being run first or last. You can use a benchmarking library like phpbench to fix that issue.

2. Remove randomness There are calls to rand in the tests that can affect performance. A test cases should be as static as possible to improve repeatability. Instead of matching a random route, you can match the first, middle and last route.

3. Improve test routes All test routes follow a simple pattern and have the same prefix. This can skew benchmark results for libraries that can exploit this unrealistic configuration. Instead, it would be more appropriate to use a set of routes that one might realistically encounter in real life applications. For example, these are the routes that I use in my own benchmarks: https://github.com/hbgl/php-routing-bench/blob/main/routes/akaunting.php (taken from open source application Akaunting).

4. Make sure that tests play well with opcache This symfony test in particular does not work well with opcache and results in a 10x performance degradation. The problem is that the cache file is written for each run, which invalidates the opcode cache. Solution: Create cache files before a run in a different process.

Let me know if you need clarification on any of the points.

[alexdodonov]

@hbgl Sounds quite reasonable. Thanks for this issue. I shall try to implement first improvements ASAP.

[alexdodonov]

@hbgl Hi! I have implemented 1 and 2. So far so good )

[alexdodonov]

@hbgl Hi! Could you please explain #4?

[hbgl]

@alexdodonov I think the general idea was that you want to make the benchmarks as realistic as possible. This means that opcache should be enabled and all the relevant PHP files should be cached.

Those two sections of code cause a severe performance degradation, which I believe are related to opcache caching: https://github.com/alexdodonov/mezon-router-benchmark/blob/92aa900940bde8848f683be4801335d60c321815/tests/symfony/single-request/compiled-url-matcher-1000.php#L12-L14 https://github.com/alexdodonov/mezon-router-benchmark/blob/92aa900940bde8848f683be4801335d60c321815/tests/symfony/single-request/compiled-url-matcher-1000.php#L19-L21

Try the following: 1) Run the Symfony benchmarks as is. Note the results. 2) Comment out these two sections and run the benchmarks again. I am seeing a 5x improvement here.

I think the solution here is to first have a preprocessing step that creates the cache files in one process and then to run the benchmarks in a new process.

[alexdodonov]

Hi! Looks like now results will be more relevant. Try:

composer benchmark2

[alexdodonov]

Looks like everythong OK now. Please feel free to reopen this ussue if necessary.