Benchmark Improvements

[adambrett]

Hi,

I originally just wanted to test adding my little router package to see how it compared, but when it totally blew all the others away I knew something was wrong because I wrote mine with absolutely no consideration for speed.

A bit on investigation showed that the issue was the way in which things are dispatched, and that my router does the matching when routes are added, and then discards anything else immediately. That lead me to the conclusion that a fairer comparison is the assembly of the router as well as dispatch process, that lead to the following figures:

Test Name	Time	+ Interval	Change
AdamBrett\Router - unknown route (1000 routes)	0.0313445032	+0.0000000000	baseline
AdamBrett\Router - last route (1000 routes)	0.0320222467	+0.0006777436	2% slower
FastRoute - last route (1000 routes)	0.1440973312	+0.1127528280	360% slower
FastRoute - unknown route (1000 routes)	0.1482576191	+0.1169131160	373% slower
Aura v2 - unknown route (1000 routes)	0.1633502930	+0.1320057899	421% slower
Aura v2 - last route (1000 routes)	0.1638579160	+0.1325134128	423% slower
Pux PHP - unknown route (1000 routes)	0.2488152713	+0.2174707681	694% slower
Pux PHP - last route (1000 routes)	0.2528938979	+0.2215493947	707% slower
Symfony2 - last route (1000 routes)	0.4613442034	+0.4299997002	1372% slower
Symfony2 - unknown route (1000 routes)	0.4683836937	+0.4370391905	1394% slower

Clearly this still isn't right, as FastRoute should be far faster than my package, which lead me to separate out the Dumped Symfony2 router, and write some rudimentary caching for the other routes (so the impact of building the router becomes negligible), which lead to the following figures:

Test Name	Time	+ Interval	Change
Symfony2 Dumped - unknown route (1000 routes)	0.0011271983	+0.0000000000	baseline
Symfony2 Dumped - last route (1000 routes)	0.0011997283	+0.0000725299	6% slower
FastRoute Cached - last route (1000 routes)	0.0056380630	+0.0045108646	400% slower
FastRoute Cached - unknown route (1000 routes)	0.0058212489	+0.0046940505	416% slower
AdamBrett\Router - last route (1000 routes)	0.0299270600	+0.0287998617	2555% slower
AdamBrett\Router - unknown route (1000 routes)	0.0303771615	+0.0292499632	2595% slower
Pux PHP - last route (1000 routes)	0.0309612066	+0.0298340082	2647% slower
Pux PHP - unknown route (1000 routes)	0.0314295948	+0.0303023964	2688% slower
Aura v2 - last route (1000 routes)	0.0978639692	+0.0967367709	8582% slower
Aura v2 - unknown route (1000 routes)	0.1023280859	+0.1012008876	8978% slower

As the way my router is designed means it's impossible to cache I have left it uncached. I suspect that Symfony comes out so much faster because it's writing a PHP class rather than serializing and unserializing the router object.

I guess this shows a few things:

1) The rankings are slightly different from the original benchmark because the way routes are assembled has an impact 2) It's important to cache your router in production

[tyler-sommer]

Hi @adambrett. Thanks for taking the time to check this out!

Your router is extremely interesting, though, as you stated since it cannot be cached and must be "built" each time it doesn't quite fit with the existing tests. The implementation might, however, offer some potential advantages that the others do not. I really like the idea of a complete, perhaps even un-optimized (ie. we remove the symfony2 optimized test from this benchmark) benchmark.

What I've done is I've heavily refactored how the benchmarks are setup, and I added support for multiple benchmarks using a BenchmarkCollection. If you have some time, check out the latest changes in master. Hopefully the changes I've made will help make the project more maintainable, and more importantly, allow us to create multiple benchmarks and have them exist simultaneously.

I would love it if you wanted to add a third, new benchmark that tests the complete operation from start to finish for each router, but even if not, thanks for contributing!

[adambrett]

Thanks, I'll take a look over the next couple of days and see what I can do.