TRPB
commented
9 years ago Well spotted. Test one should be the time it takes from after configuration to creating the object graph. However, I think you're right that it's inconsistent... and it's going to differ depending on implementation.
The difference between test 1 and test 2 should be that in test 2, any files the container needs should already be loaded. Test 1 should include the time it takes the container to autoload any required files, test 1 should not. I'll update the tests accordingly.
Having said that, now you've pointed it out, it would probably make more sense to start the timer at the very top of the script, as a container that loads all its files in its constructor will score higher than one that loads them as it needs them, which, as a metric, isn't very useful.
As per #19, I don't know precisely what these benchmarks were supposed to measure, but again, they appear to be inconsistent.
pimple/test1.php starts the timer immediately before entering the loop.
aura/test.php on the other hand, starts the timer before creating and configuring the container.
Maybe you should add an actual specification, prescribing what is supposed to be benchmarked, and how?
Or at least, add a "Contributing" headline to the README and designate one of the benchmarks as point of reference?