RFC: Implement performance benchmarks

[staabm]

after we finished propel 2.0 we should implement some performance benchmarks (in case we have enough time, we could also created them before, to compare e.g. Propel1 & Propel2) for very common usage scenarios.

A great lib for that could be https://github.com/polyfractal/athletic

Using a benchmark we could also verify if things like a external Hydrator lib https://github.com/propelorm/Propel2/issues/425 will speedup things for us.

[willdurand]

See: https://code.google.com/p/php-orm-benchmark/

[motin]

I recently submitted a PR with benchmarks for Propel 2 to the library that @willdurand mentioned: https://github.com/kenjis/php-orm-benchmark/pull/7

UPDATE: Sorry, this is not the same library as @willdurand mentioned, but a completely different one: https://github.com/kenjis/php-orm-benchmark

[Big-Shark]

Cool, I try yesterday cloning this rep and starting benchmark, but that fuel version does not work on php7 ( But I have my own benchmarks https://github.com/Big-Shark/forked-php-orm-benchmark

[motin]

@Big-Shark If that link would have been available in this issue earlier today I would had based my PR on that repo instead. Do you have any results to share based on those benchmarks?

[Big-Shark]

@motin Yesterday results

|-------------------------------------------------------------------------------------------------------|
| Library                          | Insert | findPk | complex| hydrate|  with  | memory usage |  time  |
|----------------------------------| -------| -------| -------| -------| -------| -------------| -------|
|                              PDO |     36 |     40 |     24 |     31 |     33 |    1,710,056 |   0.17 |
|----------------------------------| -------| -------| -------| -------| -------| -------------| -------|
|                         Propel20 |    101 |     56 |     50 |    142 |    178 |    6,291,456 |   0.54 |
|                Propel20WithCache |     57 |     26 |     40 |    110 |    132 |    6,291,456 |   0.38 |
|           Propel20FormatOnDemand |     57 |     27 |     41 |     98 |    134 |    6,291,456 |   0.37 |
|----------------------------------| -------| -------| -------| -------| -------| -------------| -------|
|                       Doctrine24 |    104 |    122 |     39 |    114 |    101 |   14,680,064 |   0.66 |
|              Doctrine24WithCache |     97 |    108 |     38 |    113 |    100 |   14,680,064 |   0.48 |
|           Doctrine24ArrayHydrate |    103 |    117 |     40 |     72 |     76 |   14,680,064 |   0.43 |
|          Doctrine24ScalarHydrate |    104 |    131 |     42 |     70 |     69 |   14,680,064 |   0.44 |
|         Doctrine24WithoutProxies |    102 |    120 |     44 |     91 |    169 |   14,680,064 |   0.55 |
|----------------------------------| -------| -------| -------| -------| -------| -------------| -------|
|                        DoctrineM |    100 |    117 |     41 |    117 |    103 |   14,680,064 |   0.50 |
|               DoctrineMWithCache |     99 |    117 |     39 |    117 |    103 |   14,680,064 |   0.51 |
|            DoctrineMArrayHydrate |     99 |    119 |     40 |     72 |     74 |   14,680,064 |   0.43 |
|           DoctrineMScalarHydrate |     94 |    113 |     40 |     61 |     64 |   14,680,064 |   0.40 |
|          DoctrineMWithoutProxies |     97 |    116 |     39 |     89 |    157 |   14,680,064 |   0.52 |
|----------------------------------| -------| -------| -------| -------| -------| -------------| -------|
|                         Eloquent |    221 |    138 |     44 |     82 |    133 |    8,388,608 |   0.63 |
|             EloquentWithoutEvent |    193 |    131 |     43 |     79 |    128 |    8,388,608 |   0.58 |
|-------------------------------------------------------------------------------------------------------|

[motin]

@Big-Shark Great stuff! However, it only benchmarks the slowest ORM's out there, according to the benchmark quoted in https://github.com/propelorm/Propel2/issues/1062 - what about adding Yii 1, Yii 2, Phalcon and Fuel to the list?

[Big-Shark]

@motin I'll be glad to every PR) Very strange, but in my test, the Propel is 2 times faster than Eloquent, and in the test of kenjis, Eloquent faster than Propel.

[motin]

@Big-Shark I think it depends on how complex the testing scenario is. According to personal experience, Propel gives a huge performance boost in complex (aka real world) scenarios. Kenjis test is extremely simplistic, and if someone actually had to implement such a simple scenario, an ORM is not even necessary.

It would be just awesome to see how at least Yii 2 performs in the more complex scenarios that your benchmark set-up offers.

Btw, how should your results be interpreted? Memory and time are self-explanatory, but the other columns?

[motin]

Update, Yii 1 and 2 results are available here: https://github.com/cebe/php-orm-benchmark

[Big-Shark]

@motin This is a different tasks, and their time. Hmm Yii was slower in Cebe tests.

[Big-Shark]

@motin https://github.com/propelorm/Propel2/issues/478 old discussion

[motin]

@Big-Shark Thanks I had missed the old discussion. As you see, I consolidated efforts so that Yii 1, Yii 2 and LessQL show up in the benchmark results: https://github.com/Big-Shark/forked-php-orm-benchmark/pull/3

[motin]

Please do not the major differences between the two benchmarks:

Kenji's benchmark (https://github.com/kenjis/php-orm-benchmark, results for Propel 2 available at https://github.com/propelorm/Propel2/issues/1062#issue-120764284) tests the average execution time and memory usage of one very simplistic method served by a web server using MySQL as the dbms.

François's benchmark (https://github.com/Big-Shark/forked-php-orm-benchmark) tests the single execution of more complex cases but does it all from the command line, using SQLite as the dbms.

Also, at least hhvm is known to be optimized mainly for multiple serial and/or simultaneous requests, and needs tweaks to perform well on the command line. There is also the RepoAuthoritative mode which is claimed to yield -20% response time. This is relevant when comparing to extension-based ORMs like Phalcon.

Ideally, we should have benchmarks that tests complex scenarios served by a web server and evaluate server response time and total strain on the server under high concurrency, with and without cache. This is the thing that matters in the end for most people evaluating ORM performance.

[marcj]

@motin I completely agree with you. Synthetic benchmarks on the whole framework makes no sense expect to increase the speed while developing on it. We should as stated evaluate only application benchmarks to compare with other ORMs out there.

[motin]

Do we know of any good reference application for PHP ORM's - something like http://todomvc.com/ "Helping you select an MV* framework" but instead "Helping you select an ORM"?

[motin]

If we can't find an existing solution out there, I suggest we set up a REST API usage scenario, where one or many selected API blueprints are implemented using various ORM:s. Preferably, this/these REST API blueprint(s) should already be used in production somewhere, so that we can get accurate server logs to convert into a realistic testing scenario where lots of concurrent users are running CRUD operations against the REST API.

[motin]

Great comment on the subject of Hello World benchmarks: https://github.com/facebook/hhvm/issues/6645#issuecomment-163670484

[marcj]

I'm not aware of such an application like todomvc comparing directly ORMs featureset/performance. Maybe we should create one. ;o)