added benchmark. closes lstrojny#15

[staabm]

closes lstrojny#15

after some try&error I got the benchmark running on travis.

I used Symfony/Yaml as a references suite, so I don't need to introduce a big test-suite only for benchmarking which would slow down the CI build considerably.

ATM it seems we have an overhead of ~ 80%

[staabm]

the integration is not that pretty right now, but it works as a first step.

[staabm]

any feedback?

[lstrojny]

Sorry, was out of the country. Will have a look today.

[staabm]

np, was also on vacation... take your time. thanks.

[lstrojny]

Initial result of the benchmark: it’s about 90 - 100% slower than stock PHPUnit. Not accessing SQLite makes it only 35% slower so there is significant performance to be gained from optimizing the storage code. Still, the sorting code being that slow makes me worry a bit as I don’t think there is so much to be gained.

Can you reproduce similar results? (for the benchmark without sqlite I commented applied the following patch)

diff --git a/src/PHPUnit/Runner/CleverAndSmart/Storage/Sqlite3Storage.php b/src/PHPUnit/Runner/CleverAndSmart/Storage/Sqlite3Storage.php
index fc813a3..3664b04 100644
--- a/src/PHPUnit/Runner/CleverAndSmart/Storage/Sqlite3Storage.php
+++ b/src/PHPUnit/Runner/CleverAndSmart/Storage/Sqlite3Storage.php
@@ -21,6 +21,7 @@ class Sqlite3Storage implements StorageInterface

     public function __construct($fileName = '.phpunit-cas.db')
     {
+        return;
         $this->db = new SQLite3($fileName);

         // method introduced in php 5.3.3
@@ -72,6 +73,7 @@ class Sqlite3Storage implements StorageInterface

     public function record(Run $run, TestCase $test, $time, $status)
     {
+        return;
         $this->transactional(array($this, 'doRecord'), $run, $test, $time, $status);
     }

@@ -82,6 +84,7 @@ class Sqlite3Storage implements StorageInterface

     public function getRecordings(array $types, $includeTime = true)
     {
+        return array();
         $query = 'FROM {{prefix}}result
                 WHERE result_state IN (%s)
                 GROUP BY result_identifier

Benchmark results stock (btw. very neatly formatted):

PHPUnit\Tests\Runner\CleverAndSmart\Benchmark\RunSuiteEvent
  No Group
    Method Name                                 Iterations    Average Time      Ops/s    Relative
    ------------------------------  ----------  ------------ --------------   ---------  ---------
    plainSuite                    : [Baseline] [30        ] [0.6080054601034] [1.64472]
    instrutmentedSuite            :            [30        ] [1.2040698528290] [0.83052] [198.04%]

Benchmark results without SQLite:

PHPUnit\Tests\Runner\CleverAndSmart\Benchmark\RunSuiteEvent
  No Group
    Method Name                                 Iterations    Average Time      Ops/s    Relative
    ------------------------------  ----------  ------------ --------------   ---------  ---------
    plainSuite                    : [Baseline] [30        ] [0.6344240585963] [1.57623]
    instrutmentedSuite            :            [30        ] [0.8530421892802] [1.17227] [134.46%]

[staabm]

fixed most of your comments... more to come.

I added a new MockedStorage for testing the case you illustrated without the sqlite overhead to get a better picture of the actual sorting overhead.

No plans yet how to improve the sorting, but we cannot optimise what we cannot measure ;-).

[staabm]

soo I am ready for now...

@lstrojny please have another look. thanks for your feedback.

The results on travis look more or less like what you already summarized...

[lstrojny]

Awesome!