foreach with generators performance significantly less than php 5.5

[stuartcarnie]

php 5.5.3

vs

HipHop VM v2.1.0-dev (rel) Compiler: heads/master-0-ga5ca12110e8916913c650befebe6e63eaa0dc87f Repo schema: 410dfcf824e18d6de251d6b6088bfb6febc66aa5

<?php

function xrange($from, $to) {
    while ($from < $to) {
        yield $from++;
    }
}

function test_generator($iterations) {
    $o = [];
    foreach(xrange(0, $iterations) as $i) {
        $o []= $i;
    }
    return $o;
}

function bench($function, $iterations=1000000) {
    $startTime = microtime(true);
    $startMem = memory_get_usage(true);
    $res = call_user_func($function, $iterations);
    $endTime = microtime(true);
    $endMem = memory_get_usage(true);
    $time = $endTime - $startTime;
    $mem = $endMem - $startMem;
    printf("%s: time = %f s, mem %d kb\n", $function, $time, $mem/1024);
}

$options = getopt('', ['times::']);

if (isset($options['times'])) {
    $count = intval($options['times']);
} else {
    $count = 5;
}

if ($count <= 0) $count = 5;

printf("Running %d times\n", $count);

while (--$count) {
    bench('test_generator');
}

[edwinsmith]

Did you run hhvm with -vEval.Jit=1? Also, the main loop of the benchmark is in the php toplevel file, so we wouldn't JIT-compile it anyway. if you wrap the main code in a function you'll get better results.

[stuartcarnie]

@edwinsmith I did use -v Eval.Jit=1 and the test_generator function should be optimized, which is where all the work is done.

@scannell, I did test for vs foreach and certainly that makes a huge difference, which makes sense given the JIT likely optimizes the loop to an increment, compare and jump vs a function call. The yield version came about for some other memory benchmarks I was verifying using array vs ArrayObject. I set memory_limit to 256M to run under zend55 and the xrange version ran a fair amount faster that hhvm.

One thing that was great is hhvm used less than 1/3 the memory of zend.

[stuartcarnie]

This is the output of zend55 vs hhvm

└─[$]> php yield_perf.php
Running 5 times
test_generator: time = 0.209098 s, mem 141312 kb
test_generator: time = 0.207385 s, mem 140544 kb
test_generator: time = 0.170235 s, mem 140032 kb
test_generator: time = 0.170571 s, mem 139776 kb
┌─[contatta@ubuntu] - [~/dev/hhvm] - [Tue Sep 17, 10:40]
└─[$]> hhvm -v Eval.Jit=1 yield_perf.php
Running 5 times
test_generator: time = 0.461920 s, mem 49159 kb
test_generator: time = 0.461509 s, mem 49152 kb
test_generator: time = 0.458255 s, mem 49152 kb
test_generator: time = 0.457396 s, mem 49152 kb
┌─[contatta@ubuntu] - [~/dev/hhvm] - [Tue Sep 17, 10:40]
└─[$]>

[scannell]

Changing title. It seems to be the foreach here -- in this benchmark we're still way faster:

<?php

function xrange($from, $to) {
    while ($from < $to) {
        yield $from++;
    }
}

function test_generator($iterations) {
    $o = [];
    for ($i = 0 ; $i < $iterations; ++$i) {
        $o []= xrange(0, $iterations);
    }
    return $o;
}

function bench($function, $iterations=120000) {
    $startTime = microtime(true);
    $startMem = memory_get_usage(true);
    call_user_func($function, $iterations);
    $endTime = microtime(true);
    $endMem = memory_get_usage(true);
    $time = $endTime - $startTime;
    $mem = $endMem - $startMem;
    printf("%s: time = %f s, mem %d kb\n", $function, $time, $mem/1024);
}

$options = getopt('', ['times::']);

if (isset($options['times'])) {
    $count = intval($options['times']);
} else {
    $count = 5;
}

if ($count <= 0) $count = 5;

printf("Running %d times\n", $count);

while ($count--) {
    bench('test_generator');
}

$ ~jdelong/bin/zend55 ~/scratch/genbench.php Running 5 times test_generator: time = 0.174564 s, mem 8704 kb test_generator: time = 0.162268 s, mem 0 kb test_generator: time = 0.183380 s, mem 0 kb test_generator: time = 0.164871 s, mem 0 kb test_generator: time = 0.164794 s, mem 0 kb $ hhvm ~/scratch/genbench.php Running 5 times test_generator: time = 0.028410 s, mem 4 kb test_generator: time = 0.017732 s, mem 0 kb test_generator: time = 0.017844 s, mem 0 kb test_generator: time = 0.018134 s, mem 0 kb test_generator: time = 0.017781 s, mem 0 kb

[stuartcarnie]

@scannell you are right, however your test code only stores the generator rather than the result of iterating the generator. This code shows using a while loop vs the earlier tests with a foreach is significantly faster in hhvm vs while is slower in zend55.

One issue I noticed is that I had to call $gen->next() outside the while loop in hhvm or it crashed, so that is a difference between zend. I'll create a separate issue for that zend incompatibility.

yield_with_while.php

<?php

function xrange($from, $to) {
    while ($from < $to) {
        yield $from++;
    }
}

function test_generator($iterations) {
    $o = [];
    $gen = xrange(0, $iterations);
    $gen->next();
    while ($gen->valid()) {
        $o []= $gen->current();
        $gen->next();
    }
    return $o;
}

function bench($function, $iterations=100000) {
    $startTime = microtime(true);
    $startMem = memory_get_usage(true);
    $o = call_user_func($function, $iterations);
    $endTime = microtime(true);
    $endMem = memory_get_usage(true);
    $time = $endTime - $startTime;
    $mem = $endMem - $startMem;
    printf("%s: time = %f s, mem %d kb\n", $function, $time, $mem/1024);
}

$options = getopt('', ['times::']);

if (isset($options['times'])) {
    $count = intval($options['times']);
} else {
    $count = 5;
}

if ($count <= 0) $count = 5;

printf("Running %d times\n", $count);

while ($count--) {
    bench('test_generator');
}

Results:

┌─[contatta@ubuntu] - [~/dev/hhvm] - [Tue Sep 17, 11:25]
└─[$]> hhvm -v Eval.Jit=1 yield_with_foreach.php
Running 5 times
test_generator: time = 0.462663 s, mem 49159 kb
test_generator: time = 0.455075 s, mem 49152 kb
test_generator: time = 0.459674 s, mem 49152 kb
test_generator: time = 0.460977 s, mem 49152 kb
┌─[contatta@ubuntu] - [~/dev/hhvm] - [Tue Sep 17, 11:25]
└─[$]> hhvm -v Eval.Jit=1 yield_with_while.php
Running 5 times
test_generator: time = 0.056093 s, mem 49160 kb
test_generator: time = 0.052530 s, mem 49152 kb
test_generator: time = 0.051756 s, mem 49152 kb
test_generator: time = 0.051430 s, mem 49152 kb
test_generator: time = 0.051663 s, mem 49152 kb
┌─[contatta@ubuntu] - [~/dev/hhvm] - [Tue Sep 17, 11:25]
└─[$]> php yield_with_foreach.php
Running 5 times
test_generator: time = 0.214958 s, mem 141312 kb
test_generator: time = 0.207230 s, mem 140544 kb
test_generator: time = 0.171117 s, mem 140032 kb
test_generator: time = 0.170858 s, mem 139776 kb
┌─[contatta@ubuntu] - [~/dev/hhvm] - [Tue Sep 17, 11:25]
└─[$]> php yield_with_while.php
Running 5 times
test_generator: time = 0.408069 s, mem 141312 kb
test_generator: time = 0.407820 s, mem 140544 kb
test_generator: time = 0.368915 s, mem 140032 kb
test_generator: time = 0.364851 s, mem 139776 kb
test_generator: time = 0.367285 s, mem 139520 kb

[scannell]

Cool, thanks for the investigation. I've flagged the foreach internally as something we should look at at some point.

[Aatch]

Ok, so I decided to do some investigation further here. Turns out it has nothing to do with generators being slow. Anytime you use an Iterator object you get the performance drop seen here. On my machine, with almost exactly the same benchmark as given above, using a while directly is 5 times faster.

More investigation seems to show that the significant number of calls to drive an Iterator object is the most likely culprit. IterNext for an Iterator object has to re-enter the VM to do the call, which none of the other iterator types have to do. The best answer I can see is to try and hoist as much of this logic as we can up to bytecode level, which has the nice side-effect of making it more visible to other optimisations.