Techempower - First results

[Wulfklaue]

First bench results for Swoole are in:

https://www.techempower.com/benchmarks/#section=test&runid=ec8d20d8-2530-4a7d-a9c2-157bebdb9b81

JSON serialization is now almost double thanks to the BASE change. 646,732 -> 1,153,488

Some other results are faster then PHP and almost Go level. But some are remarkably slower despite using the exact same code as the PHP7-Raw ( with the only exception being that Swoole using a case statement ).

Worst results are the plaintext ... What makes no sense at all php-swoole | 1,090 with errors. When my expectation was close to 2.000.000.

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/PHP/php/swoole-server.php

What normally needs to result in a very simple result, is so strange.

[twose]

There is one, you shouldn't use PDO in swoole, it will block the server.

$db = new Swoole\Coroutine\Mysql;
$server = [
    'host'     => '127.0.0.1',
    'user'     => 'root',
    'password' => 'root',
    'database' => 'test',
];
$db->connect($server);
$stmt = $db->prepare('SELECT * FROM `user` WHERE id=?');
$ret = $stmt->execute([1]);

[Wulfklaue]

@twose

Updated the source with the non blocking mysql.

So the new tests results are in and they are bad...

https://www.techempower.com/benchmarks/#section=test&runid=acde94c4-fd61-4e83-91b4-15a110b8c695&hw=ph&test=json

End result is horrible results for any test involving the database. And updating the version to 2.1.3 also did not resolve the plaintext results.

I am done trying to fix this. Its up to the Swoole developer to figure out himself but from my own experience the coroutine tends to be way worse then simply using the default php "blocking" database access.

It also does not help that:

In file included from /tmp/swoole-src-2.1.3/src/os/base.c:19:0:
/tmp/swoole-src-2.1.3/src/os/base.c: In function 'swAio_handler_read_file':
/tmp/swoole-src-2.1.3/include/swoole.h:345:41: warning: format '%s' expects argument of type 'char *', but argument 6 has type 'void *' [-Wformat=]
     snprintf(sw_error,SW_ERROR_MSG_SIZE,"%s(:%d): " str " Error: %s[%d].",__func__,__LINE__,##__VA_ARGS__,strerror(errno),errno);\
                                         ^
/tmp/swoole-src-2.1.3/src/os/base.c:367:9: note: in expansion of macro 'swSysError'
         swSysError("open(%s, O_RDONLY) failed.", event->req);
         ^~~~~~~~~~
/tmp/swoole-src-2.1.3/include/swoole.h:345:41: warning: format '%s' expects argument of type 'char *', but argument 6 has type 'void *' [-Wformat=]
     snprintf(sw_error,SW_ERROR_MSG_SIZE,"%s(:%d): " str " Error: %s[%d].",__func__,__LINE__,##__VA_ARGS__,strerror(errno),errno);\
                                         ^
/tmp/swoole-src-2.1.3/src/os/base.c:375:9: note: in expansion of macro 'swSysError'
         swSysError("fstat(%s) failed.", event->req);
         ^~~~~~~~~~
/tmp/swoole-src-2.1.3/src/os/base.c: In function 'swAio_handler_write_file':
/tmp/swoole-src-2.1.3/include/swoole.h:345:41: warning: format '%s' expects argument of type 'char *', but argument 6 has type 'void *' [-Wformat=]
     snprintf(sw_error,SW_ERROR_MSG_SIZE,"%s(:%d): " str " Error: %s[%d].",__func__,__LINE__,##__VA_ARGS__,strerror(errno),errno);\
                                         ^
/tmp/swoole-src-2.1.3/src/os/base.c:426:9: note: in expansion of macro 'swSysError'
         swSysError("open(%s, %d) failed.", event->req, event->flags);
         ^~~~~~~~~~

You see stuff like this during compilation. I get the feeling that Swoole needs better testing.

[twose]

How to view stress test error log? It's obviously unreasonable that plaintext is slower than JSON. Swoole needs better testing indeed, We are working hard for it.

[Wulfklaue]

@twose

I think this is what your looking for:

http://tfb-logs.techempower.com/round-16/preview-1/php-swoole/

But they do not show any clear errors. And given the timestamps, i feels that this is not the latest run with the coroutines.

Latest run:

https://tfb-status.techempower.com/results/acde94c4-fd61-4e83-91b4-15a110b8c695

The Travis build of the last successful commits ( Swoole starts around line 1560 ):

https://travis-ci.org/TechEmpower/FrameworkBenchmarks/jobs/381516883

I am using the exact same build and have no issue on small scale tests ( plaintext and json ). So whatever is going wrong with the plaintext is related to the mass amount of requests being pushed at the same time.

[Wulfklaue]

Here are the real result logs for this benchmark run:

https://tfb-status.techempower.com/unzip/results.2018-05-25-20-11-22-271.zip/results/20180523080155/php-swoole

Error log:

https://tfb-status.techempower.com/unzip/results.2018-05-25-20-11-22-271.zip/results/20180523080155/php-swoole/run/php-swoole.log

Plaintext:

https://tfb-status.techempower.com/unzip/results.2018-05-25-20-11-22-271.zip/results/20180523080155/php-swoole/plaintext/raw.txt

Notice how in the warmup plaintext is pushing Requests/sec: 605332.47 But when the pipeline.lua Requests/sec: 16.69

When comparing the json, that does not use the pipeline.lua we get fast results...

Database tests:

php-swoole: Fatal error: Uncaught Swoole\Coroutine\MySQL\Exception: connect to mysql server[tfb-database:3306] failed. in /swoole/swoole-server.php:28
php-swoole: Stack trace:
php-swoole: #0 /swoole/swoole-server.php(28): Swoole\Coroutine\MySQL->connect(Array)
php-swoole: #1 {main}
php-swoole:   thrown in /swoole/swoole-server.php on line 28

            $db = new Swoole\Coroutine\Mysql;
            $server = [
                'host' => 'tfb-database',
                'user' => 'benchmarkdbuser',
                'password' => 'benchmarkdbpass',
                'database' => 'hello_world'
            ];
            $db->connect($server);

Line 28 is $db->connect($server);

Coroutine Mysql has no connection pool? When i tried the Coroutine PostgreSQL a while ago in one of my own test, the results ended up being bad compared to the standard PHP postgreSQL driver ( that pooled the connection ).

[twose]

@Wulfklaue Thanks for your contribution! I can see the error of MySQL connection, the reason for it is too many connections cause MySQL server down. You are right that native Swoole has no connection pool, it implements in the PHP frameworks such as swoft, easyswoole and so on...If you want to implement it in the test script, you have more works to do. But I can't find any error log about the plaintext, the tests run successfully on my mac, I think some problems are before this stranger result.

[Wulfklaue]

@twose

Mysql: I will see about implementing the connection pool in TFB when i have time. The fact that Swoole has no internal connection pool and people do not know about it is kind of unexpected. Other users can run into this issue and blame it on Swoole. My experience with the new Co/postgresql driver, i ended up blamed Swoole as having bugs when in reality it was simply not pooling ( what one expects ).

And take in account a PHP connection pool will be slower / consume more memory, then one written in C, internally in Swoole. ;) Especially for micro-benchmarks like TFB.

plaintext: I am fairly sure that the pipeline.lua is the reason. On the warmup test Swoole can hit 600k without issue but the moment the pipeline.lua has been added to the tests, the results drop like a brick.
We have json results that pull 1100k and the code is almost identical to the plaintext output with only the encoding and different header. The big difference being that pipeline.lua script that gets added for plaintext after the warmup.

Take in account, test on your mac are at best 8 threads hitting Swoole. Even with pipeline.lua increasing pressure. It feels like so many threads on TFB + the extra pressure from pipeline.lua is blocking ports or something like that.

Just a stupid idea but is reuseport or the lack of it possible the issue? Just a stab in the dark...

[matyhtf]

Do not use domain name, try ip address.

$db = new Swoole\Coroutine\Mysql;
            $server = [
                'host' => co::gethostbyname('tfb-database'),
                'user' => 'benchmarkdbuser',
                'password' => 'benchmarkdbpass',
                'database' => 'hello_world'
            ];
            $db->connect($server);

[Wulfklaue]

@matyhtf

Committed the change at Techempower.

[embluk]

Wait so is using the built-in MySQL driver from Swoole not good when handling multiple requests because it does not pool?

[twose]

@Wulfklaue You should check and save IP address before server start because gethostbyname have a coroutine switch spending. And I'm sorry about that I don't know what is pipeline.lua ... is it a test script? how can I find it?

@embluk it's easy to make a connection pool in PHP when use swoole, every swoole-frameworks implement it. here is an example:

$pool = new RedisPool();
$server = new Swoole\Http\Server('127.0.0.1', 9501);

$server->on('Request', function($req, $resp) use ($pool) {
    // get a client from pool
    $redis = $pool->get();
    //failed
    if ($redis === false)
    {
        $resp->end("ERROR");
        return;
    }
    $result = $redis->hgetall('key');
    $resp->end(var_export($result, true));
    //release it
    $pool->put($redis);
});

$server->start();

class RedisPool
{
    protected $pool;

    function __construct()
    {
        $this->pool = new SplQueue;
    }

    function put($redis)
    {
        $this->pool->push($redis);
    }

    function get()
    {
        if (count($this->pool) > 0)
        {
            return $this->pool->pop();
        }

        // No idle connection to create a new connection
        $redis = new Swoole\Coroutine\Redis();
        $res = $redis->connect('127.0.0.1', 6379);
        if ($res == false)
        {
            return false;
        }
        else
        {
            return $redis;
        }
    }
}

[Wulfklaue]

@twose

I just merged it in as i currently do not have a lot of time for development / testing. If somebody has time to improve the test, the code located now at the following location:

https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/PHP/swoole

Lua:

https://tfb-status.techempower.com/unzip/results.2018-05-25-20-11-22-271.zip/results/20180523080155/php-swoole/plaintext/raw.txt

Check the right side. This is how tfb its tests are run for plaintext. Warm up and then actual benching using the lua.pipelines script.

The pipeline.lua is located at the following location:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/dcf79e33d54f8c7c7ff98dca61512d3a617132ab/toolset/wrk/pipeline.lua

[twose]

@Wulfklaue I recurred it on my mac, you are correct, when we use the pipeline.lua script, some problems occurred.

[Wulfklaue]

@twose

Updated the test with a reworked version of the reddis pool:

/**
 * Class MySQLPool
 *
 * Deal with the fact that Swoole 2.1.3 has no build in database pooling
 */
class MySQLPool
{
    protected $pool;
    var $pool_count = 0;

    var $server = [
        'host' => 'tfb-database',
        'user' => 'benchmarkdbuser',
        'password' => 'benchmarkdbpass',
        'database' => 'hello_world'
    ];

    function __construct()
    {
        $this->pool = new \SplQueue;
    }

    function put($redis)
    {
        $this->pool->push($redis);
        $this->pool_count++;
    }

    function get()
    {
        if ( $this->pool_count > 0)
        {
            $this->pool_count--;
            return $this->pool->pop();
        }

        // No idle connection to create a new connection
        $db = new Swoole\Coroutine\Mysql;
        $db->connect($this->server);

        if ($db == false)
            return false;
        else
            return $db;
    }
}

Removed the count function and used a integer variable for the count, as this is much more efficient then calling the count function on each request. Better also update the reddis example above.

Please note:

    function __construct()
    {
        $this->pool = new \SplQueue;
        $this->server['host'] = Swoole\Coroutine::gethostbyname('tfb-database');
    }

Using Swoole\Coroutine::gethostbyname('tfb-database'); like this results in a Segmentation fault (core dumped) on run..

[twose]

@Wulfklaue

$server->on('workerStart', function () use (&$tfb_database_ip) {
    $tfb_database_ip = co::gethostbyname('tfb-database');
});
$server->on('request', function (Request $req, Response $res) use ($tfb_database_ip) {
    $db = new Swoole\Coroutine\Mysql;
        $server = [
            'host' => $tfb_database_ip,
            'user' => 'benchmarkdbuser',
            'password' => 'benchmarkdbpass',
            'database' => 'hello_world'
        ];
}

About coredump: gethostbyname must be called in the coroutine We fix it on 4.0 (You'd better use 2.x)

And then, you can use SplQueue::count function because it's O(1) instead of traversing the queue

static zend_long spl_ptr_llist_count(spl_ptr_llist *llist) /* {{{ */
{
    return (zend_long)llist->count;
}

or

if($db = $this->pool->pop()) {
    return $conn;
}

You can always be sure that PHP have alreay done everything you want... XD

[twose]

About pipeline, matyhtf will find the problem and solve it. But we are more interested in database benchmarks, that is the pain points of traditional language but our coroutine solved.

[Wulfklaue]

@twose

Updated the code base ( but with a twist ):

$pool = new MySQLPool();

$server->on('workerStart', function () use (&$pool) {
    $pool->set_host_ip();
});

class MySQLPool
{
    function set_host_ip()
    {
        if( empty( $this->server['host'] ) )
        {
            $tfb_database_ip = Swoole\Coroutine::gethostbyname('tfb-database');
            $this->server['host'] = $tfb_database_ip;
        }
    }
}

This will reduce the time needed ( we are talking about 40 workers and 40 lookups before ).

SplQueue::count

    function get()
    {
        if($db = $this->pool->pop())
            return $db;

        // No idle connection to create a new connection
        $db = new Swoole\Coroutine\Mysql;
        $db->connect($this->server);

        if ($db == false)
            return false;
        else
            return $db;
    }

Kind of expect that to fail:

swoole: Fatal error: Uncaught RuntimeException: Can't pop from an empty datastructure in /swoole/swoole-server.php:155
swoole: Stack trace:
swoole: #0 /swoole/swoole-server.php(155): SplDoublyLinkedList->pop()
swoole: #1 /swoole/swoole-server.php(29): MySQLPool->get()
swoole: #2 {main}
swoole:   thrown in /swoole/swoole-server.php on line 155

Variable counting is simpler. Reverted to that and passes the tests.

lua.pipeline

If @matyhtf can figure out the lua.pipeline and have a release version, then we can test all this code in when the server is ready for the next run ( 40 hours from now ). No pressure :)

[twose]

@Wulfklaue Sorry, I confuse the SplQueue::pop method with the array_pop, you can also use count method, it's O(1).

[Wulfklaue]

@twose

New results are in:

https://www.techempower.com/benchmarks/#section=test&runid=2c113b42-5dbb-4eea-a0b9-b4d6863f5cad&hw=ph&test=fortune

Fortunes:

10 | swoole | 277,395 70 | php-php5-raw | 107,848 | 25.3% | Plt | PHP

Note: Swoole is the second fasted Mysql based test. We need a PosgresSQL version of the test.

Updates:

46 | hhvm | 4,652 132 | swoole | 2,425 |  

Seems to be too slow on this one. I need to verify the logs.

Multiple queries:

25 | swoole | 20,991 |   50 | php-raw7 | 17,710

Not bad again. Six position MySQL test. Clearly we need to add the postgreSQL test for Swoole.

Single queries:

96 | php-php5-raw | 125,801 127 | swoole | 87,486

Errors on this test. Will need to figure out what goes wrong here.

Plaintext

259 | swoole | 1,092

Still horrible results until @matyhtf fixes this issue in the Swoole source.

Json

18 | swoole | 1,135,887

Like always great result here. Swoole is always benching between top ten and twenty on this one.

Conclusion:

1. Seems when i have time, i need to write the postgreSQL version of the test also as we have nothing more to gain from the MySQL test ( in some of the benchmarks ).

2. Some errors on a few tests are still present and clearly make 2 of the test too low.

3. Coroutines seem to be doing their jobs. We just need to figure out why one of the tests have errors.

Two have errors. The plaintext we know. The Single queries is strange. And one test is too low but no errors reported.

All the rest of the test put Swoole in the top 10 for MySQL or darn close to the top 10. So nice job matyhtf.

[Wulfklaue]

Almost forgot, here is the log:

https://tfb-status.techempower.com/unzip/results.2018-06-06-00-32-22-549.zip/results/20180603120747/swoole/run/swoole.log

It looks like the two tests where the result are too low, they are both using the prepare statement. The same statement that is giving errors.

swoole: Warning: Swoole\Coroutine\MySQL::prepare(): mysql client is waiting response, cannot send new sql query. in /swoole/swoole-server.php on line 38
swoole: Warning: Swoole\Coroutine\MySQL::prepare(): mysql client is waiting response, cannot send new sql query. in /swoole/swoole-server.php on line 38
swoole: Fatal error: Uncaught Error: Call to a member function execute() on boolean in /swoole/swoole-server.php:44

Wild guess but it looks because we are pooling the connection and then pushing the exact same prepare statement, it is shocking on this.

Update 2:

Implementing the postgresql version.

Testing:

The open issues are the plaintext result and the prepare statement that chokes.

When these last two issues are fixed, the test results will be stable.

[Wulfklaue]

Here is some bad new for postgres:

swoole-postgres: [2018-06-06 15:41:14 *10.1]    ERROR   swReactorEpoll_add(:132): add events[fd=-1#1, type=20, events=4] failed. Error: Bad file descriptor[9].
swoole-postgres: Warning: Swoole\Coroutine\PostgreSQL::connect(): swoole_event_add failed. in /swoole/swoole-server.php on line 244
swoole-postgres: [2018-06-06 15:41:14 *10.1]    ERROR   swReactorEpoll_del(:148): epoll remove fd[0#1] failed. Error: No such file or directory[2].
swoole-postgres: Fatal error: Uncaught Error: Call to undefined method Swoole\Coroutine\PostgreSQL::prepare() in /swoole/swoole-server.php:28
swoole-postgres: Stack trace:
swoole-postgres: #0 /swoole/swoole-server.php(170): {closure}('postgres', -1)
swoole-postgres: #1 {main}
swoole-postgres:   thrown in /swoole/swoole-server.php on line 28

Docker file:

FROM php:7.2

ENV SWOOLE_VERSION=2.1.3

RUN     apt-get update && apt-get install -y libpq-dev \
        && cd /tmp && curl -sSL "https://github.com/swoole/swoole-src/archive/v${SWOOLE_VERSION}.tar.gz" | tar xzf - \
        && cd swoole-src-${SWOOLE_VERSION} \
        && phpize && ./configure --enable-coroutine --enable-coroutine-postgresql > /dev/null && make > /dev/null && make install > /dev/null \
        && docker-php-ext-enable swoole

WORKDIR /swoole
COPY swoole-server.php swoole-server.php
COPY php.ini /usr/local/etc/php/

CMD sed -i 's|NUMCORES|'"$(nproc)"'|g' swoole-server.php && \
    php swoole-server.php

Its kind of annoying that the postgreSQL driver is not part of Swoole coroutines default. But the above error seems to stem from the driver.

Update;

Missing in the swoole_postgresql_coro.c drivers are functions like: "prepare", "execute", ...

That explain the issue... unfortunately i can not use the normal query or fetch statement for two of the techempowered tests. It relies too much on prepared statements, using anything else will affect the results too much.

[twose]

@Wulfklaue Postgres module's author is @yunnian , it may have some more differences with MySQL. And your code can be optimized again. e.g., the prepared statement can be persistence to reduce network communication at least one time and reduce compilation expense on the server side, it will be faster again. the swoole coroutine is a modern network engine, so the best is to use PHP7 language grammar, use strong type code style.

The bug of the pipeline may be due to we have not supported it under the hood, we will figure out it later.

[Wulfklaue]

For the postgres code activation, simply use the following:

,
"postgres": {
"db_url": "/db_postgres",
"query_url": "/db_postgres?queries=",
"fortune_url": "/fortunes_postgres",
"update_url": "/updates_postgres?queries=",
"port": 8080,
"approach": "Realistic",
"classification": "Platform",
"database": "Postgres",
"framework": "None",
"language": "PHP",
"flavor": "PHP7",
"orm": "Raw",
"platform": "None",
"webserver": "swoole",
"os": "Linux",
"database_os": "Linux",
"display_name": "Swoole-Postgres",
"notes": "",
"versus": "php"
}

To activate the postgres code ( if it gets updated with the prepare etc statements ). And update the swoole version number in the docker files. Same for if the plaintext bug is solved => update the swoole number in the docker files.

Anyway, i am closing this topic so somebody else can take over the Techempowered code development as i lack the time.

[vasily-kartashov]

Any progress on pipeline.lua issue?

[twose]

@vasily-kartashov I am afraid we can't support it (async design reason), it's an enhancement.