multiple services on same event loop

[ClosetGeek-Git]

Hi, I'm considering swoole for multiple reasons but I would need to run multiple services on the same event loop, such as an Websocket server and redis server on the same loop, or an HTTP server and ZMQ server/router on same loop. How is this possible?

[ghost]

@matyhtf

how to run http, ws and redis on the same port / loop? I would like to do this, too... 😕

@ClosetMonkey have you an solution?

[ClosetGeek-Git]

@flddr not same port, just in the same event loop as you can with reactphp and workerman. No solution yet, but if I don't get a response as to how (if?) it is done properly I will likely start each individually using swoole\proccess and communicate between them using the chan or table classes.

[ghost]

I am actually confused. I want to run http on Port 80 and upgrade to ws even on the same port. ws and wss are upgrades on 80 and 443 ?

And i do not want to run nginx in front of... :|

@ClosetMonkey redis works for me in http_server this way, but maybe it's not safe

<?php

$redis = new swoole_redis;
$http = new swoole_http_server('0.0.0.0', 9501);

$http->on('request', function ($request, $response) use ($redis) {
    $redis->connect('127.0.0.1', 6379, function (swoole_redis $redis, $rdata) use ($response) {
      $redis->set('key', 'swoole', function (swoole_redis $redis, $rdata) use ($response) {
          $redis->get('key', function (swoole_redis $redis, $rdata) use ($response) {
              $response->header('Content-Type', 'text/html; charset=utf-8');
              $response->end($rdata);
              $redis->close();
          });
      });
    });
});

$http->start();

... just a test ...

EDIT: while benching i got PHP Warning: Swoole\Redis::connect(): failed to connect to the redis-server[127.0.0.1:6379], Erorr: setsockopt(TCP_NODELAY): Invalid argument[1].

So, code above does not really work in async?!

[ghost]

I can actually not understand if https on Port 443 and wss on Port 1234 becomes CORS/SOP error? Like PWAs, which needs https? Or in mobile apps? ... confused ...

This way one can simply run separate files, like php http.php and php websocket.php - and those would be two swoole side by side? Principially this way it works - but good? Don't know...

[ClosetGeek-Git]

@flddr swoole_websocket_server inherits from swoole_http_server so you can use any swoole_http_server methods on your swoole_websocket_server object (such as swoole_http_server->on('request' ... ). This will allow ws and http to run on the same port (including 80) since ws just upgrades the http connection by default. swoole_websocket_server also inherits swoole_server, which allows you to use the swoole_server->set() method to define ssl properties for HTTPS but haven't used swoole with an ssl service yet so I can't really comment.

<?php
$server = new swoole_websocket_server("127.0.0.1", 80);

$server->on('start', function ($server) {
    // http 'start'
    echo "Swoole http server is started at http://127.0.0.1:80\n";
});

$server->on('request', function ($request, $response) {
    // http 'request'
    $response->header("Content-Type", "text/plain");
    $response->end("Hello World\n");
});

$server->on('open', function($server, $req) {
    // ws 'open'
    echo "connection open: {$req->fd}\n";
});

$server->on('message', function($server, $frame) {
    // ws 'message'
    echo "received message: {$frame->data}\n";
    $server->push($frame->fd, json_encode(["hello", "world"]));
});

$server->on('close', function($server, $fd) {
    // ws 'close'
    echo "connection close: {$fd}\n";
});

$server->start();

[ghost]

@ClosetMonkey thanks for teaching this good way - don't know, i must been drunken 😆

Thats completely correct and fine - so, now we need to know how to run this including redis 😈

[ghost]

Ok, but this way

$redis = new swoole_redis;
$http = new swoole_http_server('0.0.0.0', 9501);

$http->on('request', function ($request, $response) use ($redis) {
    $redis->connect('127.0.0.1', 6379, function (swoole_redis $redis, $rdata) use ($response) {
      $redis->set('key', 'swoole', function (swoole_redis $redis, $rdata) use ($response) {
          $redis->get('key', function (swoole_redis $redis, $rdata) use ($response) {
              $response->header('Content-Type', 'text/html; charset=utf-8');
              $response->end($rdata);
              $redis->close();
          });
      });
    });
});

$http->start();

should be correct?! My errors TCP_NODELAY relies on Windows WSL i guess now - i have some errors to unlimit ubuntu 18.04 and redis correct - do not know how to do this as in 16.04

What do you think @ClosetMonkey ? About this async redis?

But - my php workers in task-manager are running permanently and CPU is 100% after that...

[ghost]

Running this on unlimited linux works, but i guess i need correct error-handling:

Swoole http server is started at http://127.0.0.1:9501
PHP Warning:  Swoole\Redis::__call(): redis client connection is closed. in /home/flddr/y.php on LINE X
PHP Warning:  Swoole\Redis::__call(): redis client connection is closed. in /home/flddr/y.php on LINE Y
[2018-09-21 11:03:44 *2003.2]   NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=13] does not exists.
[2018-09-21 11:03:44 *2004.3]   NOTICE  swFactoryProcess_finish (ERROR 1004): send 173 byte failed, because connection[fd=12] is closed.
[2018-09-21 11:03:44 *2002.1]   NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=9] does not exists.

This happens while benchmarking; the workers are running permanently after that; CPU 100%.

With simple curl this works as aspected @ClosetMonkey

@twose can you help error-handling:

<?php

$redis = new swoole_redis;

$server = new swoole_websocket_server("127.0.0.1", 9501);

$server->on('request', function ($request, $response) use ($html, $redis) {
    $redis->connect('127.0.0.1', 6379, function (swoole_redis $redis, $rdata) use ($response) {
        $redis->set('key', 'swoole', function (swoole_redis $redis, $rdata) use ($response) {
            $redis->get('key', function (swoole_redis $redis, $rdata) use ($response) {
                $response->header('Content-Type', 'text/html; charset=utf-8');
                $response->end($rdata);
                $redis->close();
            });
        }); # LINE Y
      }); # LINE X
});

Can it be done this way?

[ghost]

$server->on('request', function ($request, $response) use ($html, $redis) {
    $redis->connect('127.0.0.1', 6379, function (swoole_redis $redis, $rdata) use ($response) {
        if ($rdata) {
            $redis->set('key', 'swoole', function (swoole_redis $redis, $rdata) use ($response) {
                if ($rdata) {
                    $redis->get('key', function (swoole_redis $redis, $rdata) use ($response) {
                        if ($rdata) {
                            $response->header('Content-Type', 'text/html; charset=utf-8');
                            $response->end($rdata);
                            $redis->close();
                        } else $redis->close();
                    });
                } else $redis->close();
            });
        } else $redis->close();
      });
});

left only:

Swoole http server is started at http://127.0.0.1:9501
[2018-09-21 11:44:59 $3220.0]   WARNING swManager_check_exit_status: worker#0 abnormal exit,status=0, signal=11
[2018-09-21 11:44:59 $3220.0]   WARNING swManager_check_exit_status: worker#1 abnormal exit,status=0, signal=11

after heavy benchmarking. What happens?

[ghost]

How do we prevent

[2018-09-21 12:14:37 *3754.2]   NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=7] does not exists.

?

[ghost]

lastly, i end up with code like

$server->on('request', function ($request, $response) use ($redis, $server) {
    $redis->connect('127.0.0.1', 6379, function (swoole_redis $redis2, $rdata) use ($response, $server) {

        if ($rdata) $redis2->set('key', 'swoole', function (swoole_redis $redis3, $rdata) use ($response, $server) {

            if ($rdata) $redis3->get('key', function (swoole_redis $redis4, $rdata) use ($response, $server) {

                if ($rdata) {
                    if ($server->connections[$response->fd]) {
                        $response->header('Content-Type', 'text/html; charset=utf-8');
                        $response->end($rdata);
                    }
                    $redis4->close();
                } else $redis3->close();

            });

        });

      });
});

but it stays with

Swoole http server is started at http://127.0.0.1:9501
[2018-09-21 12:52:58 $4976.0]   WARNING swManager_check_exit_status: worker#0 abnormal exit,status=0, signal=11
[2018-09-21 12:52:58 $4976.0]   WARNING swManager_check_exit_status: worker#2 abnormal exit,status=0, signal=11
[2018-09-21 12:52:58 $4976.0]   WARNING swManager_check_exit_status: worker#3 abnormal exit,status=0, signal=11

when benchmarking - only curl works.

Everytime workers with abnormal exit runs detached 100% in background :confused:

[ghost]

OH,

this way it fully works:

$server->on('request', function ($request, $response) use ($server) {

    $redis = new swoole_redis; # CHANGED TO INSIDE REQUEST

    $redis->connect('127.0.0.1', 6379, function (swoole_redis $redis2, $rdata) use ($response, $server) {

        if ($rdata) $redis2->set('key', 'swoole', function (swoole_redis $redis3, $rdata) use ($response, $server) {

            if ($rdata) $redis3->get('key', function (swoole_redis $redis4, $rdata) use ($response, $server) {

                if ($rdata) {
                    if ($server->connections[$response->fd]) {
                        $response->header('Content-Type', 'text/html; charset=utf-8');
                        $response->end($rdata);
                    }
                } else {
                    if ($server->connections[$response->fd]) {
                        $response->header('Content-Type', 'text/html; charset=utf-8');
                        $response->end("nodata");
                    }
                }
                $redis4->close();

            });
            else $redis3->close();

        });
        else $redis2->close();

      });

    # HOW TO CLOSE HERE? OR ALL IN ALL BETTER?

});

but on the next runs redis is getting slower - the closing isnt correct i guess?

@ClosetMonkey this way it runs lastly without errors, but i guess, it could be faster. First run is about 11k and then - after and after - it stays at about 1k.

Sometimes i got [2018-09-21 13:36:52 *2967.1] NOTICE swFactoryProcess_finish (ERROR 1004): send 173 byte failed, because connection[fd=24] is closed. - but this is due to response-error?

[ghost]

This is shorter

$server->on('request', function ($request, $response) use ($server) {

    $client = new swoole_redis;
    $client->connect('127.0.0.1', 6379, function (swoole_redis $client, $result) use ($response, $server) {
        $client->set('key', 'swoole', function (swoole_redis $client, $result) use ($response, $server) {
            $client->get('key', function (swoole_redis $client, $result) use ($response, $server) {

                if ($server->connections[$response->fd]) {
                    $response->header('Content-Type', 'text/html; charset=utf-8');
                    $response->end($result);
                }

                $client->close();
            });
        });
    });

});

But how to check this if ($server->connections[$response->fd]) { correct?

It lastly errors

[2018-09-21 14:03:05 *4021.1]   NOTICE  swFactoryProcess_finish (ERROR 1004): send 173 byte failed, because connection[fd=31] is closed.
[2018-09-21 14:03:05 *4022.2]   NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=32] does not exists.

[twose]

You just get a "NOTICE", it's usual, it means the client closed the connection, you can ignore it or set a higher log_level. About signall 11, What is your swoole version? async redis client is nearly deprecated now, we use coroutine, write the async program with sync style code. You can see some examples in: https://github.com/swoole/swoole-src/tree/master/tests/swoole_redis_coro https://github.com/swoole/swoole-src/tree/master/examples/coroutine/redis

[twose]

when benchmarking with time parameter, when the test timeout, test script will force disconnect the connection, so the swoole server will notice you connection[fd=\d+] does not exist.

and you benchmark script was wrong, without any connection pool, too many connections would lead to block.

[twose]

By the way, please create a new issue to discuss the other problem.

[ghost]

@twose :+1:

Is this correct? It runs error-free while benchmarking:

$server->on('request', function ($request, $response) use ($server) {

    go(function ()  use ($response, $server) {
        $redis = new Co\Redis;
        $redis->connect('127.0.0.1', 6379);
        $redis->set('key', 'test');
        if ($server->connections[$response->fd]) { # THIS HERE CORRECT, TOO ?? :)
            $response->header('Content-Type', 'text/plain; charset=utf-8');
            $response->end($redis->get('key'));
        }
        $redis->close();
    });

});

@ClosetMonkey isn't that about your redis-in-the-same-eventloop question? Please excuse me if i dissed your issue :confused:

[ghost]

About signall 11, What is your swoole version?

4.2.1

[ghost]

Sorry, but

$server->on('request', function ($request, $response) use ($server) {

    go(function ()  use ($response, $server) {
        $redis = new Co\Redis;
        $redis->connect('127.0.0.1', 6379);
        if ($server->connections[$response->fd]) {
            $response->header('Content-Type', 'text/plain; charset=utf-8');
            $response->end($redis->get('key'));
        }
        $redis->close();
    });

});

could it be redis is getting very slow when total requests >= redis maxclients?

So $redis->close(); is wrong or what else? I just testet 5000 connections 10 times and redis maxclients is 50000 in this setup test... After that it reduces from ~6k to 0.5k / s :confused:

On ubuntu 18.04 i have installed libhiredis-dev - not the right one? php --ri swoole gives:

swoole

swoole support => enabled
Version => 4.2.1
Author => Swoole Group[email: team@swoole.com]
coroutine => enabled
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => enabled
http2 => enabled
pcre => enabled
zlib => enabled
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
redis client => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.aio_thread_num => 2 => 2
swoole.display_errors => On => On
swoole.use_namespace => On => On
swoole.use_shortname => On => On
swoole.fast_serialize => Off => Off
swoole.unixsock_buffer_size => 8388608 => 8388608

[ClosetGeek-Git]

Yes, the question was how to start multiple services on the same event loop. Not clients but listening server objects.

[ghost]

But you would not start a redis service - just using it with a client? What should a redis service be if not the client which uses the redis-server?

Are you talking about Swoole\Redis\Server? What the usecase for that, please?

[ClosetGeek-Git]

It's for creating custom redis services. A service like this is especially useful if your using redis just for messaging layer.

[ghost]

Ok, thats for sure a different question... But mine is morely done here - didn't know, could make sense to close the issue and open it as a new one :)

[twose]

@flddr about the async client coredump, please find your core file (ulimit -c unlimited), and use gdb then enter bt to check the call backtrace and the coredump position, create a new issue for the coredump.

[twose]

about multiple services on the same event loop, I will give you a general example soon.

[twose]

@ClosetMonkey @flddr I said I would give you an example: on https://github.com/swoole/swoole-src/blob/f11021e885d6cfc264fe18a797de73f5b368ec58/tests/swoole_http_server/mixed_server.phpt Only need fifty lines of code, we can run a mixed server (http/http2/websocket/tcp). It has not merge into master branch, you can remove open_http2_protocol and it will work fine.

$tcp_options = [
    'open_length_check' => true,
    'package_length_type' => 'n',
    'package_length_offset' => 0,
    'package_body_offset' => 2,
];

$pm = new ProcessManager;
$pm->initFreePorts(2);
// client side
$pm->parentFunc = function ($pid) use ($pm, $tcp_options) {
    go(function () use ($pm, $tcp_options) {
        // http
        $http_client = new Swoole\Coroutine\Http\Client('127.0.0.1', $pm->getFreePort(0));
        assert($http_client->post('/', 'Swoole Http'));
        var_dump($http_client->body);

        // http2
        $http2_client = new Swoole\Coroutine\Http2\Client('127.0.0.1', $pm->getFreePort(0));
        $http2_client->connect();
        $http2_request = new swoole_http2_request;
        $http2_request->method = 'POST';
        $http2_request->data = 'Swoole Http2';
        $http2_client->send($http2_request);
        $http2_response = $http2_client->recv();
        var_dump($http2_response->data);

        // websocket
        $http_client->upgrade('/');
        $http_client->push('Swoole Websocket');
        var_dump($http_client->recv()->data);

        // tcp
        $tcp_client = new Swoole\Coroutine\Client(SWOOLE_TCP);
        $tcp_client->set($tcp_options);
        $tcp_client->connect('127.0.0.1', $pm->getFreePort(1));
        $tcp_client->send(tcp_pack('Swoole Tcp'));
        var_dump(tcp_unpack($tcp_client->recv()));

        $pm->kill();
    });
};
// server side
$pm->childFunc = function () use ($pm, $tcp_options) {
    $server = new swoole_websocket_server('127.0.0.1', $pm->getFreePort(0), SWOOLE_BASE);
    $server->set([
        'worker_num' => 1,
        'log_file' => '/dev/null',
        'open_http2_protocol' => true
    ]);
    $server->on('workerStart', function () use ($pm) {
        $pm->wakeup();
    });
    // http && http2
    $server->on('request', function (swoole_http_request $request, swoole_http_response $response) {
        $response->end('Hello ' . $request->rawcontent());
    });
    // websocket
    $server->on('message', function (swoole_websocket_server $server, swoole_websocket_frame $frame) {
        $server->push($frame->fd, 'Hello ' . $frame->data);
    });
    // tcp
    $tcp_server = $server->listen('127.0.0.1', $pm->getFreePort(1), SWOOLE_TCP);
    $tcp_server->set($tcp_options);
    $tcp_server->on('receive', function (swoole_server $server, int $fd, int $reactor_id, string $data) {
        $server->send($fd, tcp_pack('Hello ' . tcp_unpack($data)));
    });
    $server->start();
};
$pm->childFirst();
$pm->run();

[ghost]

Very interesting, much thanks :)

[sauriio]

What about a server running and an async client on the same event loop? i've tried to figure it out but i couldn't. if you have a working example, please post it.

[dschissler]

@twose Please post an example with multiple async clients on the same loop. I can't use Swoole because this is my use case.

[twose]

@dschissler README: https://github.com/swoole/swoole-src#mysql https://github.com/swoole/swoole-src#kinds-of-clients https://github.com/swoole/swoole-src#the-simplest-example-of-a-connection-pool

[dschissler]

@twose Thanks for the links but I just can't figure out how to integrate ZMQ at this time. I will continue reading about Swoole until I can figure it out. The problem is that ZMQ would need to be wrapped and I have no idea how to go about doing that.

[ClosetGeek-Git]

@dschissler It should be possible to add ZMQ to the loop using https://github.com/swoole/zmq

[asavchenko]

@twose how to fix this code, I want to start coroutine server + HTTP server at the same time

<?php
require_once dirname(__DIR__) . "/vendor/autoload.php";

use Swoole\Coroutine\Server;
use Swoole\Coroutine\Server\Connection;

co\run(function () {
    go(function() {
        $http = new swoole_http_server("0.0.0.0", 49501);

        $http->on('request', function ($request, $response) {
            $response->end("<h1>Hello World. #".rand(1000, 9999)."</h1>");
        });
        $http->start(); // throws an error Swoole\Server::start(): eventLoop has already been created, unable to start Swoole\Http\Server
    });

    go(function() {
        $tcpServer = new Server("0.0.0.0", 9507);
        $tcpServer->handle(function (Connection $conn) {
            static $i = 0;
            $i++;
            go(function() use ($conn, $i) {
                echo "new connection has opened " . $i . PHP_EOL;
                while (true) {
                    if ($data = $conn->recv()) {
                        echo $i . ": " . $data . PHP_EOL;
                        $conn->send($data);
                    } else {
                        $conn->close();
                        break;
                    }
                }
            });
        });
        $tcpServer->start();
    });
});

[huanghantao]

@twose how to fix this code, I want to start coroutine server + HTTP server at the same time

<?php
require_once dirname(__DIR__) . "/vendor/autoload.php";

use Swoole\Coroutine\Server;
use Swoole\Coroutine\Server\Connection;

co\run(function () {
    go(function() {
        $http = new swoole_http_server("0.0.0.0", 49501);

        $http->on('request', function ($request, $response) {
            $response->end("<h1>Hello World. #".rand(1000, 9999)."</h1>");
        });
        $http->start(); // throws an error Swoole\Server::start(): eventLoop has already been created, unable to start Swoole\Http\Server
    });

    go(function() {
        $tcpServer = new Server("0.0.0.0", 9507);
        $tcpServer->handle(function (Connection $conn) {
            static $i = 0;
            $i++;
            go(function() use ($conn, $i) {
                echo "new connection has opened " . $i . PHP_EOL;
                while (true) {
                    if ($data = $conn->recv()) {
                        echo $i . ": " . $data . PHP_EOL;
                        $conn->send($data);
                    } else {
                        $conn->close();
                        break;
                    }
                }
            });
        });
        $tcpServer->start();
    });
});

Maybe you should use multiple processes:

<?php

use Swoole\Coroutine\Server;
use Swoole\Coroutine\Server\Connection;
use Swoole\Http\Server as HttpServer;
use Swoole\Process\Manager;

$pm = new Manager;

$pm->add(function () {
    $http = new HttpServer("0.0.0.0", 49501);

    $http->on('request', function ($request, $response) {
        $response->end("<h1>Hello World. #".rand(1000, 9999)."</h1>");
    });
    $http->start();
});
$pm->add(function () {
    $tcpServer = new Server("0.0.0.0", 9507);
    $tcpServer->handle(function (Connection $conn) {
        static $i = 0;
        $i++;
        go(function() use ($conn, $i) {
            echo "new connection has opened " . $i . PHP_EOL;
            while (true) {
                if ($data = $conn->recv()) {
                    echo $i . ": " . $data . PHP_EOL;
                    $conn->send($data);
                } else {
                    $conn->close();
                    break;
                }
            }
        });
    });
    $tcpServer->start();
}, true);

$pm->start();

[asavchenko]

huh, but new problems arose. It looks like I can't use channels within this solution. And can't do something like ch->push(command) from web and pass it to the connected listeners. I've tried using tables, but it's not exactly what I need. I just need an immediate reaction to commands. Send a command, receive it from another part of the logic, process that command, send back a response. Please help. @huanghantao

[huanghantao]

huh, but new problems arose. It looks like I can't use channels within this solution. And can't do something like ch->push(command) from web and pass it to the connected listeners. I've tried using tables, but it's not exactly what I need. I just need an immediate reaction to commands. Send a command, receive it from another part of the logic, process that command, send back a response. Please help. @huanghantao

Can you express what you mean in pseudo code?

[asavchenko]

Click to expand

```php add(function () use ($chan) { $http = new HttpServer("0.0.0.0", 49501); $http->on('request', function ($request, $response) use ($chan) { $chan->push(['action' => 'say_hi_from_web']); $response->end("

Hello World!

"); }); $http->start(); }); $pm->add(function () use ($chan) { $tcpServer = new Server("0.0.0.0", 9507); $tcpServer->handle(function (Connection $conn) use ($chan) { go(function() use ($conn, $chan) { while (true) { $command = $chan->pop(); switch ($command['action']) { case 'say_hi_from_web': $conn->send('hi from web!'); } } }); }); $tcpServer->start(); }, true); $pm->start(); ```

[huanghantao]

@asavchenko You can try this example:

<?php

use Swoole\Coroutine;
use Swoole\Coroutine\Server;
use Swoole\Coroutine\Server\Connection;
use Swoole\Http\Server as HttpServer;
use Swoole\Process;
use Swoole\Process\Manager;
use Swoole\Coroutine\Socket;

use function Swoole\Coroutine\run;

$proc = new Process(function (Process $proc) {
    $http = new HttpServer("0.0.0.0", 49501);

    $http->on('request', function ($request, $response) use ($proc) {
        /** @var Socket $socket */
        $socket = $proc->exportSocket();
        $ret = $socket->send(json_encode(['action' => 'say_hi_from_web']));
        $response->end("<h1>Hello World. #".rand(1000, 9999)."</h1>");
    });
    $http->start();
}, null, SOCK_DGRAM);

$proc->start();

run(function () use ($proc) {
    $tcpServer = new Server("0.0.0.0", 9507);

    $tcpServer->handle(function (Connection $conn) use ($proc) {
        Coroutine::create(function() use ($conn, $proc) {
            /** @var Socket $socket */
            $socket = $proc->exportSocket();
            while (true) {
                $command = $socket->recv();
                $command = json_decode($command, true);
                switch ($command['action']) {
                    case 'say_hi_from_web':
                        $conn->send('hi from web!');
                }
            }
        });
    });

    $tcpServer->start();
});

$proc->wait();