Problem with more than 1k concurrent requests

[leoshtika]

Hi,

I have created an application that sends/receives packets using parallel non-blocking IO with Amp. It is running with the help of a cron-job (every 5 minutes) sending a concurrent request to modbus controllers (temperature sensors) and it saves all the responses.

Up to ~1K sensors everything is working fine.

When increased to 1.5k for about 30% of the requests I am getting the following message:

Connecting to tcp://...:502 failed: timeout exceeded (10000 ms)

Did I hit some kind of threshold?

I checked the Apache node and I have the following limits. It's much higher than 1000.

ulimit -Hn 524290 ulimit -Sn 524290 sysctl fs.file-max fs.file-max = 13065208

Here is the data collection method

private function collectData()
{
    $holdingRegistersRequests = ReadRegistersBuilder::newReadHoldingRegisters()
        ->allFromArray($this->_fc3Requests)
        ->build();
    $inputRegistersRequests = ReadRegistersBuilder::newReadInputRegisters()
        ->allFromArray($this->_fc4Requests)
        ->build();
    $requests = ArrayHelper::merge($holdingRegistersRequests, $inputRegistersRequests);

    $promises = [];
    foreach ($requests as $request) {
        $promises[] = call(function () use ($request) {
            try {
                $socket = yield connect($request->getUri());
                yield $socket->write($request);

                $data = yield timeout($socket->read(), self::TIMEOUT_MILLISECONDS);
                if ($data === null) {
                    return [
                        'addresses' => $request->getAddresses(),
                        'data' => null,
                    ];
                } else {
                    return [
                        'addresses' => $request->getAddresses(),
                        'data' => $request->parse($data),
                        'function_code' => $request->getRequest()->getFunctionCode()
                    ];
                }
            } catch (\Exception $exception) {
                return [
                    'addresses' => $request->getAddresses(),
                    'data' => $exception,
                ];
            } finally {
                if (isset($socket)) {
                    $socket->close();
                }
            }
        });
    }

    try {
        // Run multiple request in parallel using non-blocking php stream io
        $responses = wait(all($promises));
        $this->parseAndSave($responses);
    } catch (\Throwable $e) {
        Yii::info($e->getMessage(), __METHOD__);
        print_r($e->getMessage());
    }
}

Any suggestions would be greatly appreciated!

[aldas]

those requests go to some PLC? From my own experience I have seen PLCs (Wago for example) that can only do 10 parallel open sockets and over that I would see timeouts in php.

If you can, you should try to determine if the bottleneck is on PHP side or PLC side.

One way to alleviate this problem would be to create some kind of connection pool that limits connection to same target (PLC) to certain number of open sockets. From googling I see that Amp has some kind of limitpool already implemented. See this file https://github.com/amphp/http-client/blob/master/src/Connection/StreamLimitingPool.php

[aldas]

btw if you figure this out. please let me know and write about your solution here. This is now second time someone is asking similar question. Probably not the last one

[leoshtika]

Hello Martti, thank you for your quick response. You are right about the limitations of a modbus controller, but in my case I am requesting data from different controllers (different IPs from a VPN and external IPs). So I think the problem is in PHP side.

Any other ideas?

[aldas]

Okey, this needs some creative debugging.

maybe try to use ReactPHP to see if there is a difference. I have example https://github.com/aldas/modbus-tcp-client/blob/master/examples/example_parallel_requests_reactphp.php

maybe its AMP bottleneck

[aldas]

and if you divide your cron-jobs to per 1k and run N cron-jobs in parallel are problems still there? To see if there is a limit for one php process how many connections it can handle.

[leoshtika]

It seems that AMP needs a PHP extension ('ev', 'event' or 'uv') if the concurrent socket connections are more than 1024. Check here the requirements: https://github.com/amphp/amp#requirements Installing 'ev' and 'event' extension (I was not sure which one was better) fixed my problem. Now I have 1.5 connections and it is running without problem. Thank you for the support.