RouterOS reboot command terminates the API connection without response resulting in PHP timeout

[jonth93]

When sending the /system/reboot command with the read() function it waits on a response despite setting the timout and attempts.

Would be good if I could just send the command to the router singularly.

I have already tried the write() function, which passes and allows laravel to continue but doesn't actually execute on the router.

Version of RouterOS 6.44.5

To Reproduce $this->client = new Client([ 'host' => $router->ip->address, 'user' => %%, 'pass' =>%%, 'timeout' => 5, 'attempts' => 1, 'legacy' => ($router->legacy == 0 ? false : true) ]); $this->client->write('/system/reboot')->read();

Expected behavior Not wait on responce and return.

[jonth93]

Traced to line 53 in 'routeros-api-php\src\Streams\ResourceStream.php'

// TODO: Ignore errors here, but why? $result = @fread($this->stream, $length);

[EvilFreelancer]

Hello! Hm, interesting issue, error was ignorred due to custom exception reason, but probably better to read error from fread and add it to exception. I'll check it today

[jonth93]

Figured it out.

When FRead timeouts out it returns a result of ''

Changing the read funciton in ResourceStream.php slightly, allows the process to continue without throwing a Larvel exception, or php maximum timeout exception.

Start line 54 if (is_resource($this->stream)) { $result = @fread($this->stream, $length); if($result == '') { throw new StreamException("Connection lost"); } }

There may be a more elegant way of doing this, but this worked for me.

[matracine]

Hello @jonth93 , I had this problem recently. The fact is sometimes, the ROS API returns legitimate empty result and this case does not have to be treated has an error. You have to determine if this return value is due to fread error or a regular return value from ROS API... In my case, after a timeout the fread starts to reply quickly with empty results, so in the read method I could catch this by looking for consecutives empty responses (>3) and then sending a protocol error exception.

@EvilFreelancer : The reboot hang problem can be handled by detecting the "!fatal" response from the API, read the result and then close the connexion. But this need to change some logic of your API, you have to parse the results sent by the ROS API and take actions in consequence.

[jonth93]

Hi @matracine

Thanks for the feedback.

What I've done in the mean time since the API can return an empty result is read the stream meta data for the timed_out value, as so:

if(stream_get_meta_data($this->stream)['timed_out'] == true) { throw new StreamException("Connection lost"); }

This does add a delay to the reboot command result. Dumping the results show that I get a blank string, then !done, then several blanks. Attempting to read each output from fread still results in a timeout depsite the !done being present.

Not sure of a way to read each output from fread to assess if !done is present or if that is even necessary?

[matracine]

Hi @jonth93 , You are right, no !fatal is emmited by the router when a reboot command is sent (just tested it). When I reboot my router, I've got : !done <blank> <Disconnection>

Reading the Client::readRaw method, it seems to catch the case by checking a !done response then a emty line before terminating the read loop. So I tried to reproduce your problem without succes : no timeout, the router reboots and the read() call returns an empty result set without blocking. Tried on 6.40.4 and 6.44.5 ROS versions.

[Alion548]

RouterOS cannot reconnect automatically after reboot.

[asafov]

My solution:

1. Open console and use this:

/file print file=reboot.txt    
/file set [find name="reboot.txt"] contents=0

2. Go to web interface System -> Scripts -> Add new Name: reboot-device

Policy: all

Source: /file set reboot.txt contents="1"

3. Go to web interface System -> Scheduler -> Add new Name: watch-reboot

Interval: 1 min

Policy: all

On Event:

:local needReboot [/file get reboot.txt contents];

:if ($needReboot != 0) do={
    /file set "reboot.txt" contents="0"
    /system reboot
}

At client:

    protected $client = NULL;

    public function __construct ()
    {
        $this->client = new \RouterOS\Client([
            'host' => get_system_setting ('mikrotik_api_host'),
            'user' => get_system_setting ('mikrotik_api_user'),
            'pass' => get_system_setting ('mikrotik_api_pass'),
            'port' => get_system_setting ('mikrotik_api_port'),
            'ssh_port' => get_system_setting ('mikrotik_ssh_port'),
        ]);
    }
...
    public function reboot () {
        $data = $this->client->query(['/system/script/run', '=.id=reboot-device'])->read (false);
    }

[EvilFreelancer]

Hello @asafov you may also try it via SNMP:

https://wiki.mikrotik.com/wiki/Manual:SNMP

~ $ snmpset -c public -v 1 192.168.1.1 1.3.6.1.4.1.14988.1.1.7.1.0 s 1

• public - name of SNMP community
• 192.168.1.1 - IP-address of router
• 1.3.6.1.4.1.14988.1.1.7.1.0 - OID, which mean reboot

But also need not forgot to enable SNMP server on router and configure communities.

[naive17]

I got that devastating issue on my 1072 with 6.47 ros, any permanent solution to continue using these apis (without a reboot, i need them for my main customers network )?

[EvilFreelancer]

@koso00 hello! I need try to reproduce this issue, when my connection lost the library reconnects. Maybe some issues with hardware or some traffic loops issue on your network? Did you checked CPU usage on routers?

If device is not available then library can't connect to device, probably in similar situations best way is reboot via SNMP

[naive17]

@koso00 hello! I need try to reproduce this issue, when my connection lost the library reconnects. Maybe some issues with hardware or some traffic loops issue on your network? Did you checked CPU usage on routers?

Yep absolutely, my network is stable since years and as soon as i started using these apis with casual timing at the login the router will hang and it will need to be rebooted. The router is perfectly okay with low cpu usage on average

[EvilFreelancer]

How do you use the library? In the sense of what data do you get from routers? Traffic statistic, maybe list of connected users via hotspot?

[naive17]

I remove ip from an address list and then repopulate it with other ips, max 50/60 per time

[EvilFreelancer]

So, you removed IP through which the library was connected to API of device and then connection was lost?

[naive17]

Nope, i made a script to block a bunch of ips with this library, it just work for like a day and then it crashes the router when the api library connects (we monitored logs and when the router blocks it's caused by the api access). The script run every 10 minutes and with the previous library i used it worked like a charm for years. As soon as i switched to ssh or another library the reboots disappeared.

[EvilFreelancer]

@koso00 can you please create a separate issue with a sample code (which use the library) that hangs your router?

[majsterkoo]

Hi, when I run reboot command, read() function going to freeze without timeout effect. When I try simple debug, it is freeze on fread command. How can I achieve to timeout when connection is lost on reboot?

$client = new \RouterOS\Client([
      'host' => '****',
      'user' => 'admin',
      'pass' => '*****',
      'timeout' => 5,
      'attempts' => 1,
      //'legacy' => ($router->legacy == 0 ? false : true)
]);
$client->write('/system/reboot')->read();

I tried version 1.3.2 & 1.2.2. In my case, simple timeout is ok...

EDIT: Ok, actually, I set

stream_set_blocking($socket, TRUE );

in openSocket procedure. After this the stream_set_timeout is working for lost connection... and then in readRAW while loop I'm detecting number of timeout and when it reach attempts, it thrown timeout exception. Meybe not good solution, but for me for now, its ok...

[EvilFreelancer]

@majsterkoo nice idea, thanks, i'll try it.

[majsterkoo]

For all who are struggling with this problem:

Full changes in source code for working timeout below. After changes, you need just set timeout and blocking to true in Config object.

// Create config object with parameters
$config =
    (new Config())
        ->set('host', '127.0.0.1')
        ->set('pass', 'admin')
        ->set('user', 'admin')
        ->set('ssh_port', 22222)
        ->set('timeout', 30)
        ->set('blocking', true);

// Initiate client with config object
$client = new Client($config);

Tested for no ssl and ssl connections. Reconnecting after timeout needs to be solved out of this library. I recommend using timeout bigger than a few seconds (slow reading, waiting for changes in config etc). I think good value is about a 30 second.

Question before PR is if this an acceptable solution or there is a need for another enhancements (for example: autoreconnect in library).

https://github.com/majsterkoo/routeros-api-php/commit/0dbaede834de6c04bd6b69700b368a718faa7e65

[EvilFreelancer]

@majsterkoo Hello! Yes, it is acceptable solution, you may create a PR, and on review stage will need to think about enabling this feature by default (if it's not break current stable version).

[majsterkoo]

@EvilFreelancer Meybe using new config param instead of timeout? Because timeout is using during creating connection if I'm not wrong? This timeout is defaultly set to 10 second. If you set bigger value (30, 60, 90 sec) you will be waiting during trying to connect into offline client too long before timeout. But when you use small value, then your connection timed out to early (I think 10 second is so small for waiting to wireless changes (when client disconnect from AP) on wifi client etc...). Meybe add socket_timeout value to config? I can add it to PR.

[EvilFreelancer]

@majsterkoo hello!

https://github.com/EvilFreelancer/routeros-api-php/blob/master/src/SocketTrait.php#L64

Here is the usage of stream_set_timeout (alias of socket_set_timeout), options parameter is timeout, so for changing socket timeout you just need to change this parameter.

[majsterkoo]

Yes, but stream_set_timeout is used in stream_socket_client too. And this is two different situations.

From php docs ---

stream_socket_client(
    string $remote_socket,
    int &$errno = ?,
    string &$errstr = ?,
    float $timeout = ini_get("default_socket_timeout"),
    int $flags = STREAM_CLIENT_CONNECT,
    resource $context = ?
): resource

timeout: Number of seconds until the connect() system call should timeout.

Note: To set a timeout for reading/writing data over the socket, use the stream_set_timeout(), as the timeout only applies while making connecting the socket. ---

So I think it would be good to have these values separate.

Now there is one timeout property, so you can have

• timeout for reading/writing and connect_timeout for connection timeout.
• timeout for connection timeout and socket_timeout for reading/writing.
• something else...

[EvilFreelancer]

Oh, I did not know this, thanks a lot for the explaining, I will add an option now.

[EvilFreelancer]

Hello everyone! I've added in 1.4.1 a few timeouts:

• timeout for connecting to router via socket (it was already)
• socket_timeout for reading answers from buffer
• ssh_timeout for reading /export results.

Check readme for details.

[EvilFreelancer]

Hope it helps to solve this issue about reboot :)

[EvilFreelancer]

Thanks to @majsterkoo for adding socket_blocking parameter of config, it should help with this issue.

Please update to 1.4.2 for testing of new feature.

[EvilFreelancer]

Seems everything works fine now.