search

MadaMzandu / uisp-ros-plugin

This is a follow up replacement for the previous Uisp api for mikrotik routers.
32 stars 9 forks source link

Plans try to load #17

Closed solufi closed 2 years ago

solufi commented 2 years ago

Hi,

I installed the plugin, Added mikrotik device but when I want to got in « plan » the page just keep loading and never show up.

Already have multiple plan made in UISP.

maybe you have a solution I am in version 1.4.3 of UISP

MadaMzandu commented 2 years ago

Hi, Please confirm the webhook was created after the installation. There is an installation video at below link if it will be helpful.

https://user-images.githubusercontent.com/59873984/154258060-21ae43bd-2b5b-4a22-b6e3-601f6ca64cd0.mp4

Thanks

solufi commented 2 years ago

I tried, the webhook is working if I test the webhook. The section ''plan'' just keep loading. I tried on 2 different UISP Vm and they have both the same problem. The mikrotik router is green , but plan just keep loading

MadaMzandu commented 2 years ago

It sounds like plugin is failing to access the ucrm rest api. Try to check for errors from console when accessing the plans page

tail -n 100 -f /home/unms/data/ucrm/log/ucrm/nginx/ucrm_error.log

Otherwise plugin works on 1.4.3 and 1.4.4 without problems.

solufi commented 2 years ago

Did you already saw that problem : (this is my UISP, the one of a friend working fine)

2022/05/02 19:16:26 [error] 794#794: 572229 FastCGI sent in stderr: "PHP message: PHP Warning: Invalid argument supplied for foreach() in /data/ucrm/data/plugins/ros-plugin/lib/admin_devices.php on line 202" while reading response header from upstream, client: 172.18.251.4, server: , request: "POST /crm/_plugins/ros-plugin/public.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9021", host: "portail.solufi.ca" 2022/05/02 19:16:26 [error] 794#794: 572229 FastCGI sent in stderr: "PHP message: PHP Warning: Cannot modify header information - headers already sent by (output started at /data/ucrm/data/plugins/ros-plugin/lib/admin_devices.php:202) in /data/ucrm/data/plugins/ros-plugin/lib/api_router.php on line 103" while reading upstream, client: 172.18.251.4, server: , request: "POST /crm/_plugins/ros-plugin/public.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9021", host: "portail.solufi.ca" 2022/05/02 19:16:32 [error] 794#794: 572260 FastCGI sent in stderr: "PHP message: PHP Warning: Invalid argument supplied for foreach() in /data/ucrm/data/plugins/ros-plugin/lib/admin_devices.php on line 202" while reading response header from upstream, client: 172.18.251.4, server: , request: "POST /crm/_plugins/ros-plugin/public.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9021", host: "portail.solufi.ca" 2022/05/02 19:16:32 [error] 794#794: 572260 FastCGI sent in stderr: "PHP message: PHP Warning: Cannot modify header information - headers already sent by (output started at /data/ucrm/data/plugins/ros-plugin/lib/admin_devices.php:202) in /data/ucrm/data/plugins/ros-plugin/lib/api_router.php on line 103" while reading upstream, client: 172.18.251.4, server: , request: "POST /crm/_plugins/ros-plugin/public.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9021", host: "portail.solufi.ca" 2022/05/02 19:16:33 [error] 794#794: 572265 FastCGI sent in stderr: "PHP message: PHP Warning: Invalid argument supplied for foreach() in /data/ucrm/data/plugins/ros-plugin/lib/admin_devices.php on line 202" while reading response header from upstream, client: 172.18.251.4, server: , request: "POST /crm/_plugins/ros-plugin/public.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9021", host: "portail.solufi.ca" 2022/05/02 19:16:33 [error] 794#794: 572265 FastCGI sent in stderr: "PHP message: PHP Warning: Cannot modify header information - headers already sent by (output started at /data/ucrm/data/plugins/ros-plugin/lib/admin_devices.php:202) in /data/ucrm/data/plugins/ros-plugin/lib/api_router.php on line 103" while reading upstream, client: 172.18.251.4, server: , request: "POST /crm/_plugins/ros-plugin/public.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9021", host: "portail.solufi.ca" 2022/05/02 19:29:58 [error] 794#794: 574891 FastCGI sent in stderr: "PHP message: PHP Warning: Invalid argument supplied for foreach() in /data/ucrm/data/plugins/ros-plugin/lib/admin_devices.php on line 202" while reading response header from upstream, client: 172.18.251.4, server: , request: "POST /crm/_plugins/ros-plugin/public.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9021", host: "portail.solufi.ca" 2022/05/02 19:29:58 [error] 794#794: 574891 FastCGI sent in stderr: "PHP message: PHP Warning: Cannot modify header information - headers already sent by (output started at /data/ucrm/data/plugins/ros-plugin/lib/admin_devices.php:202) in /data/ucrm/data/plugins/ros-plugin/lib/api_router.php on line 103" while reading upstream, client: 172.18.251.4, server: , request: "POST /crm/_plugins/ros-plugin/public.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9021", host: "portail.solufi.ca" 2022/05/03 01:13:11 [error] 794#794: 625354 FastCGI sent in stderr: "PHP message: PHP Warning: Invalid argument supplied for foreach() in /data/ucrm/data/plugins/ros-plugin/lib/admin_mt_contention.php on line 9" while reading response header from upstream, client: 172.18.251.4, server: , request: "POST /crm/_plugins/ros-plugin/public.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9021", host: "portail.solufi.ca" 2022/05/03 01:13:11 [error] 794#794: 625354 FastCGI sent in stderr: "PHP message: PHP Warning: Cannot modify header information - headers already sent by (output started at /data/ucrm/data/plugins/ros-plugin/lib/admin_mt_contention.php:9) in /data/ucrm/data/plugins/ros-plugin/lib/api_router.php on line 103" while reading upstream, client: 172.18.251.4, server: , request: "POST /crm/_plugins/ros-plugin/public.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9021", host: "portail.solufi.ca"

MadaMzandu commented 2 years ago

The error shows the plugin is not able to pull both the list of service plans and devices. This could be a dns problem on the VM. Is it possible to confirm that the VM is able to resolve portail.solufi.ca? Try:

  1. On the VM ping portail.solufi.ca and see if this resolves to the correct ip address or
  2. Try to access the crm using ip address instead of hostname just to confirm if the problem is dns related.
solufi commented 2 years ago

Yes it’s all working/resolving perfectly that’s strange

MadaMzandu commented 2 years ago

If you are able to provide anydesk access you can use my public email address and I can have a closer look. That way if its a bug, I can pick it up.

kostopravby commented 2 years ago

Get the same problem but with list of devices on mikrotik. Cycling requests with the same payload: image payload: changeType":"admin","target":"devices","action":"services Browser console: The resource https://xxxx/crm/js/routing?callback=fos.Router.setData was preloaded using link preload but not used within a few seconds from the window's load event. Please make sure it has an appropriate as value and it is preloaded intentionally. public:1 The resource https://xxx/crm/_ws/socket.io/socket.io.js was preloaded using link preload but not used within a few seconds from the window's load event. Please make sure it has an appropriate as value and it is preloaded intentionally. UISP 1.4.4

MadaMzandu commented 2 years ago

Hi Solufi's problem was resolved by renaming one of the service plans. This is a problem with the plugins cache failing to handle some Utf8 (French) characters. I don't think this is the same problem, have you added any devices at all.

kostopravby commented 2 years ago

yes. after 1-3 min of waiting I'm able to see one device on my mikrotik. all requests have the same request parameters and the same response. so I can't understand why the same information is requested several times

On Thu, May 5, 2022, 15:52 Madalitso Mzandu @.***> wrote:

Hi Solufi's problem was resolved by renaming one of the service plans. This is a problem with the plugins cache failing to handle some Utf8 (French) characters. I don't think this is the same problem, have you added any devices at all.

— Reply to this email directly, view it on GitHub https://github.com/MadaMzandu/uisp-ros-plugin/issues/17#issuecomment-1118509834, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHESJIMNKDYWBQNKVTEWPRDVIO77FANCNFSM5UUTLIEA . You are receiving this because you commented.Message ID: @.***>

MadaMzandu commented 2 years ago

The page polls for the status and stats of the devices every few seconds, that is why you are seeing multiple request. After the page loads is the status of the device green - online or grey - offline?

kostopravby commented 2 years ago

mikrotik status is green. tabs with settings and service plans load fast. but when I click on the mikrotik on "devices" tab I must wait for 1-3 minutes before the device list will be shown

On Thu, May 5, 2022, 16:12 Madalitso Mzandu @.***> wrote:

The page polls for the status and stats of the devices every few seconds, that is why you are seeing multiple request. After the page loads is the status of the device green - online or grey - offline?

— Reply to this email directly, view it on GitHub https://github.com/MadaMzandu/uisp-ros-plugin/issues/17#issuecomment-1118531873, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHESJIKXKSGTPDP3U4ST2ITVIPCLHANCNFSM5UUTLIEA . You are receiving this because you commented.Message ID: @.***>

MadaMzandu commented 2 years ago

It sounds like there is some delay between the mikrotik and the ucrm. This could be due network delay between the ucrm server and the mikrotik or slow mikrotik response due to low resources. Are you able to verify ping responses between ucrm and mikrotik and also that mikrotik is not responding normally.

kostopravby commented 2 years ago

webhooks work very fast. so I think it is not a problem with network connectivity. I see log in attempts via api on mikrotik when webhook works but I don't see any log in attempts when I open devices list.

MadaMzandu commented 2 years ago

If there are no accounts on the mikrotik try deleting the device and see if the page loads faster. The page does not login when loading it just checks if mikrotik's port 8728 is responding, so you wont see a log entry.

kostopravby commented 2 years ago

Probably I found the problem. https://github.com/MadaMzandu/uisp-ros-plugin/blob/0a83f3faa8c824084519a0572b0ba1e0fa8a3de0/src/lib/admin_devices.php#L27 If you have 10-20 clients this is ok. But if you have 3-5k clients it will be slow. for example in my production UISP this request takes 28.37s

MadaMzandu commented 2 years ago

This could be a problem but it's not used during the device listing. But the following could be the problem.

https://github.com/MadaMzandu/uisp-ros-plugin/blob/0a83f3faa8c824084519a0572b0ba1e0fa8a3de0/src/lib/admin_devices.php#L244

I will verify and let you know

MadaMzandu commented 2 years ago

I have verified this call is able to respond in max 4 milliseconds on a system with 10,000 clients. This cannot be the cause of the problem.

plugin

kostopravby commented 2 years ago

Made some changes in file: https://github.com/MadaMzandu/uisp-ros-plugin/blob/0a83f3faa8c824084519a0572b0ba1e0fa8a3de0/src/lib/admin_devices.php#L25

use Ubnt\UcrmPluginSdk\Service\PluginLogManager;
public function services(): void
    {
        $log = PluginLogManager::create();
        $start = microtime(true);
        $services = $this->get_map('clients/services');
        $services_end = $start-microtime(true);
        $clients =  $this->get_map();
        $clients_end = $start-microtime(true);
        $records = $this->service_records();
        $records_end = $start-microtime(true);
        $ids = array_keys($records);
        $this->result = [];
        foreach($ids as $id){
            $this->result[$id] = $services[$id];
            $this->result[$id]['attributes'] = $this->fix_attributes($services[$id]['attributes']);
            $this->result[$id]['client'] = $clients[$services[$id]['clientId']];
            $this->result[$id]['record'] = $records[$id];
        }
        $end= microtime(true)-$start;
        $msg='services-'.count($services).' Execution time: services-'.$services_end.' clients-'.$clients_end.' records-'.$records_end.'Final-'.$end;
        $log->appendLog($msg);
}

Open plugin, click on mikrotik to see devices list. plugin.log in data folder:

services-176 Execution time: services--0.90634918212891 clients--21.899232149124 records--21.921039104462Final-21.921066045761
services-176 Execution time: services--0.93416595458984 clients--26.862191915512 records--26.863629102707Final-26.863658905029
services-176 Execution time: services--0.75213694572449 clients--25.840822935104 records--25.841444015503Final-25.841464996338
services-176 Execution time: services--1.6741211414337 clients--21.097496986389 records--21.098315000534Final-21.098341941833
services-176 Execution time: services--1.1164231300354 clients--17.966539144516 records--17.967211008072Final-17.967232942581
services-176 Execution time: services--1.3928837776184 clients--19.92697095871 records--19.933280944824Final-19.933397769928
services-176 Execution time: services--3.0287520885468 clients--24.080121040344 records--24.084404945374Final-24.084428071976
services-176 Execution time: services--2.2682089805603 clients--25.855792999268 records--25.856449842453Final-25.856472969055
MadaMzandu commented 2 years ago

This is the expected behavior AFTER the device listing when you click one of the listed devices to list the services on the device. I misunderstood that you were having trouble listing the devices NOT listing the services on the device.

Unfortunately the ucrm api does not provide a method to list the services gradually. Not only that but the ucrm api is not very responsive to begin with.

Perhaps in a future release the ucrm service data will be cached and the cache will be used for the listing. This will allow more responsive listing.

MadaMzandu commented 2 years ago

Correction on the above, the ucrm api call for listing services now has filters for limit,offset and custom attributes , so probably the next version will be optimized for responsiveness.

kostopravby commented 2 years ago

My researches: table "services" contains serviceId and clientId fields. so we can request from ucrm services and clients by ID. something like this

public function services(): void {
        $records = $this->service_records();
        $this->result = [];

        foreach ($records as $record) {
            $service = $this->get_map('clients/services/' . $record['id']);
            $client = $this->get_map('clients/' . $record['clientId']);
            $id = $record['id'];
            $this->result[$id] = $service;
            $this->result[$id]['attributes'] = $this->fix_attributes($service['attributes']);
            $this->result[$id]['client'] = $client;
            $this->result[$id]['record'] = $record;
        }
    }
private function service_records(): array {
        $id = $this->data->id;
        $services = $this->db()->selectServicesOnDevice($id);

        return $services;
    }
 private function get_map($type = 'clients'): array {
        $api = new API_Unms();
        $api->assoc = true;
        $array = $api->request('/' . $type);

        return $array;
    }

Also there is many unnecessary fields in response for request "changeType: "admin", target: "devices", action: "services".... seems like only client name, client type and several custom attributes are needed. will be nice to modify function "get_map" to return only necessary fields depending on type (client or service). also request payload data must contain only device ID without username, password and etc. Will be nice to see ability to set update interval or disable auto update for "Client services" table. It can significantly reduce average response time and server load

MadaMzandu commented 2 years ago

Thanks for the valuable input. I will work most of your suggestions into the the next release. I think now that the api call has filters, it will be easy to make the listing more responsive. I will keep this issue open until the next release.

MadaMzandu commented 2 years ago

This has been resolved in version 1.8.5 with the introduction of a cache. The limit and offset options introduced in the crm api do not improve the performance of the crm api, therefore a cache has been prefered as it provides a better user experience. Please test and let me know if there are problems