Very Slow ELK Events

[theplaceilive]

I'm currently running ELK Node Server v3.9.0 on a Polisy Pro running IoX Version 5.7.0 with PG3x Version 3.2.6.

I've noticed significant Polisy Program delays when an ELK door sensor or motion detector is violated. I've seen delays as long as 10 or 15 seconds from the time the ELK sensor was violated. This delay condition seems to happen only after Polisy has been running for a day or two.

FYI; This condition was experience prior to the recent 5.7.0 upgrade. It appeared somewhere along the line of IoX, PG3x and Node Server updates but don't know for sure when.

I've found performing just a reboot of Polisy temporarily resolves the issue and makes ELK events very responsive again. I've also tried disabling and reenabling just the ELK Node Server, but it didn't appear to resolve the delay issue.

Invoking the Polisy reboot, or Upgrading Packages on Polisy (which reboots) resolves the issue every time. It behaves as if system resources are being consumed after running for some time that requires a Polisy reboot to resolve.

I've reviewed logs but haven't seen anything conclusive. Please let me know if you need to see specific logs.

I'd appreciate your assistance to help resolve this issue.

[jimboca]

My guess would be that something else on your Polisy is chewing up all the cpu and slowing everything else down. Can you ssh to your polisy, run the command "uptime" and show me the output?

In the PG3 UI under Elk node sever -> Log -> Change "Level" to "Debug + Modules" then activate a sensor, then Download Log Package, and email that file to me at jimbo.automates@gmail.com and tell me the sensor name.

[theplaceilive]

Thanks for the quick response.

I rebooted my Polisy earlier today, so the ELK related events are currently working fine. I suspect the problem will begin to appear tomorrow or the next day. I'll provide the additional information needed when the condition reappears.

Thanks again!

[theplaceilive]

I noticed ELK event slight delays after my Polisy Pro was running (since its last reboot) for 1 day and significant delays after running for 2 days. I've since simulated the problem with the ELK Node Server Log level set to Debug + Modules and sent a copy of the log file via email to jimbo.automates@gmail.com. The ELK Zone used for the test was zone_30:Master Motion.

The ELK motion detector violation triggers an IoX program that turns on a light within the associated bedroom when dark. The delay between seeing the motion detector blink from the motion being detected and the light turning on was approximately 20-seconds. I monitored the associated Motion Detector node status in IoX and it was also delayed.

[theplaceilive]

I currently have a ticket open with Universal Devices. They'll be taking a look at my system remotely.

I refrained from rebooting for a couple of days and the delays are currently around 12 seconds long. Will provide an update accordingly.

[theplaceilive]

Universal Devices accessed my Polisy system. It appears upgrading PG3x to Version 3.2.8 on the existing IoX Version: 5.7.0 and removing an unselected IoX entry resolved the delay issue. My Polisy has been running for several days and no longer experiences ELK Node Server delays.

[jimboca]

That's great news. What was the removing an unselected IoX entry ?

[theplaceilive]

The MAC for my prior ISY994 was still listed within the IoX Configuration pulldown within the PG3x interface. While it wasn't selected as active, it was no longer needed so UD suggested it be removed.

It's not clear if the old "inactive / unselected" ISY entry was part of the issue. I knew that entry remained after migrating from ISY994 to IoX on Polisy, but it wasn't selected with a checkmark as being active. I originally used Polisy to host Node Servers for the ISY994 prior to migrating. Regardless, that old unselected ISY entry is now removed.

I forgot to mention, they also updated FreeBSD to version 13.2-RELEASE-p1 while they remotely accessed my Polisy. It's also not clear if the FreeBSD upgrade was related.

I personally suspect the update to PG3x 3.2.8 may have been the fix. The ELK Node Server delays started happening somewhere along the upgrade path of the various versions of IoX and PG3x.

[jimboca]

Thanks for the details.