light status and motion sensor states are not reliably detected

pigeonman99 commented 6 months ago

I have a liftmaster security 2.0 garage opener and a "smart" control panel with motion sensor. just installed and wired up ratgdo with homekit-ratgdo firmware yesterday.

everything works fine except i've noticed that light status and motion sensor states are not reliably detected. i did the following:

i walk up to the control panel
motion sensor in the control panel automatically turns on the light

then i observed that maybe half of the time the firmware would say "Motion: true, Light On: true", but sometimes both just stay false as if nothing happened. also because of that, my homekit light would stay out-of-sync with the actual light status.

wonder if that's a known issue?

jgstroud commented 6 months ago

I have never noticed this behavior with mine. Is the webUI correct when this happens?

pigeonman99 commented 6 months ago

No, the WebUI is also incorrect when this happens. i.e. when i walk up to the panel and the physical light is turned on, WebUI would still show "Light On: false" and "Motion: false".

I also noticed that sometimes when i walk up to the panel and i checked the WebUI, the device seems to be lagging. I tried refreshing the webpage and it would sometimes give me an error page. Also noticed that sometimes (but not always), about 30 seconds later, eventually it'll show "Light On: true". I suspect there's some sort of lagging issue going on.

my ratgdo is 2.53 btw... i just very recently purchased it.

jgstroud commented 6 months ago

Your unresponsiveness makes me think there is some network issue. Do you have a good wifi connection? Are you by any chance running homebridge on your network and using the BonjourHAP mdns backend? There is a know issue, which I really should document, where this mdns backend sends malformed mdns packets which wreaks havoc on the ratgdo.

pigeonman99 commented 6 months ago

Yes I have a good wifi connection. I have 3 UniFi APs spread across the house. One of them is right above the garage. I'm able to consistently ping the device and it would respond quickly. Also, when I press the WebUI or Homekit button to open/close the garage door, it responds immediately every time without fail.

I have homebridge and I'm on Ciao mdns not Bonjour.

jgstroud commented 6 months ago

I can't speak to ciao, I only know bonjourhap has a problem and avahi works well. This is likely not your problem given the other info you provided. Do you have multiple homehubs?

pigeonman99 commented 6 months ago

by "home hubs" do you mean homekit homes? if so, then no i only have one.

jgstroud commented 6 months ago

Yes, I meant homekit hubs, but it doesn't matter looking back at some of the info you provided. I think we need serial logs to see what's happening. Seems like maybe something flakey in the serial communications. If you can connect a usb cable to it while it's running and grab the output of the serial logs that would be great. We'll have remote logging enabled in the next release, but I haven't had time to get that fully tested and released yet. Hopefully soon.

pigeonman99 commented 6 months ago

sure i can do that, but how do i connect to the device via usb "while it's running"? if i disconnect the usb cable from power source the device would power off.

pigeonman99 commented 6 months ago

or do you mean i should plug the usb to my laptop to power it on, perform the same actions, and capture the log? if so yeah i can do that.

jgstroud commented 6 months ago

yes, unfortunately that's the only way at the moment.

pigeonman99 commented 6 months ago

Here's the log file. This is what I did:

plug the device via USB to my laptop. The garage light is initially on because I'm already inside the garage and triggered the sensor.
I walked away from the garage, waited about 5 min until the homekit light status says "off"
I walked into the garage and triggered the motion sensor to turn the light on.
It took about 4 min 20 sec for the homekit light status to say "on". (I also checked the WebUI and it was the same status)
4.5 min later, the garage light turned off automatically
It took about 5 min for the homekit light status to say "off". (again, same for WebUI)
I went back to the garage to download the log file, which then triggered the motion sensor and the light to turn on again.

esp-web-tools-logs.txt

jgstroud commented 6 months ago

It looks like your ratgdo isn't seeing any of the messages from your wall panel. I never see it detect a motion packet. Normally when we see a motion detect event, we send a getStatus message which tells us the light is on.

pigeonman99 commented 6 months ago

hmm, any ideas why this might be happening? the motion sensor on the panel is definitely working, because when i walk in front of it, it automatically turns on the light.

I also saw a few times that the WebUI actually showed "Motion: true" when i walk in front of the panel. But that only happened 2-3 times. Most other times, the WebUI stays "Motion: false".

jgstroud commented 6 months ago

It's really hard to say. Based on what you describe, my next step would probably be to look at the signal on an oscilloscope. Seems to me like you have no issues with receiving messages from the GDO itself, but you do have issues with the signal from the wall panel. And it seems like occasionally it actually does get a message. It just sounds like marginal signal from the wall panel. The GDO and the ratgdo may have different switching thresholds which is why the gdo works. I wonder if @PaulWieland has any thoughts on why you wouldn't see the commands from the wall panel.

pigeonman99 commented 6 months ago

I just tried re-flashing the device with esphome-ratgdo, and it was able to detect motion sensor and light status correctly. See log below:

jgstroud commented 6 months ago

Ok. That's really interesting because the logs with the homekit firmware weren't even seeing the motion messages at all. If you switch back is it still broken?

jgstroud commented 6 months ago

Your logs didnt even show receiving an unknown command. I looked at the esphome code and it seems to be doing the same thing.

pigeonman99 commented 6 months ago

I flashed back to homekit-ratgdo firmware (i'm on 1.3.2 now), and now my device is super unresponsive... the homepage doesn't load completely after more than a minute, the homekit device says "no response"... not sure what's going on. i'm gonna try to re-flash it again later to see if this fixes it.

jgstroud commented 6 months ago

wow, that's so strange. you should do a full erase when you switch firmwares. not sure if you did that or not.

jgstroud commented 6 months ago

just comparing the code between the esphome firmware and ours, it seems to be doing all the same things. Your logs seem to have unusual homekit traffic, but I can't put my finger on what or why. Also, in your logs it rebooted without any crashlog. I think this was a watchdog timeout.

Adding @dkerr64 to see if he has any ideas.

dkerr64 commented 6 months ago

Very baffling symptoms. I would start by setting up a laptop or desktop with open browser, enable logging to javascript console, open up the console and confirm that you see log messages.

Then using a phone/tablet open the same webpage. You can use that to turn on/off lights, do that and watch the javascript console and check that the logs show that the light on/off commands are seen. If that is all good, then go to the garage with your phone/tablet and turn lights on/off from the wall panel. The phone/tablet should update right away, and when you go back to your laptop the console should have a log of it all.

pigeonman99 commented 6 months ago

i just re-flashed to 1.3.2 and made sure i've checked the box to erase everything before flashing. now the device is still really laggy. this seems to be a new problem since 1.3.2 - when i pnig the device i'm getting almost half of the packets timing out. the WebUI now times out really frequently. I added the device to homekit and now it frequently shows "No Response" for the garage door.

I swear that didn't happen with 1.2.1 from the last few days. When i was on 1.2.1, homekit never showed me "No Response". This seems to be a different wifi connection related issue since 1.3.2?

jgstroud commented 6 months ago

I just double checked and both of my devices are pinging consistently with < 5ms ping times and 0 drops. Just wanted to make sure we didnt have some major regression. You stated you have 3 UniFi APs. Can you double check the device is connecting to the nearest AP? Maybe even try locking it to that nearest AP.

There are some differences in how this firmware operates than the esphome variant since we run the homekit stack on device. I wonder if you have some network configuration that's causing an issue. If you are up for it, you can run Wireshark on your computer and have it remotely look at the network traffic on your UnifiAP and see if you see anything interesting. Your symptoms seem very similar to what I saw when running the BonjourHAP mdns (which you already confirmed you are not using). However, maybe something else on your network is causing an issue.

pigeonman99 commented 6 months ago

I just tried doing 300 pings over wifi when I was still on homekit-ratgdo. Then flash to esphome-ratgdo and did another 300 pings. Nothing else has changed. here are the results:

with homekit-ratgdo 1.3.2:

% ping -c 300 10.x.x.x
--- 10.x.x.x ping statistics ---
300 packets transmitted, 153 packets received, 49.0% packet loss
round-trip min/avg/max/stddev = 4.555/66.842/286.180/67.124 ms

with esphome-ratgdo 2024.4.1:

% ping -c 300 10.x.x.x
--- 10.x.x.x ping statistics ---
300 packets transmitted, 300 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.357/7.134/67.252/7.225 ms

For each time, I double checked with my UniFi controller and ensured the device was connected to the nearest AP. From the UniFi controller, when I was on homekit-ratgdo, I saw "WiFi Experience" hovering around 80% to 90% (yellow zone). But when I was on esphome-ratgdo, it was consistently at 100% (green zone).

I think this proves that there are some wifi connection issues with the latest homekit-ratgdo firmware, at least in the particular network setup that i'm in. I can try upgrading my UniFi APs to the latest version and then try again later.

jgstroud commented 6 months ago

Don't take my previous comment the wrong way. I totally believe you are having issues. I only compared to my devices to make sure we didn't have a fundamental regression,. And despite what some people think, there is no competition here with the esphome firmware. If that works for you, that might be your best solution. I would however, like to help you resolve your problem and hopefully we fix a bug in the process.

Did you by any chance check to see if the device is connecting to the correct AP? Maybe it's connecting to a far away AP and causing issues.

The Unifi wifi experience is a bit of magic. It can be yellow for a lot of reasons. As a minor correction to your statement, it proves there is a network problem, but I can't say for sure it's a WiFi problem. My point being, as I stated earlier, things work a little different with the HK firmware given it runs the HK stack on devices. So, it might not be a Wifi connection issue, but more to do with the actual network traffic.

jgstroud commented 6 months ago

Sorry, I see you did answer my question about it being connect to the correct AP

jgstroud commented 6 months ago

I kinda don't think this is a Wifi issue, but for the moment won't rule it out. I think there are only minimal things that could be different there from the esphome variant.

Sounds more like a network issue to me. There are a lot of things I would try and if you are willing to work through them, I would really like to understand this problem.

First if you device is paired to homekit, can you start by removing it from homekit and reset the homekit setting on the ratgdo. See if the ping problem persists.

Another thing to try is put the device on an isolated SSID / VLAN. Put your laptop on the same VLAN and see if the pings return to normal.

Then I would like for you to do a Wireshark packet capture. You don't have to share that with me if you are not comfortable with it, but maybe I can help walk you through things to look for. If you dont know how to do this, I can walk you through it. Its not too hard.

pigeonman99 commented 6 months ago

@jgstroud thanks so much for your help so far! I too just want to do as much as I can to help make this firmware better.

I've been running esphome-ratgdo for about a day now. I used it with homebridge-ratgdo to bridge ratgdo with HK. I want to run it like this for a few more days to see if it's a stable solution for me before I re-flash to homekit-ratgdo to try out your 3 suggestions above. Here's the observation so far:

Pings continues to be fine with 0 packet loss
HK integration is rock solid, no "No Response" experienced
Motion detection from gdo panel is not 100% reliable. occasionally (maybe 2 out of 10 times), it's still not able to detect motion, but its reliability on this area is still much better than homekit-ratgdo which couldn't detect motion almost all the time, but perhaps it was complicated by the fact that i was having connection issues with homekit-ratgdo so even though motion was detected the log and update weren't correctly transferred to my WebUI so i couldn't see them. I also saw a related issue just filed on the esphome side, so perhaps this is an issue introduced by the base ratgdo library recently.

Meanwhile, I'm also contemplating on whether my gdo panel 880LMW is defective and should I try replacing it with a new one. Are you also on this model? If so, were you able to get motion detection and light status reported to you reliably 100% of the time? I'd love to hear your advice on whether I should try swapping the panel.

davey242 commented 6 months ago

I just got my board in the mail today and installed the latest firmware with for HomeKit. I didn't have motion showing on the web ui but my lights don't come on with motion. I realized on the wall panel there is a switch to turn off motion with was off. I turned the switch on and motion is detected, lights go on, web UI show motion and HomeKit shows motion triggered.

Maybe try turn your wall panel motion switch off and back on and see if there is a difference.

You can also test the responsiveness with the obstruction sensor. When I block and unblock the sensor, it is pretty much instantly show the transition in the web ui.

pigeonman99 commented 6 months ago

Thanks @davey242, do you mean the panel's Menu settings > Light settings to turn the motion sensor on/off? If so, I've tried it multiple times and it doesn't appear to improve the ratgdo's motion sensing reliability.

The obstruction sensors actually worked 100% for me. So it's just the motion sensor on the panel not being reliable.

davey242 commented 6 months ago

I have the 881LMW. There is no LCD/menu just a physical switch to enable / disable motion detection.

Was thinking maybe if the contacts were going it would not work all the time but that still doesn't take into account that the web did not show the motion when the lights turns on by motion. So I guess ignore that LOL.

Have you checked/ verified that the wires are securely connected and clean/no signs of corrosion on the wire?

jgstroud commented 6 months ago

His lights always turn on which tells me the motion sensor is working and sending the messages. But the ratgdo isn't always seeing the messages which makes me think there is some signaling problem. I'm not sure what if anything can be done about it.

pigeonman99 commented 6 months ago

update on the original issue (motion sensor unreliable) - I took the gdo panel out and saw that one of the connected wire snapped right off from the panel. apparently the contractor who installed the panel did a really poor job wiring it. I took the time to re-wire everything, and now esphome-ratgdo is able to detect motion every single time.... turns out it was a hardware installation issue! :/ i will close the issue now.

for the secondary problem (wifi issue), i haven't had any time to re-flash back to homekit-ratgdo, but i think if no one else is reporting this problem, it's probably not worth anyone's time to look into it.

ratgdo / homekit-ratgdo

light status and motion sensor states are not reliably detected #160