Any interest in my "man in the middle" solution for Sec+ 1.0 and 0x37 wall control?

mcomer-captech commented 8 months ago

Hey, checking to see if there might be any interest in a solution I have up and running for a problem with older GDOs. Specifically, I have a Sec+ 1.0 "purple button" opener and an older wall control which came with it. It turns out that this wall control is of the 0x37 variety - that is, it does not query door status in the same way other controls do (with a 0x38 command). The folks over in the RATGDO code-base are trying to solve this by injecting a 0x38 into the command stream when they detect a 0x37 wall panel - but for whatever reason this does not work for me, something about the timing is not quite right and my opener is not responding to the 0x38 when it is sent like this (though it does respond when this is done in wall control emulation).

So - I built a work-around. I added a 12v source and 2 additional MOSFETS so that my project is now man-in-the-middle between the GDO and wall control. My board does wall control emulation for the GDO and GDO emulation for the wall control. I relay commands from the wall control to the opener (toggle light, toggle door, toggle lock) and relay status from the opener to the wall control. I have this up and running with the ESPHome code-base from the RATGDO folks, somewhat modified to add the separate interface to the wall control.

Right now I have this all soldered up on a prototype board but am thinking of taking the KiCAD schematics and PCB files from here, modifying, and ordering something more "professional".

Would be happy to contribute back if there is any interest? My use-case may be a little niche, I don't know. Not sure how many folks have older GDOs (Sec+ 1.0) and 0x37 wall controls.

TheCranston commented 8 months ago

@mcomer-captech I'm very interested. It's been in the back of my mind that the proverbial "man-in-the-middle" behaviour would solve several issues at once. Please count me in. Happy to help with code or hardware as I can.

foreverimagining commented 8 months ago

Definitely very interesting. And nowhere near as niche as you'd expect.

I don't have a Security+1.0 GDO myself, but there are a lot of people in the HA forums that do and have problems with it, as well as people with "dumb" or smart-but-not-compatible wall panels that don't necessarily want to have to replace them. I wonder if this would be the solution for them, and it could probably be made as an add-on board to original-flavor RATGDOs (though I suspect some soldering would be necessary).

Do you ever see any of the reported issues like the GDO opening by itself at random, general command unreliability, or the lock/light buttons on the wall panel not functioning? I'm pretty sure there have been more issues with Security+1.0, but those are the ones I remember off the top of my head.

mcomer-captech commented 8 months ago

Ok, thanks for the feedback, sounds like I should create some schematics and a board layout for what I have done and should share. I also have my code additions already available online, but not ready to link to it yet as it does need some cleanup. (Doesn't code always?)

Another couple of questions then @TheCranston @foreverimagining:

Right now I have a little 12v "wall wart" power supply providing the extra 12v source needed for the wall control. My thinking is that I will feed this from my opener's battery backup instead - and while I am at it, will add a little 5v regulator to power the microcontroller. That would make the entire setup power-fail safe as the opener, controller, and power to the wall control would all be coming from the opener's backup. Thoughts?
I am currently using a D1 Mini Lite in my setup - and so to add the extra serial comms for the wall control I had to sacrifice a couple of pins. There are not enough pins available for 2x serial to the GDO, 2x serial to the wall control, 3x dry contact in (open, close, light), and 2x status outputs (door, obstruction). To keep everything I'd need a device with more pins exposed. Should I just design around a device with more available GPIOs? (It didn't matter to me since I don't need the dry contact or status output features - but I guess you never know who might want them.)

mcomer-captech commented 8 months ago

Do you ever see any of the reported issues like the GDO opening by itself at random, general command unreliability, or the lock/light buttons on the wall panel not functioning? I'm pretty sure there have been more issues with Security+1.0, but those are the ones I remember off the top of my head.

I have only had my solution actually installed and running for about 2 days, but so far I have seen none of those issues. Wall control button presses work great, there have been no unexpected openings or closings, etc.

The only odd glitch that I see every once in a while - and I saw this PRIOR to adding my extra wall-control comms - is a little "bounce" in the light toggle command from ESPHome. Occasionally I will hit "toggle light" (while it is off) and the UI in Home Assistant toggles to on, toggles back to off, then toggles back to on (where it is supposed to be) - all within a second or two. I think I know what is going on there and I think it is a fundamental problem in how the opener sends status, how that gets reported back to ESPHome / HomeAssistant, etc. Essentially - there is an ongoing update of light and lock status (either the wall controller or the emulation code is sending a 0x3a to request status) and then that status has to get propagated up to HA via the API. This means that there is latency in that update, so an "old" status can come in even after the command to toggle has been sent. It's a UI glitch only and it sorts itself out quickly, so...

foreverimagining commented 8 months ago

Another couple of questions then

Having the option to run it off a 12v wall supply or the battery backup would probably be best. There seem to be a fair few people who have come to this repository that have battery backups and want to make use of them, but I've always thought not having one was more the norm for most GDOs, so having the choice would benefit everyone. You can probably check out the other schematics/boards in this repository to see how they've done it if you want to compare. As for the microcontroller, correct me if I'm wrong, but they actually run on 3.3V and the boards have a voltage regulator to step the 5V from USB down to the required 3.3V. The RATGDO boards also have a 3.3V IN pin broken out for an alternative to USB power (though I don't know how many people take that option since everyone probably has a few USB cables/bricks lying around). It would make more sense, then, to use 3.3V connected to the 3.3V pin, unless someone has a specific need for 5V like the ultrasonic sensor I'm adding to mine.
When it comes to Security+ GDOs, I expect most people keep to the TX, RX, Blk/Obst side of the board and ignore the dry contact/status output side, which is meant to create greater compatibility with other types of GDOs but otherwise is largely unnecessary. The exception to that might be, for instance, gate openers, but they have a lot of problems to contend with getting RATGDO to work in general. I would say keep the scope smaller to start with and drop the dry contact side. Dry contact support could be added later as a separate, dedicated board or a larger, combined board if there are people who need both. To support people with Security+1.0 and original RATGDOs, it wouldn't be too far outside of that same, smaller scope to make another board with just the additional middleman components. Since there are unnecessary components connected between the pins and the breakouts, though, I think the options will be either to create a three board stack, RATGDO/Middleman/D1 Mini, or to solder wires directly to the pins on the ESP8266 module to have access to them. Unfortunately, because of regular design changes that we in the 3D printing community abhor, the first option will only work with v2.5, and the second option would be necessary for v2.5x's.

a little "bounce"

If this is what I think it is, I've seen it happen on various switches/lights in Home Assistant and would also consider it to be a mild buffering-type annoyance rather than a possible issue with the device. Which means so far so good. Long term reliability can really only be determined over time, but sounds very promising so far.

mcomer-captech commented 8 months ago

Quick update on this: I wanted to let my solution "bake" a little bit before I moved forward with anything. I can happily report that in a couple of weeks of testing now, I have had absolutely zero issues. No unexplained door openings. No odd behavior from the wall control buttons, controls in Home Assistant, controls in Apple Home (I expose the door and related items via the Homebridge integration in HA). The motion sensor on the wall control turns on the light and that is immediately reflected in HA. Etc. I am very happy with this.

I do have some time away scheduled for later this week so I will be away from home for a couple of weeks. Once I am back my plan is to:

Produce a schematic and pcb design in KiCad, have some prototype boards made, etc.
Update my software solution to include the latest from the Sec+1.0 work on the RATGDO side (my current code is based on what they had a few days prior to their merge to master so I need to make a few updates.)

In the meantime - if anyone is super-interested in this, absolutely needs a solution for the Sec+1.0 opener and 0x37 wall control, etc - ping me and I can hopefully share more details prior to getting all of the polishing-up done.

jrsphoto commented 8 months ago

I think I also have an older purple button opener but not sure what opener I have. My opener looks like this. Just wondering your your version is what I'm looking for.

ur7x commented 7 months ago

Count me in too... Post your schematic when you are ready... I have a couple of Security 1+ Red button openers with 889LM wall panels and I would be happy to build one and test it...

Presently I'm running MQTT on the original schematic here. With one week of use my experience is: 1) The RATGDO software will report a MQTT "Syncing" status a couple of times a day. "Covers" in HA do not understand what syncing is and this seems to result in phantom door close commands and light toggle commands. "Syncing" status events last about 30 seconds and during this time the doors (and lights) are unresponsive. 2) The light control in RATGDO is a toggle (now confirmed) so if the light and the light's status get out of sync an "ON" command from HA will turn off lights that are already on.
3) So far only one phantom garage door open... But that is one too many.

Happy with the local solution and happy with local light control (even if it is flaky) It is ironic that MyQ forced this solution upon us and the resulting "hack" already has more functionality than the cloud system.

windsurferwill commented 7 months ago

I am interested. I have a security 1 GDO (purple) with a 398. I can’t switch to a 889 control panel because it’s too different and harder to use functions like lock out. But the door status with the 398 doesn’t work. Man in the middle sounds like a great solution. What can I do to help? Oh, I have the latest ratgdo v2.52 running in MQTT mode with HA. Works flawlessly except the cover doesn’t report door status.

mitchjs commented 7 months ago

im jumping in here, i did work (just for fun) on the HOMEKIT version of ratgdo... i added 1.0 support while doing this... i learned alot... and i totally rewrote how SEC+1.0 works and know why there are issues... i believe my code for SEC+1.0 is the most reliable one to date

its about knowing when you can "TALK" on the buss, and how to avoid bit collisions.

i only have a 889lm, and worked with that... not sure what a 398 is if you have an iphone/ipad and want to try my firmware out (why its still sitting as a PULL REQUEST, i dont know)

the MQTT fw could get cleaned up, not a fan cause i dont feel like running mqtt broker (but i have one running for testing this stuff) the esphome code is well written... i worked on the HOMEKIT version for a friend... as i really didnt need it for my situation but its rocking...

i think i got it nailed, i have had zero false openings, and 100% accurate status the only time i see collision is with light, and its rare... and my code does its best to trap the garbled byte out... so within fw 100ms status gets correct on next poll by 889lm

@mcomer-captech i think the man-in-the middle could solve 100% of the knowing when to talk issues right now i use timing, when the byte is transmitted from the lm889 and the GDO byte is sent back and i then i know i have a "window" to send my byte out (if i have one to send) i also have code to detect collision garbage and ignore you can look at my source in comms.cpp and i be happy to work with anyone is your schematic for ESP32 or the esp8266 i think the code would work in either me fan of esp32 variants

i do electronics too, and was thinking of producing a ESP32 based one myself but since i haven't had real need for this personally... i guess been lazy (my 3rd party control still works on HE, dont know why but it does)

Mitch

windsurferwill commented 7 months ago

I'm currently using MQTT in Home Assistant with RATGDO, but I could use ESPHome if it worked with my Security1+ GDO. From reading the latest posts, apparently the reason my GDO is not providing open/close status is my 398LM wall control unit uses the 0x37 message. I cannot switch to the newer 889LM wall controller which uses 0x38 messages. I have confirmed that the RATGDO properly provides open/close status when I disconnect my 398LM to put RATDO into emulation mode which tells me the garage door opener itself is capable of open/close status.

@mcomer-captech , others: Could the man-in-the-middle approach solve this? Does anyone know if the 0x37 wall control requires a response to it's message? Could the man-in-the-middle intercept the 0x37 and respond with 0x38? Steve

mitchjs commented 7 months ago

Im sure i could fix any of the FW to work with 398LM, but other than this panel being old... why cant you switch to newer 889LM? if i had access to 398LM (and it worked with my gdo... which prob does)

0x38 is query. 0x37 could also be a query. as you see I have 2 unknowns (i just never seen the messages with my 889LM so that's why i call em unknown)

i was thinking of just turning the Homekit firmware, in to a direct http control... so one can make a simple Hubitat driver (as that is what i got) MQTT not a fan of having another device as a broker ESPHOME version of course is good for that ecosystem

 DoorButtonPress = 0x30,
 DoorButtonRelease = 0x31,
 LightButtonPress = 0x32,
 LightButtonRelease = 0x33,
 LockButtonPress = 0x34,
 LockButtonRelease = 0x35,

 Unkown_0x36  = 0x36,
 Unknown_0x37 = 0x37,

DoorStatus = 0x38,
ObstructionStatus = 0x39,   // this is not proven
LightLockStatus = 0x3A,

foreverimagining commented 7 months ago

why cant you switch to newer 889LM?

889LM's aren't exactly cheap to simply buy as a replacement, and despite being regarded as compatible and even recommended, they are not always problem-free. While your technique may be able to fix the code to solve some or all of those problems, some people also have wall panels with more capabilities, such as a thermometer, and they don't want to have to downgrade to an 889LM, even if the cost were easily justifiable. This man-in-the-middle option will hopefully give people options that will allow them to choose the best course of action for themselves. Though, if you are able to fix the code for 0x37 wall panels, I'm sure there are a lot of people who would be appreciative.

mitchjs commented 7 months ago

looking on ebay, 398LM isnt cheap!! if i can get one, someone loan me one... i can work on it

windsurferwill commented 7 months ago

I really appreciate you jumping in. I wish I had one to loan you but I can't take mine out of service. Would it help if I could run experiments, dumps, etc.?

mitchjs commented 7 months ago

@windsurferwill, you using the ESPHOME version of ratgdo? got an iPhone? could flash what I did for homekit and you could test on iPhone (dont need anything more than iPhone as homekit app is on it) and can add some code to log more (and i got logging in there) since i have no idea what 0x36/0x37 messages are, since for one i never seen one on my system 889LM+GDO

windsurferwill commented 6 months ago

@mitchjs: No sorry, I do not have an iPhone. I am using the MQTT version with RATGDO v2.52 hardware because from the comments on Paul's RATGDO GitHub page ESPHOME is still a work-in-progress for Security 1+ openers.