No MQTT and no web UI after initial configuration

I flashed the latest MQTT+wifi firmware (v2.54) on my new device (v2.52i), and after initial device configuration (wifi and then MQTT params + password), I am not seeing MQTT successfully connecting, and the web UI becomes unresponsive.

The device is connected to my local network, as pings to the device's IP succeed. The webserver is even listening, as a port scan shows port 80 as open. However, any requests to the web UI (even just from curl) after the initial configuration of the device (setting a password and other params).

I'm attempting to interface the device to HA running the Mosquitto broker add-on. I feel like I'm missing something fundamental here. The hardware I'll be using this with is a Chamberlain Sec+ 1 unit, so I can't use the ESPHome FW (yet).

The 3 MQTT connection attempts look just like every subsequent attempt (repeating every 5 seconds or so).

Note: passwords have been snipped/replaced. The qpass value is representative of the type of password I use for this network (3 English words separated by hyphens).

deviceName: ratgdo
microcontrollerIP: 192.168.1.185
qsid: JaredIOT
qpass: correct-horse-battery
OTApass: <snip>
mqttIP: 192.168.0.9
mqttPort: 1883
mqttuser: 
mqttpass: 
mqttTopicPrefix: /home/garage/
haDiscoveryPrefix: homeassistant
controlProtocol: secplus1
disableOTA: 
additionalParam: 
Saving setup.json
{
  "deviceName": "ratgdo",
  "microcontrollerIP": "192.168.1.185",
  "qsid": "JaredIOT",
  "qpass": "correct-horse-battery",
  "OTApass": "<snip>",
  "mqttIP": "192.168.0.9",
  "mqttPort": "1883",
  "mqttuser": "",
  "mqttpass": "",
  "mqttTopicPrefix": "/home/garage/",
  "haDiscoveryPrefix": "homeassistant",
  "disableOTA": false,
  "controlProtocol": "secplus1",
  "additionalParam": ""
}[setup.json] written correctly

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3460, room 16 
tail 4
chksum 0xcc
load 0x3fff20b8, len 40, room 4 
tail 4
chksum 0xc9
csum 0xc9
v0006a2b0
~ld

Mounting LittleFS...
[setup.json]
JSON parsed
Storage OK, restoring WiFi and MQTT config.
Connecting to: 
JaredIOT...
Using static IP address
Set WiFi output power to: 20.50
.......
WIFI CONNECTED
IP Address: 192.168.1.185
nb of attempts: 7
IMPROVhttp://192.168.1.185

WiFi connected
Local IP: 192.168.1.185
SoftAP IP: (IP unset)
Server started
Webserver ready. Open http://ratgdo.local/ in your browser
Launching webserver for improv
Starting ArduinoOTA service
doorCommandTopic: /home/garage/ratgdo/command/door
lightCommandTopic: /home/garage/ratgdo/command/light
lockCommandTopic: /home/garage/ratgdo/command/lock
Using security+ 1.0
Setup Complete
 _____ _____ _____ _____ ____  _____ 
| __  |  _  |_   _|   __|    \|     |
|    -|     | | | |  |  |  |  |  |  |
|__|__|__|__| |_| |_____|____/|_____|
version 2.54
IMPROVhttp://192.168.1.185
### MQTT DISCONNECTED ###
Connecting to
MQTT Broker...
MQTT Last Will Params: 
willTopic: /home/garage/ratgdo/status/availability
willPayload: offline
qos: 1
retain: 0
clean session: 1
MQTT attempts=
0
### MQTT DISCONNECTED ###
Connecting to
MQTT Broker...
MQTT Last Will Params: 
willTopic: /home/garage/ratgdo/status/availability
willPayload: offline
qos: 1
retain: 0
clean session: 1
MQTT attempts=
1
### MQTT DISCONNECTED ###
Connecting to
MQTT Broker...
MQTT Last Will Params: 
willTopic: /home/garage/ratgdo/status/availability
willPayload: offline
qos: 1
retain: 0
clean session: 1
MQTT attempts=
2
### MQTT DISCONNECTED ###

I have found ratgdo to be intolerant of connecting to MQTT brokers that are not in the same subnet or connecting to external brokers. It looks like you may have ratgdo on 192.168.1.xx and your broker on 192.168.0.xx. This is a big problem for my use case so I am using a bridged MQTT broker as a work-around and hoping that Paul will make some changes to allow this in the future. This should be the users choice, with respect to security considerations.

Also, there are not any helpful error messages for this condition or even for a typo in your wifi password. The only remedy seems to be reflashing the ratgdo.

Upon closer inspection of the code in https://github.com/PaulWieland/arduinoImprovBootstrapper, I believe this is occurring because the web UI isn't initialized until after MQTT successfully connects. To me this is a bad design, as mis-configuring MQTT on the initial setup screen prevents you from fixing that error moving forward without reconnecting a USB cable to re-flash.

@ellweber My network has a /23 netmask, so 192.168.0.x and 192.168.1.x addresses are in the same subnet. Even so, netmask issues did occur to me since /24 is so pervasive and I've filed bugs for other projects that hard-code it in places. I attempted to assign a static address (configured on the DHCP server for the ratgdo's MAC address) from the 0.x portion of the subnet, but that didn't seem to make a difference, so I reverted the changes before submitting logs here.

Thanks for digging into the code to explain the crash on typo/configuration anomaly problem. That shouldn’t be a difficult usability improvement to implement.

Leaving the responsibility for users to configure their network as they wish also seems easily done. Other ESP8266 applications such as Tasmota already do so.

Great find. I would actually make this ask a 2 part item:

I would suggest that this is very much a bug. The UI should never be uncreachable because an external resource, out of our control is not available. In this case, think of it this way; If one sets up an MQTT server CORRECTLY, but the MQTT server is offline/removed, the next restart of the ratgdo device would result in an inaccessable web UI to be able to resolve the issue.
In addition to the above required fix, I would strongly recommend not forcing people to enter an IP address, but rather either an IP address or an FQDN. In my case, I have setup hostname resolution on my router for "mqtt.domainname.com" --> This is an internal host, but it allows me to not have necessarily assign static IPs to devices, and or make it easier if an IP has to change. Hard coding IPs across many devices would make it a pain to make adjustments to the topology / move services to different machines, etc.

I will make a call-out to #2 , which if resolved, would make it a lot easier to perform builds on this entire codebase, and then submit pull requests with the expectation that they'll be accepted and the time not wasted.

Tonight I was able to work through a few issues and get things functional.

My initial and subsequent configuration attempts were made without authentication in mind, as I have a fairly segmented and isolated network and didn't care about auth. Apparently the Mosquitto broker as shipped as an HA addon intentionally removes the ability to configure anonymous auth. Once I created an HA user and used that during the initial config, things got closer to working.

Additionally, I think there may be an issue with subnets larger than /24. My network is a /23 and if the DHCP server hands ratgdo an address in the upper half of the 192.168.0.0/23 subnet, while my HA/MQTT broker is in the lower half, things weren't working. After I fixed the above auth problem AND assigned a fixed address on the DHCP server end in the lower half of the subnet, things started working immediately after the initial config.

I will leave this open and let the project maintainers decide if they should be split up into 2 separate issues.

Not sure if this is the same - ignore if it is - but for my initial setup with 2.56, it assigned me a 169.x.x.x address after installation whereas my devices fall in the 192.168.1.x space.

I was still able to connect to it via the web (although it took awhile) and I statically assigned it an IP in the 192.168.1.x space. Then it connected faster in the future.

169.X.Y.Z means that it either didn’t connect to your router, or it was not given an IP via DHCP.

I received my 3 ratgdo boards yesterday (2.5i) and have exactly the same issue as the OP on the one board that is configured for MQTT due to older Security 1.0 GDO. The other two are on ESPHome as they will be installed on Security 2.0 GDO's. I've reflashed the MQTT board a number of times, I see initial activity and messages on MQTT Explored, but cannot access the board directly via IP address or via "Visit" link on MQTT Device page within HAOS. I am on a Unifi network segmented via VLAN's, HAOS VM on Proxmox is at 192.168.1.68 and the ratgdo's are on a IoT VLAN at 192.168.20.xx. I can access the ESPHome ratgdo's via visit command no problem. This is mu first time using MQTT and ESPHome, but like all of the comments I've read for months have said, the ESPHome approach seems work great.

I'm open to suggestion on how to fix this or create a workaround, but also wondering when the ESPHome version will support Security 1.0 GDO's?

I'm going to install the two ESPHome boards on the GDO's and test/verify full function. Maybe I'll try to install the MQTT board too, but don't want to create a lot of work for myself is there is a fix or workaround available.

I received my 3 ratgdo boards yesterday (2.5i) and have exactly the same issue as the OP on the one board that is configured for MQTT due to older Security 1.0 GDO. The other two are on ESPHome as they will be installed on Security 2.0 GDO's. I've reflashed the MQTT board a number of times, I see initial activity and messages on MQTT Explored, but cannot access the board directly via IP address or via "Visit" link on MQTT Device page within HAOS. I am on a Unifi network segmented via VLAN's, HAOS VM on Proxmox is at 192.168.1.68 and the ratgdo's are on a IoT VLAN at 192.168.20.xx. I can access the ESPHome ratgdo's via visit command no problem. This is mu first time using MQTT and ESPHome, but like all of the comments I've read for months have said, the ESPHome approach seems work great.

I'm open to suggestion on how to fix this or create a workaround, but also wondering when the ESPHome version will support Security 1.0 GDO's?

I'm going to install the two ESPHome boards on the GDO's and test/verify full function. Maybe I'll try to install the MQTT board too, but don't want to create a lot of work for myself is there is a fix or workaround available.

I have a very similar setup and same problem.

2. In addition to the above required fix, I would strongly recommend not forcing people to enter an IP address, but rather either an IP address or an FQDN.  In my case, I have setup hostname resolution on my router for "mqtt.domainname.com" --> This is an internal host, but it allows me to not have necessarily assign static IPs to devices, and or make it easier if an IP has to change.

I think I'm experiencing an issue related to 2, here. I have currently configured my broker with Let's Encrypt SSL on 8883 only. The mqtt integration uses FQDN.

I've tried ssl://IP and just IP for ratgdo broker config. ssl://IP doesn't seem to connect at all. Using IP only shows SSL errors, which is expected, but it shows it's reaching the broker.

FQDN might solve my issue (certificate is for FQDN) or SSL/8883 MQTT isn't supported.

I received my 3 ratgdo boards yesterday (2.5i) and have exactly the same issue as the OP on the one board that is configured for MQTT due to older Security 1.0 GDO. The other two are on ESPHome as they will be installed on Security 2.0 GDO's. I've reflashed the MQTT board a number of times, I see initial activity and messages on MQTT Explored, but cannot access the board directly via IP address or via "Visit" link on MQTT Device page within HAOS. I am on a Unifi network segmented via VLAN's, HAOS VM on Proxmox is at 192.168.1.68 and the ratgdo's are on a IoT VLAN at 192.168.20.xx. I can access the ESPHome ratgdo's via visit command no problem. This is mu first time using MQTT and ESPHome, but like all of the comments I've read for months have said, the ESPHome approach seems work great.

I'm open to suggestion on how to fix this or create a workaround, but also wondering when the ESPHome version will support Security 1.0 GDO's?

I'm going to install the two ESPHome boards on the GDO's and test/verify full function. Maybe I'll try to install the MQTT board too, but don't want to create a lot of work for myself is there is a fix or workaround available.

After reading some comments made to the Reddit thread on this, my issue seems to be all network related. I have 3 networks defined on my Unifi UDM-Pro: Secure, IoT, Guest..all separated by VLANS and wifi. Proxmox/HAOS/MQTT Broker are on Secure and my ratgdo's are on the IoT wifi. ESPHome flashed units work fine, the MQTT doesn't connect back to the MQTT Broker. I just reflashed the ratgdo to latest firmware (2.57), reconfigured and the only thing I changed was the wifi network. Rebooted and it was auto discovered and available in HAOS/MQTT Broker as well as via MQTT Explorer and via both direct IP address connection & "visit" option from the device page in MQTT Broker.

I will get this installed on the GDO tonight or this weekend and let everyone know. I am also going to research, test and deploy a firewall rule that allows this to connect to the IoT network, but communicate securely back to my secure network. I still plan to convert to ESPhome once its available, great feature set, for Security 1.0.

I received my 3 ratgdo boards yesterday (2.5i) and have exactly the same issue as the OP on the one board that is configured for MQTT due to older Security 1.0 GDO. The other two are on ESPHome as they will be installed on Security 2.0 GDO's. I've reflashed the MQTT board a number of times, I see initial activity and messages on MQTT Explored, but cannot access the board directly via IP address or via "Visit" link on MQTT Device page within HAOS. I am on a Unifi network segmented via VLAN's, HAOS VM on Proxmox is at 192.168.1.68 and the ratgdo's are on a IoT VLAN at 192.168.20.xx. I can access the ESPHome ratgdo's via visit command no problem. This is mu first time using MQTT and ESPHome, but like all of the comments I've read for months have said, the ESPHome approach seems work great. I'm open to suggestion on how to fix this or create a workaround, but also wondering when the ESPHome version will support Security 1.0 GDO's? I'm going to install the two ESPHome boards on the GDO's and test/verify full function. Maybe I'll try to install the MQTT board too, but don't want to create a lot of work for myself is there is a fix or workaround available.

After reading some comments made to the Reddit thread on this, my issue seems to be all network related. I have 3 networks defined on my Unifi UDM-Pro: Secure, IoT, Guest..all separated by VLANS and wifi. Proxmox/HAOS/MQTT Broker are on Secure and my ratgdo's are on the IoT wifi. ESPHome flashed units work fine, the MQTT doesn't connect back to the MQTT Broker. I just reflashed the ratgdo to latest firmware (2.57), reconfigured and the only thing I changed was the wifi network. Rebooted and it was auto discovered and available in HAOS/MQTT Broker as well as via MQTT Explorer and via both direct IP address connection & "visit" option from the device page in MQTT Broker.

I will get this installed on the GDO tonight or this weekend and let everyone know. I am also going to research, test and deploy a firewall rule that allows this to connect to the IoT network, but communicate securely back to my secure network. I still plan to convert to ESPhome once its available, great feature set, for Security 1.0.

Initial attempt at a firewall rule didn't work. Something going on where the board is not picking up the proper subnet even with a static IP. fell back to putting it on my secure lan and it connects fine. I'm going to install that tomorrow and verify the GDO functions....and then see where it goes from there. Hopefully ESPHome is out soon for 1.0 and this problem goes away. Open to suggestions.

NukeThemTillTheyGlow:

This appears to be a persistent problem. ratgdo (for MQTT) is totally intolerant of an address for the MQTT broker that is outside the subnet that ratgdo is operating on. If ratgdo has an "IP address" of 192.168.0.xxx (either configured or assigned) then it will crash if you configure it with "MQTT server IP" of 192.168.1.yyy, or anything other than 192.168.0.yyy.

The Host ID and subnets must agree for ratgdo and its MQTT broker. This limitation restricts the way your network can be configured for ratgdo to operate.

Everything described in this thread are issues related to the way networking is implemented in ArduinoBootstrapper. I am planning on eliminating it from ratgdo, but it's going to take some time to do the rewrite.

In the meantime, I'm going to test adding MQTT support via a custom yaml file with the ESPHome port. If it works this will offer a solution for Security + 2.0 users who have more complex than average networks.

I just did a quick test and its quite easy to add the MQTT integration to the ESPHome firmware.

Steps I took

install esphome if you don't already have it installed
plug the ratgdo board in via USB
install the standard ESPhome-ratgdo firmware and connect it to your wifi. This step can be skipped if you want to provide wifi credentials in your yaml file instead of through the web gui.
clone/download the ESPHome source code to your computer
modify the v25iboard.yaml to add MQTT (see below for code snippet). Save yaml file.
from the command line: esphome run v25iboard.yaml to compile and install the esphome ratgdo firmware using your new settings

yaml for adding mqtt integration to esphome ratgdo firmware:

mqtt:
  id: mqtt_client
  broker: YOUR_MQTT_BROKER_ADDRESS
  port: 1883 #change if your mqtt broker is on a different port
  username: YOUR_MQTT_USERNAME
  password: YOUR_MQTT_PASSWORD
  discovery: false
  client_id: esphomeratgdo # must be unique or the broker will kick the client off
  topic_prefix: YOUR_PREFIX
  birth_message:
    topic: availability
    payload: online
  will_message:
    topic: availability
    payload: offline

ESPHome will then output all of the topics that are published or subscribed to:

[09:43:43][C][mqtt:133]: MQTT:
[09:43:43][C][mqtt:134]:   Server Address: 10.0.1.55:1883 (10.0.1.55)
[09:43:43][C][mqtt:136]:   Username: 'ratgdo'
[09:43:43][C][mqtt:137]:   Client ID: 'esphomeratgdo'
[09:43:43][C][mqtt:142]:   Topic Prefix: 'test_ratgdo'
[09:43:43][C][mqtt:144]:   Log Topic: 'test_ratgdo/debug'
[09:43:43][C][mqtt:147]:   Availability: 'availability'
[09:43:43][C][api:139]: API Server:
[09:43:43][C][api:140]:   Address: ratgdov25i.local:6053
[09:43:43][C][api:144]:   Using noise encryption: NO
[09:43:43][C][improv_serial:032]: Improv Serial:
[09:43:43][C][mqtt.sensor:028]: MQTT Sensor 'Openings':
[09:43:43][C][mqtt.sensor:032]:   State Topic: 'test_ratgdo/sensor/openings/state'
[09:43:44][C][mqtt.lock:034]: MQTT Lock 'Lock remotes': 
[09:43:44][C][mqtt.lock:035]:   State Topic: 'test_ratgdo/lock/lock_remotes/state'
[09:43:44][C][mqtt.lock:035]:   Command Topic: 'test_ratgdo/lock/lock_remotes/command'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Motion':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/motion/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Obstruction':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/obstruction/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Button':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/button/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Motor':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/motor/state'
[09:43:44][C][mqtt.number:033]: MQTT Number 'Rolling code counter':
[09:43:44][C][mqtt.number:034]:   State Topic: 'test_ratgdo/number/rolling_code_counter/state'
[09:43:44][C][mqtt.number:033]: MQTT Number 'Opening duration':
[09:43:44][C][mqtt.number:034]:   State Topic: 'test_ratgdo/number/opening_duration/state'
[09:43:44][C][mqtt.number:033]: MQTT Number 'Closing duration':
[09:43:44][C][mqtt.number:034]:   State Topic: 'test_ratgdo/number/closing_duration/state'
[09:43:44][C][mqtt.number:033]: MQTT Number 'Client ID':
[09:43:44][C][mqtt.number:034]:   State Topic: 'test_ratgdo/number/client_id/state'
[09:43:44][C][mqtt.cover:052]: MQTT cover 'Door':
[09:43:44][C][mqtt.cover:055]:   State Topic: 'test_ratgdo/cover/door/state'
[09:43:44][C][mqtt.cover:055]:   Command Topic: 'test_ratgdo/cover/door/command'
[09:43:44][C][mqtt.cover:057]:   Position State Topic: 'test_ratgdo/cover/door/position/state'
[09:43:44][C][mqtt.cover:058]:   Position Command Topic: 'test_ratgdo/cover/door/position/command'
[09:43:44][C][mqtt.light:078]: MQTT Light 'Light':
[09:43:44][C][mqtt.light:079]:   State Topic: 'test_ratgdo/light/light/state'
[09:43:44][C][mqtt.light:079]:   Command Topic: 'test_ratgdo/light/light/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Restart': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/restart/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/restart/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Safe mode boot': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/safe_mode_boot/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/safe_mode_boot/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Query status': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/query_status/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/query_status/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Query openings': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/query_openings/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/query_openings/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Sync': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/sync/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/sync/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Toggle door': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/toggle_door/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/toggle_door/command'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Dry contact open':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/dry_contact_open/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Dry contact close':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/dry_contact_close/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Dry contact light':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/dry_contact_light/state'

Topic YOUR_PREFIX/cover/door/state will contain the door status. You can control the door by sending payload open|close to topic YOUR_PREFIX/cover/door/command

I just did a quick test and its quite easy to add the MQTT integration to the ESPHome firmware.

Steps I took

install esphome if you don't already have it installed
plug the ratgdo board in via USB
install the standard ESPhome-ratgdo firmware and connect it to your wifi. This step can be skipped if you want to provide wifi credentials in your yaml file instead of through the web gui.
clone/download the ESPHome source code to your computer
modify the v25iboard.yaml to add MQTT (see below for code snippet). Save yaml file.
from the command line: esphome run v25iboard.yaml to compile and install the esphome ratgdo firmware using your new settings

yaml for adding mqtt integration to esphome ratgdo firmware:

mqtt:
  id: mqtt_client
  broker: YOUR_MQTT_BROKER_ADDRESS
  port: 1883 #change if your mqtt broker is on a different port
  username: YOUR_MQTT_USERNAME
  password: YOUR_MQTT_PASSWORD
  discovery: false
  client_id: esphomeratgdo # must be unique or the broker will kick the client off
  topic_prefix: YOUR_PREFIX
  birth_message:
    topic: availability
    payload: online
  will_message:
    topic: availability
    payload: offline

ESPHome will then output all of the topics that are published or subscribed to:

[09:43:43][C][mqtt:133]: MQTT:
[09:43:43][C][mqtt:134]:   Server Address: 10.0.1.55:1883 (10.0.1.55)
[09:43:43][C][mqtt:136]:   Username: 'ratgdo'
[09:43:43][C][mqtt:137]:   Client ID: 'esphomeratgdo'
[09:43:43][C][mqtt:142]:   Topic Prefix: 'test_ratgdo'
[09:43:43][C][mqtt:144]:   Log Topic: 'test_ratgdo/debug'
[09:43:43][C][mqtt:147]:   Availability: 'availability'
[09:43:43][C][api:139]: API Server:
[09:43:43][C][api:140]:   Address: ratgdov25i.local:6053
[09:43:43][C][api:144]:   Using noise encryption: NO
[09:43:43][C][improv_serial:032]: Improv Serial:
[09:43:43][C][mqtt.sensor:028]: MQTT Sensor 'Openings':
[09:43:43][C][mqtt.sensor:032]:   State Topic: 'test_ratgdo/sensor/openings/state'
[09:43:44][C][mqtt.lock:034]: MQTT Lock 'Lock remotes': 
[09:43:44][C][mqtt.lock:035]:   State Topic: 'test_ratgdo/lock/lock_remotes/state'
[09:43:44][C][mqtt.lock:035]:   Command Topic: 'test_ratgdo/lock/lock_remotes/command'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Motion':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/motion/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Obstruction':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/obstruction/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Button':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/button/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Motor':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/motor/state'
[09:43:44][C][mqtt.number:033]: MQTT Number 'Rolling code counter':
[09:43:44][C][mqtt.number:034]:   State Topic: 'test_ratgdo/number/rolling_code_counter/state'
[09:43:44][C][mqtt.number:033]: MQTT Number 'Opening duration':
[09:43:44][C][mqtt.number:034]:   State Topic: 'test_ratgdo/number/opening_duration/state'
[09:43:44][C][mqtt.number:033]: MQTT Number 'Closing duration':
[09:43:44][C][mqtt.number:034]:   State Topic: 'test_ratgdo/number/closing_duration/state'
[09:43:44][C][mqtt.number:033]: MQTT Number 'Client ID':
[09:43:44][C][mqtt.number:034]:   State Topic: 'test_ratgdo/number/client_id/state'
[09:43:44][C][mqtt.cover:052]: MQTT cover 'Door':
[09:43:44][C][mqtt.cover:055]:   State Topic: 'test_ratgdo/cover/door/state'
[09:43:44][C][mqtt.cover:055]:   Command Topic: 'test_ratgdo/cover/door/command'
[09:43:44][C][mqtt.cover:057]:   Position State Topic: 'test_ratgdo/cover/door/position/state'
[09:43:44][C][mqtt.cover:058]:   Position Command Topic: 'test_ratgdo/cover/door/position/command'
[09:43:44][C][mqtt.light:078]: MQTT Light 'Light':
[09:43:44][C][mqtt.light:079]:   State Topic: 'test_ratgdo/light/light/state'
[09:43:44][C][mqtt.light:079]:   Command Topic: 'test_ratgdo/light/light/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Restart': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/restart/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/restart/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Safe mode boot': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/safe_mode_boot/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/safe_mode_boot/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Query status': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/query_status/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/query_status/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Query openings': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/query_openings/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/query_openings/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Sync': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/sync/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/sync/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Toggle door': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/toggle_door/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/toggle_door/command'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Dry contact open':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/dry_contact_open/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Dry contact close':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/dry_contact_close/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Dry contact light':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/dry_contact_light/state'

Topic YOUR_PREFIX/cover/door/state will contain the door status. You can control the door by sending payload open|close to topic YOUR_PREFIX/cover/door/command

Interesting Paul. Still a newbie to MQTT/ESPHome, but will this combo allow for communication across subnets via ESPHome like my other two ratgdo's or will I just face the same issues. I'm ready to install the ratdgo right now with it connected to my secure network where the Proxmox/HAOS VM/MQTT Broker is. Not ideal, but hopefully gives me basic GDO functionality while I wait for ESPHome support for Security 1.0

Related question, in step 1 above you mention installing ESPHome. Is that on my Windows PC that I'm also using via Chrome to flash the board with ESPHome firmware? or the ESPHome console that I added to HAOS?

Related question, in step 1 above you mention installing ESPHome. Is that on my Windows PC that I'm also using via Chrome to flash the board with ESPHome firmware? or the ESPHome console that I added to HAOS?

His steps are for stand-alone ESPHome. If you have the ESPHome Addon installed in HAOS, you can adopt it in there instead.

NukeThemTillTheyGlow:

This appears to be a persistent problem. ratgdo (for MQTT) is totally intolerant of an address for the MQTT broker that is outside the subnet that ratgdo is operating on. If ratgdo has an "IP address" of 192.168.0.xxx (either configured or assigned) then it will crash if you configure it with "MQTT server IP" of 192.168.1.yyy, or anything other than 192.168.0.yyy.

The Host ID and subnets must agree for ratgdo and its MQTT broker. This limitation restricts the way your network can be configured for ratgdo to operate.

this isnt correct. i have 4 ratgdo's on an IoT VLAN and they communicate with my broker in the trusted VLAN

for me, this was a firewall issue (i am using a unifi gateway)

I will add that if the ratgdo cant connect to the MQTT server, the whole device was basically locked for me as well. I was not able to use an FQDN, it only worked with IP. 3 of my ratgdo devices are security 1.0 (requiring MQTT) and one device is security 2.0 (over esphome). But all 4 devices are on the IoT vlan.

NukeThemTillTheyGlow: This appears to be a persistent problem. ratgdo (for MQTT) is totally intolerant of an address for the MQTT broker that is outside the subnet that ratgdo is operating on. If ratgdo has an "IP address" of 192.168.0.xxx (either configured or assigned) then it will crash if you configure it with "MQTT server IP" of 192.168.1.yyy, or anything other than 192.168.0.yyy. The Host ID and subnets must agree for ratgdo and its MQTT broker. This limitation restricts the way your network can be configured for ratgdo to operate.

this isnt correct. i have 4 ratgdo's on an IoT VLAN and they communicate with my broker in the trusted VLAN

for me, this was a firewall issue (i am using a unifi gateway)

I will add that if the ratgdo cant connect to the MQTT server, the whole device was basically locked for me as well. I was not able to use an FQDN, it only worked with IP. 3 of my ratgdo devices are security 1.0 (requiring MQTT) and one device is security 2.0 (over esphome). But all 4 devices are on the IoT vlan.

Outstanding post! I followed your firewall rule, created a new one to replace my old (very similar, I was also trying to accept only the IP and MAC address of the ratgdo..mmight try to add that later), and reconnected the ratgdo to the IoT network and it works! Glad I waited to install, will get to that in the next few days. Thank you!

I think that I may have mistakenly added my rule attempt below the "drop all inter-VLAN traffic" rule. Always good to take your time and remember these rules get evaluated top to bottom.

ETA: I updated the firewall rule to add the mac address of the ratgdo so that its device specific, not a general open port for any device. Tests fine. Installing board on GDO in the next hour or less.

ETA 2/Final: Install complete and functional! Simple 3 wire install, no bypass, USB power brick and cable. Didn't turn off the power, kept it very simple. Worked immediately. All of the basic MQTT supported functionality is working. Now off of config my Android Auto UI and get this working in my car. The journey is almost over! lol

But then I want to start some automation for departures and arrivals, tie in some lighting, security system arm/disarm, etc. It never ends! Thanks for everyone's help and can't wait for ESPHome support for Security 1.0!!!

I just did a quick test and its quite easy to add the MQTT integration to the ESPHome firmware.

Steps I took

install esphome if you don't already have it installed
plug the ratgdo board in via USB
install the standard ESPhome-ratgdo firmware and connect it to your wifi. This step can be skipped if you want to provide wifi credentials in your yaml file instead of through the web gui.
clone/download the ESPHome source code to your computer
modify the v25iboard.yaml to add MQTT (see below for code snippet). Save yaml file.
from the command line: esphome run v25iboard.yaml to compile and install the esphome ratgdo firmware using your new settings

yaml for adding mqtt integration to esphome ratgdo firmware:

mqtt:
  id: mqtt_client
  broker: YOUR_MQTT_BROKER_ADDRESS
  port: 1883 #change if your mqtt broker is on a different port
  username: YOUR_MQTT_USERNAME
  password: YOUR_MQTT_PASSWORD
  discovery: false
  client_id: esphomeratgdo # must be unique or the broker will kick the client off
  topic_prefix: YOUR_PREFIX
  birth_message:
    topic: availability
    payload: online
  will_message:
    topic: availability
    payload: offline

ESPHome will then output all of the topics that are published or subscribed to:

[09:43:43][C][mqtt:133]: MQTT:
[09:43:43][C][mqtt:134]:   Server Address: 10.0.1.55:1883 (10.0.1.55)
[09:43:43][C][mqtt:136]:   Username: 'ratgdo'
[09:43:43][C][mqtt:137]:   Client ID: 'esphomeratgdo'
[09:43:43][C][mqtt:142]:   Topic Prefix: 'test_ratgdo'
[09:43:43][C][mqtt:144]:   Log Topic: 'test_ratgdo/debug'
[09:43:43][C][mqtt:147]:   Availability: 'availability'
[09:43:43][C][api:139]: API Server:
[09:43:43][C][api:140]:   Address: ratgdov25i.local:6053
[09:43:43][C][api:144]:   Using noise encryption: NO
[09:43:43][C][improv_serial:032]: Improv Serial:
[09:43:43][C][mqtt.sensor:028]: MQTT Sensor 'Openings':
[09:43:43][C][mqtt.sensor:032]:   State Topic: 'test_ratgdo/sensor/openings/state'
[09:43:44][C][mqtt.lock:034]: MQTT Lock 'Lock remotes': 
[09:43:44][C][mqtt.lock:035]:   State Topic: 'test_ratgdo/lock/lock_remotes/state'
[09:43:44][C][mqtt.lock:035]:   Command Topic: 'test_ratgdo/lock/lock_remotes/command'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Motion':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/motion/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Obstruction':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/obstruction/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Button':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/button/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Motor':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/motor/state'
[09:43:44][C][mqtt.number:033]: MQTT Number 'Rolling code counter':
[09:43:44][C][mqtt.number:034]:   State Topic: 'test_ratgdo/number/rolling_code_counter/state'
[09:43:44][C][mqtt.number:033]: MQTT Number 'Opening duration':
[09:43:44][C][mqtt.number:034]:   State Topic: 'test_ratgdo/number/opening_duration/state'
[09:43:44][C][mqtt.number:033]: MQTT Number 'Closing duration':
[09:43:44][C][mqtt.number:034]:   State Topic: 'test_ratgdo/number/closing_duration/state'
[09:43:44][C][mqtt.number:033]: MQTT Number 'Client ID':
[09:43:44][C][mqtt.number:034]:   State Topic: 'test_ratgdo/number/client_id/state'
[09:43:44][C][mqtt.cover:052]: MQTT cover 'Door':
[09:43:44][C][mqtt.cover:055]:   State Topic: 'test_ratgdo/cover/door/state'
[09:43:44][C][mqtt.cover:055]:   Command Topic: 'test_ratgdo/cover/door/command'
[09:43:44][C][mqtt.cover:057]:   Position State Topic: 'test_ratgdo/cover/door/position/state'
[09:43:44][C][mqtt.cover:058]:   Position Command Topic: 'test_ratgdo/cover/door/position/command'
[09:43:44][C][mqtt.light:078]: MQTT Light 'Light':
[09:43:44][C][mqtt.light:079]:   State Topic: 'test_ratgdo/light/light/state'
[09:43:44][C][mqtt.light:079]:   Command Topic: 'test_ratgdo/light/light/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Restart': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/restart/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/restart/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Safe mode boot': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/safe_mode_boot/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/safe_mode_boot/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Query status': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/query_status/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/query_status/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Query openings': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/query_openings/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/query_openings/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Sync': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/sync/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/sync/command'
[09:43:44][C][mqtt.button:029]: MQTT Button 'Toggle door': 
[09:43:44][C][mqtt.button:030]:   State Topic: 'test_ratgdo/button/toggle_door/state'
[09:43:44][C][mqtt.button:030]:   Command Topic: 'test_ratgdo/button/toggle_door/command'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Dry contact open':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/dry_contact_open/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Dry contact close':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/dry_contact_close/state'
[09:43:44][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'Dry contact light':
[09:43:44][C][mqtt.binary_sensor:023]:   State Topic: 'test_ratgdo/binary_sensor/dry_contact_light/state'

Topic YOUR_PREFIX/cover/door/state will contain the door status. You can control the door by sending payload open|close to topic YOUR_PREFIX/cover/door/command

I tried the above, but still not able to get MQTT to connect. I've tested my mqtt broker from another machine so I don't think it's an issue with my network. I'm getting some errors on connecting to the ESPHome API, so I may be missing a step to setup ESPHome correctly?

>esphome run v25iboard.yaml
INFO ESPHome 2023.12.6
INFO Reading configuration v25iboard.yaml...
INFO Updating https://github.com/ratgdo/esphome-ratgdo@None
INFO Updating https://github.com/ratgdo/esphome-ratgdo@None
INFO Detected timezone 'America/New_York'
INFO Generating C++ source...
INFO Compiling app...
Processing ratgdov25i (board: d1_mini; framework: arduino; platform: platformio/espressif8266@3.2.0)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/d1_mini.html
PLATFORM: Espressif 8266 (3.2.0) > WeMos D1 R2 and mini
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 @ 3.30002.0 (3.0.2)
 - tool-esptool @ 1.413.0 (4.13)
 - toolchain-xtensa @ 2.100300.220621 (10.3.0)
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ off, Compatibility ~ soft
Found 43 compatible libraries
Scanning dependencies...
Dependency Graph
|-- ESPAsyncTCP-esphome @ 2.0.0
|-- ESPAsyncWebServer-esphome @ 3.1.0
|-- ESP8266WiFi @ 1.0
|-- ESP8266mDNS @ 1.2
|-- AsyncMqttClient-esphome @ 2.0.0
|-- ArduinoJson @ 6.18.5
|-- Secplus @ 1.0.0+sha.f98c322
|-- EspSoftwareSerial @ 8.1.0+sha.57f1aa7
|-- Improv @ 1.2.3
Building in release mode
Retrieving maximum program size .pioenvs\ratgdov25i\firmware.elf
Checking size .pioenvs\ratgdov25i\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=====     ]  45.0% (used 36892 bytes from 81920 bytes)
Flash: [=====     ]  54.5% (used 569473 bytes from 1044464 bytes)
======================================================================================================================================================================================================= [SUCCESS] Took 5.53 seconds =======================================================================================================================================================================================================
INFO Successfully compiled program.
Found multiple options for uploading, please choose one:
  [1] COM6 (USB-SERIAL CH340 (COM6))
  [2] Over The Air (192.168.1.81)
(number): 2
INFO Connecting to 192.168.1.81
INFO Uploading .esphome\build/ratgdov25i\.pioenvs\ratgdov25i\firmware.bin (573632 bytes)
INFO Compressed to 380072 bytes
Uploading: [============================================================] 100% Done...

INFO Upload took 4.85 seconds, waiting for result...
INFO OTA successful
INFO Successfully uploaded program.
INFO Starting log output from 192.168.1.81 using esphome API
WARNING Can't connect to ESPHome API for ratgdov25i @ 192.168.1.81: Error connecting to [AddrInfo(family=<AddressFamily.AF_INET: 2>, type=<SocketKind.SOCK_STREAM: 1>, proto=6, sockaddr=IPv4Sockaddr(address='192.168.1.81', port=6053))]: [WinError 121] The semaphore timeout period has expired (SocketAPIError)
INFO Trying to connect to ratgdov25i @ 192.168.1.81 in the background
INFO Successfully connected to ratgdov25i @ 192.168.1.81 in 0.013s
INFO Successful handshake with ratgdov25i-174667 @ 192.168.1.81 in 0.054s
[20:38:01][W][ratgdo:468]: Not connected to GDO, ignoring command: GET_STATUS
[20:38:10][W][ratgdo:468]: Not connected to GDO, ignoring command: GET_STATUS
[20:38:21][W][mqtt:323]: MQTT Disconnected: TCP disconnected.
[20:38:21][I][mqtt:234]: Connecting to MQTT...
[20:38:23][W][ratgdo:468]: Not connected to GDO, ignoring command: GET_STATUS
[20:38:23][D][ratgdo:545]: Triggering sync failed actions.
[20:38:52][W][mqtt:323]: MQTT Disconnected: TCP disconnected.
[20:38:52][I][mqtt:234]: Connecting to MQTT...
[20:39:22][W][mqtt:323]: MQTT Disconnected: TCP disconnected.
[20:39:22][I][mqtt:234]: Connecting to MQTT...
[20:39:52][W][mqtt:323]: MQTT Disconnected: TCP disconnected.

I was able to get the solution above to work by doing a power cycle on ratgdo and then running esphome again.

I'm able to view mqtt status updates on a remote machine to the light by toggling the light in the web interface without it being connectted to RDO. However, if I send an mqtt message it doesn't appear to be received at the web interface. Is this expected without it being connected to RDO? I wanted to test it out before wiring the board up.

I was able to get the solution above to work by doing a power cycle on ratgdo and then running esphome again.

I'm able to view mqtt status updates on a remote machine to the light by toggling the light in the web interface without it being connectted to RDO. However, if I send an mqtt message it doesn't appear to be received at the web interface. Is this expected without it being connected to RDO? I wanted to test it out before wiring the board up.

I added the following to my v25iboard.yaml and it is receiving the mqtt message, but not updating the state in the web interface. Do I need to replicate all of the mqtt logic in here to act on those messages?


on_message:
    topic: ratgdo-back/light/light/state
    then:
      - logger.log: "Received light state"

the state in ratgdo should reflect the actual state of the gdo. The last message received via mqtt would be irrelevant

the state in ratgdo should reflect the actual state of the gdo. The last message received via mqtt would be irrelevant

I connected the ratgdo to my GDO and am able to monitor activity via mqtt, but sending mqtt messages from my automation system to ratgdo don't seem to take any action. Is there something else I need to do to get 2-way mqtt functionality?

I've been trying to get RatGDO setup with Mosquitto Broker on home assistant. I was having the same Web UI issue as others, but I don't have any network segmentation or firewall rules. I think in my case the issue might be Mosquitto Broker doesn't accept IP Address for the host. I setup MQTT Explorer and was able to connect to my broker with a FQDN, but not with the IP. This makes me think that if the RatGDO firmware allowed FQDN everything would work.

ratgdo / mqtt-ratgdo

No MQTT and no web UI after initial configuration #29