0.81: certain non-discoverable and manually-defined MQTT devices disappeared after upgrade

[NearlCrews]

Home Assistant release with the issue: 0.81

Last working Home Assistant release (if known): 0.80.3

Operating environment (Hass.io/Docker/Windows/etc.): Docker under Ubuntu

Component/platform: MQTT

Description of problem: The majority of my MQTT devices disappeared after the upgrade:

Oddly enough, the MQTT devices from esphomeyaml were just fine. The ones missing are all being sent by the SmartThings MQTT Bridge or from Blue Iris. I'm guessing it's because the esphomeyaml devices all get added to the entity registry when they're detected. The SmartThings MQTT Bridge and Blue Iris do not have these entries. I even when through each manually-defined MQTT entry and added an entity_id hoping it would correct the issue. No luck.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

binary_sensor:
- platform: mqtt
  state_topic: "smartthings/Sensor - Basement Doorwall/contact"
  name: "Basement Doorwall Contact"
  entity_id: 'binary_sensor.living_room_doorwall_contact'
  payload_on: "open"
  payload_off: "closed"
  device_class: door

Traceback (if applicable):

Additional information:

[NearlCrews]

FYI it's definitely related to 0.81 and not my environment. I downgraded to 0.80.3 and everything came back as expected:

...so I'm thinking my attempt at adding an entity_d to all of them was kind of pointless. This is also in the 0.81 logs at startup:

2018-10-27 08:58:03 INFO (MainThread) [homeassistant.components.mqtt.discovery] Found new component: switch nodemcu_valve_nodemcu_valve_switch
2018-10-27 08:58:03 INFO (MainThread) [homeassistant.components.mqtt.discovery] Found new component: switch Sonoff-Utility-Room-Bench_0
2018-10-27 08:58:03 INFO (MainThread) [homeassistant.components.mqtt.discovery] Found new component: switch Basement-Fan-Outlet_0
2018-10-27 08:58:03 INFO (MainThread) [homeassistant.components.mqtt.discovery] Found new component: switch Sonoff-Basement-Fan-Outlet_0
2018-10-27 08:58:03 INFO (MainThread) [homeassistant.components.mqtt.discovery] Found new component: switch sonoff_utility_light_ceiling_sonoff_utility_light_ceiling_switch

Please note those are all of my esphomeyaml and espurna MQTT devices. I'm not seeing the equivalent entries for my SmartThings MQTT Bridge or Blue Iris devices. Please let me know if there's anything I can provide to help get this resolved.

[NearlCrews]

One more comment based on this thread: it's definitely the manual ones that are disappearing while the auto-detected ones are just fine. I can't imagine this is intended functionality for MQTT and Home Assistant.

@OttoWinter, could this be related to all of the MQTT changes that were put in for the device registry? I don't believe the SmartThings MQTT Bridge and Blue Iris send a "proper" identifier in the MQTT events, but the manual MQTT entries worked before 0.81.0 was released.

[OttoWinter]

Hmmm, it probably is due to that. However, I don't really understand why.

One question: Is it only happening to manual entities of components (for example sensor) that also have MQTT discovery equivalents? For example, do none of the manual sensors work? Does it only work for components that do not have a MQTT discovery device too?

[NearlCrews]

@OttoWinter it was every MQTT sensor and binary_sensor that were manually defined. They've never appeared in the device registry but always worked before. Most of them look like this:

binary_sensor:

- platform: mqtt
  name: "FYC Motion"
  state_topic: blue_iris/binary_sensor/fyc_motion/state
  payload_on: "ON"
  payload_off: "OFF"
  device_class: motion

- platform: mqtt
  state_topic: "smartthings/Sensor - Laundry Room Door/contact"
  name: "Laundry Room Door Contact"
  payload_on: "open"
  payload_off: "closed"
  device_class: door

Auto-discovery has never worked with these since the topics aren't standard to HA, so they're all manually defined. Let me know if I can provide anything to assist. I'd love to see this one resolved.

[NearlCrews]

@OttoWinter I went back and it seemed like most of the manually-defined MQTT sensors seemed to be alright, e.g. temperature and humidity. These were not auto-discoverable. Scrolling to the bottom of #16943, I've got MQTT switches and a cover in my setup, but they're all auto-discovered since they're running esphomeyaml. Almost all of my SmartThings and Blue Iris sensors were binary_sensors, which are not auto-discoverable and disappeared after the update.

The missing ones seem to coincide with (1) not being auto-discoverable and (2) being one of the components listed at the bottom of #16943. I'm not the best person for debugging what changed but please let me know if there's any way I can assist.

[asantaga]

just for reference I have the same problem and yes my sensors are all manually defined too...

[NearlCrews]

Minor update: I noticed my Ring binary_sensor components disappeared too.

They're definitely not MQTT since they're using Ring's API. It almost seems like nothing is getting loaded from my binary_sensor include. The same MQTT sensors work just fine. Literally nothing changed in my config from 0.80.3 to 0.81.1. Does anyone have a clue? I turned on debug and I'm not seeing anything obvious other than it never seems to touch the binary_sensor directory.

[drogfild]

After 0.81 automations that have mqtt as a trigger doesn't get triggered. I'm not using mqtt in binary sensors and don't user any fancy include. Automations.yaml is of course included in normal way.

[NearlCrews]

Identified the cause in #17935.