Devices discovered, but not showing up in MQTT

[caspermltov]

Describe the bug Upon running the Shellies Announce Gen2 automation, we can trace the output of the subsequent Shellies Discovery Gen2 that is being called.

params:
  domain: python_script
  service: shellies_discovery_gen2
  service_data:
    id: shelly1pmminig3-5432046c9db0
    device_config:
      ble:
        enable: false
        rpc:
          enable: true
        observer:
          enable: false
      cloud:
        enable: true
        server: shelly-100-eu.shelly.cloud:6022/jrpc
      input:0:
        id: 0
        name: null
        type: switch
        enable: true
        invert: false
        factory_reset: true
      mqtt:
        enable: true
        server: xx.xx.xx.xx:1883
        client_id: shelly1pmminig3-5432046c9db0
        user: mqtt
        topic_prefix: shelly
        rpc_ntf: true
        status_ntf: true
        use_client_cert: false
        enable_rpc: true
        enable_control: true
      script:1:
        id: 1
        name: shellies_discovery_gen2_script_20240216
        enable: true
...

We can see that the shellies are being found. The log file from MQTT itself, also suggest so:

2024-07-11 00:21:03.748 DEBUG (MainThread) [homeassistant.components.mqtt.client] Mosquitto broker: register write 84
2024-07-11 00:21:03.748 DEBUG (MainThread) [homeassistant.components.mqtt.client] Transmitting message on shelly/rpc: '{"id":1,"src":"shellies_discovery","method":"Shelly.GetConfig"}', mid: 431, qos: 0
2024-07-11 00:21:03.763 DEBUG (MainThread) [homeassistant.components.mqtt.client] Mosquitto broker: unregister write 84
2024-07-11 00:21:03.814 DEBUG (MainThread) [homeassistant.components.mqtt.client] Received message on shellies_discovery/rpc (qos=0): b'{"id":1,"src":"shellyplus1-e465b8f30000","dst":"shellies_discovery","result":{"ble":{"enable":false,"rpc":{"enable":true},"observer":{"enable":false}},"cloud":{"enable":true,"server":"shelly-100-eu.shelly.cloud:6022/jrpc"},"input:0":{"id":0,"name":null,"type":"switch","enable":true,"invert":false,"factory_reset":true},"mqtt":{"enable":true,"server":"xx.xx.xx.xx:1883","client_id":"shellyplus1-e465b8f30000","user":"mqtt","topic_prefix":"shelly","rpc_ntf":true,"status_ntf":true,"use_client_cert":false,"enable_rpc":true,"enable_control":true},"script:1":{"id":1, "name":"shellies_discovery_gen2_script_20240216", "enable":true},"switch:0":{"id":0, "name":null,"in_mode":"follow","initial_state":"match_input", "auto_on":false, "auto_on_delay":60.00, "auto_off":false, "auto_off_delay":60.00},"sys":{"device":{"name":null,"mac":"E465B8F30000","fw_id":"20231031-152227/1.0.7-g5db02bd","discoverable":true,"eco_mode":false,"addon_type":"sensor"},"location":{"tz":null,"lat":null,"lon":null},"debug":{"level":2,"file_level":null,"mqtt":{"enable":false},"websocket":{"enable":false},"udp":{"addr":null}},"ui_data":{},"rpc_udp":{"dst_addr":null,"listen_port":null},"sntp":{"server":"time.google.com"},"cfg_rev":13},"temperature:100":{"id":100, "name":null, "report_thr_C": 0.1, "offset_C": 0.0},"wifi":{"ap":{"ssid":"ShellyPlus1-E465B8F30000","is_open":true, "enable":false, "range_extender": {"enable":false}},"sta":{"ssid":"xxxxxx","is_open":false, "enable":true, "ipv4mode":"dhcp","ip":null,"netmask":null,"gw":null,"nameserver":null},"sta1":{"ssid":null,"is_open":true, "enable":false, "ipv4mode":"dhcp","ip":null,"netmask":null,"gw":null,"nameserver":null},"roam":{"rssi_thr":-80,"interval":60}},"ws":{"enable":false,"server":null,"ssl_ca":"ca.pem"}}}'
2024-07-11 00:21:03.961 DEBUG (MainThread) [homeassistant.components.mqtt.client] Received message on shellies_discovery/rpc (qos=0): b'{"id":1,"src":"shelly1pmminig3-5432046c9db0","dst":"shellies_discovery","result":{"ble":{"enable":false,"rpc":{"enable":true},"observer":{"enable":false}},"cloud":{"enable":true,"server":"shelly-100-eu.shelly.cloud:6022/jrpc"},"input:0":{"id":0,"name":null,"type":"switch","enable":true,"invert":false,"factory_reset":true},"mqtt":{"enable":true,"server":"xx.xx.xx.xx:1883","client_id":"shelly1pmminig3-5432046c9db0","user":"mqtt","topic_prefix":"shelly","rpc_ntf":true,"status_ntf":true,"use_client_cert":false,"enable_rpc":true,"enable_control":true},"script:1":{"id":1, "name":"shellies_discovery_gen2_script_20240216", "enable":true},"switch:0":{"id":0, "name":null,"in_mode":"follow","initial_state":"match_input", "auto_on":false, "auto_on_delay":60.00, "auto_off":false, "auto_off_delay":60.00,"power_limit":2240,"voltage_limit":280,"autorecover_voltage_errors":false,"current_limit":8.000},"sys":{"device":{"name":null,"mac":"5432046C9DB0","fw_id":"20231121-110955/1.1.99-minig3prod1-ga898543","discoverable":true,"eco_mode":false},"location":{"tz":null,"lat":null,"lon":null},"debug":{"level":2,"file_level":null,"mqtt":{"enable":false},"websocket":{"enable":false},"udp":{"addr":null}},"ui_data":{},"rpc_udp":{"dst_addr":null,"listen_port":null},"sntp":{"server":"time.google.com"},"cfg_rev":10},"wifi":{"ap":{"ssid":"Shelly1PMMiniG3-5432046C9DB0","is_open":true, "enable":false, "range_extender": {"enable":false}},"sta":{"ssid":"xxxxxx","is_open":false, "enable":true, "ipv4mode":"dhcp","ip":null,"netmask":null,"gw":null,"nameserver":null},"sta1":{"ssid":null,"is_open":true, "enable":false, "ipv4mode":"dhcp","ip":null,"netmask":null,"gw":null,"nameserver":null},"roam":{"rssi_thr":-80,"interval":60}},"ws":{"enable":false,"server":null,"ssl_ca":"ca.pem"}}}'
2024-07-11 00:21:04.034 DEBUG (MainThread) [homeassistant.components.mqtt.client] Received message on shellies_discovery/rpc (qos=0): b'{"id":1,"src":"shellypro3em-34987a45b1dc","dst":"shellies_discovery","result":{"ble":{"enable":false,"rpc":{"enable":true},"observer":{"enable":false}},"cloud":{"enable":true,"server":"shelly-100-eu.shelly.cloud:6022/jrpc"},"em:0":{"id":0, "name":null,"blink_mode_selector":"active_energy" ,"phase_selector":"all", "monitor_phase_sequence":false,"reverse":{"a":true,"b":true,"c":true}},"emdata:0":{},"eth":{"enable":true,"ipv4mode":"dhcp","ip":null,"netmask":null,"gw":null,"nameserver":null},"modbus":{"enable":true},"mqtt":{"enable":true,"server":"xx.xx.xx.xx:1883","client_id":"shellypro3em-34987a45b1dc","user":"mqtt","ssl_ca":null,"topic_prefix":"shelly","rpc_ntf":true,"status_ntf":true,"use_client_cert":false,"enable_rpc":true,"enable_control":true},"script:1":{"id":1, "name":"shellies_discovery_gen2_script_20240216", "enable":true},"sys":{"device":{"name":null,"mac":"34987A45B1DC","fw_id":"20240625-123006/1.3.3-gbdfd9b3","discoverable":true,"eco_mode":false,"profile":"triphase","addon_type":null},"location":{"tz":"Europe/Copenhagen","lat":55.446700,"lon":11.807600},"debug":{"level":2,"file_level":null,"mqtt":{"enable":false},"websocket":{"enable":false},"udp":{"addr":null}},"ui_data":{},"rpc_udp":{"dst_addr":null,"listen_port":null},"sntp":{"server":"time.google.com"},"cfg_rev":18},"temperature:0":{"id":0, "name":null, "report_thr_C": 5.0, "offset_C": 0.0},"wifi":{"ap":{"ssid":"ShellyPro3EM-34987A45B1DC","is_open":true, "enable":false, "range_extender": {"enable":false}},"sta":{"ssid":"xxxxxx","is_open":false, "enable":true, "ipv4mode":"dhcp","ip":null,"netmask":null,"gw":null,"nameserver":null},"sta1":{"ssid":null,"is_open":true, "enable":false, "ipv4mode":"dhcp","ip":null,"netmask":null,"gw":null,"nameserver":null},"roam":{"rssi_thr":-80,"interval":60}},"ws":{"enable":false,"server":null,"ssl_ca":"ca.pem"}}}'
2024-07-11 00:21:04.175 DEBUG (MainThread) [homeassistant.components.mqtt.client] Mosquitto broker: register write 84
2024-07-11 00:21:04.175 DEBUG (MainThread) [homeassistant.components.mqtt.client] Transmitting retained message on shelly/switch/shellyplus1-e465b8f30000-0/config: '{"name": "Relay 0", "cmd_t": "~rpc", "pl_off": "{\"id\":1,\"src\":\"home-assistant\",\"method\":\"Switch.Set\",\"params\":{\"id\":0,\"on\":false}}", "pl_on": "{\"id\":1,\"src\":\"home-assistant\",\"method\":\"Switch.Set\",\"params\":{\"id\":0,\"on\":true}}", "stat_t": "~status/switch:0", "val_tpl": "{%if value_json.output%}on{%else%}off{%endif%}", "stat_off": "off", "stat_on": "on", "avty": [{"t": "~online", "pl_avail": "true", "pl_not_avail": "false"}, {"t": "~status/rpc", "val_tpl": "{%if value_json.mqtt.connected%}online{%else%}offline{%endif%}"}], "uniq_id": "shellyplus1-e465b8f30000-0", "qos": 0, "dev": {"cns": [["mac", "E4:65:B8:F3:00:00"]], "name": "Shelly Plus 1", "mdl": "Shelly Plus 1", "sw": "20231031-152227/1.0.7-g5db02bd", "hw": "gen2 (SNSW-001X16EU)", "mf": "Allterco Robotics", "cu": "http://shellyplus1-e465b8f30000.local/"}, "o": {"name": "Shellies Discovery Gen2", "sw": "2.30.0", "url": "https://github.com/bieniu/ha-shellies-discovery-gen2"}, "~": "shelly/"}', mid: 432, qos: 0
2024-07-11 00:21:04.179 DEBUG (MainThread) [homeassistant.components.mqtt.client] Mosquitto broker: unregister write 84
2024-07-11 00:21:04.185 DEBUG (MainThread) [homeassistant.components.mqtt.client] Mosquitto broker: register write 84
2024-07-11 00:21:04.187 DEBUG (MainThread) [homeassistant.components.mqtt.client] Transmitting retained message on shelly/light/shellyplus1-e465b8f30000-0/config: '', mid: 433, qos: 0
2024-07-11 00:21:04.192 DEBUG (MainThread) [homeassistant.components.mqtt.client] Mosquitto broker: unregister write 84
2024-07-11 00:21:04.195 DEBUG (MainThread) [homeassistant.components.mqtt.client] Mosquitto broker: register write 84
2024-07-11 00:21:04.196 DEBUG (MainThread) [homeassistant.components.mqtt.client] Transmitting retained message on shelly/fan/shellyplus1-e465b8f30000-0/config: '', mid: 434, qos: 0
2024-07-11 00:21:04.198 DEBUG (MainThread) [homeassistant.components.mqtt.client] Mosquitto broker: unregister write 84
2024-07-11 00:21:04.206 DEBUG (MainThread) [homeassistant.components.mqtt.client] Mosquitto broker: register write 84
2024-07-11 00:21:04.208 DEBUG (MainThread) [homeassistant.components.mqtt.client] Transmitting retained message on shelly/sensor/shellyplus1-e465b8f30000-0-temperature/config: '{"name": "Relay 0 Temperature", "en": "false", "uniq_id": "shellyplus1-e465b8f30000-0-temperature", "qos": 0, "dev": {"cns": [["mac", "E4:65:B8:F3:00:00"]], "name": "Shelly Plus 1", "mdl": "Shelly Plus 1", "sw": "20231031-152227/1.0.7-g5db02bd", "hw": "gen2 (SNSW-001X16EU)", "mf": "Allterco Robotics", "cu": "http://shellyplus1-e465b8f30000.local/"}, "o": {"name": "Shellies Discovery Gen2", "sw": "2.30.0", "url": "https://github.com/bieniu/ha-shellies-discovery-gen2"}, "~": "shelly/", "avty": [{"t": "~online", "pl_avail": "true", "pl_not_avail": "false"}, {"t": "~status/rpc", "val_tpl": "{%if value_json.mqtt.connected%}online{%else%}offline{%endif%}"}], "val_tpl": "{{value_json[\"switch:0\"].temperature.tC}}", "stat_t": "~status/rpc", "unit_of_meas": "°C", "dev_cla": "temperature", "ent_cat": "diagnostic", "stat_cla": "measurement", "sug_dsp_prc": 1}', mid: 435, qos: 0
2024-07-11 00:21:04.210 DEBUG (MainThread) [homeassistant.components.mqtt.client] Mosquitto broker: unregister write 84
2024-07-11 00:21:04.215 DEBUG (MainThread) [homeassistant.components.mqtt.client] Mosquitto broker: register write 84

However, no devices show up under MQTT.

Are we doing something wrong? as far as I can tell, we have followed the instructions in the readme, and have spent a few hours trying to debug it ourselves.

However, we still see no devices within MQTT.

Expected behavior The 3 shellies to show up in MQTT

Versions:

• Home Assistant: 2024.5.5
• Shellies Discovery GEN2: newest script, fetched today
• Shelly device firmware: 20240625-123006/1.3.3-gbdfd9b3

Shellies Discovery GEN2 automation:

- id: shellies_announce_gen2
  alias: "Shellies Announce Gen2"
  trigger:
    - platform: homeassistant
      event: start
  variables:
    device_info_payload:  "{{ {'id': 1, 'src':'shellies_discovery', 'method':'Shelly.GetConfig'} | to_json }}"
    device_ids:  # enter the list of device IDs (MQTT prefixes) here
      - shelly
  action:
    - repeat:
        for_each: "{{ device_ids }}"
        sequence:
          - service: mqtt.publish
            data:
              topic: "{{ repeat.item }}/rpc"
              payload: "{{ device_info_payload }}"

- id: shellies_discovery_gen2
  alias: "Shellies Discovery Gen2"
  mode: queued
  max: 999
  trigger:
  - platform: mqtt
    topic: shellies_discovery/rpc
  action:
  - service: python_script.shellies_discovery_gen2
    data:
      id: "{{ trigger.payload_json.src }}"
      device_config: "{{ trigger.payload_json.result }}"
      discovery_prefix: "shelly"
  - service: mqtt.publish
    data:
      topic: "{{ trigger.payload_json.result.mqtt.topic_prefix }}/command"
      payload: "status_update"

Shelly device MQTT settings:

Debug log:

[bieniu]

discovery_prefix: "shelly"

I bet this is wrong.

Using shelly as a device ID is also a very bad idea. Device ID should be unique.

[JapSeyz]

Hey,

Alright. Both of them are listed as prefixes, (device_ids: # enter the list of device IDs (MQTT prefixes) here), which made us hope that we didn't have to list the exact device ids manually.

I will try to remove the discovery prefix, and use unique names, and see if that helps.

[JapSeyz]

That worked. Brilliant. Thanks!

Do you know if we can change the name of the client Id to include spaces and match the name of the device in Shelly?

Currently it gets added with the device type (ie Shelly Pro 3EM) instead of "Solar Panels", which it is called within Shelly.

[bieniu]

Do you know if we can change the name of the client Id to include spaces and match the name of the device in Shelly?

Client ID also must be unique, probably it cannot contain spaces, I'm not sure. The best solution is not to change the client ID and leave it as default.

[JapSeyz]

Client ID also must be unique, probably it cannot contain spaces, I'm not sure. The best solution is not to change the client ID and leave it as default.

Perfect, we've done that. We're still not getting the original name from Shelly, but that may not be possible?

[bieniu]

[JapSeyz]

It is me being daft, because I am trying to help a friend with HA/Shelly without really having worked with it before.

"Device configuration", does that mean the configuration within shelly, or the configuration of device-ids in the automations.yaml?

We did try changing the second name here "shellypro3em..." to solar panels, but still without luck unfortunately.

[bieniu]

"Device configuration", does that mean the configuration within shelly, or the configuration of device-ids in the automations.yaml?

"Device configuration" means the device configuration in the device web panel. I think you configure the device via the Shelly cloud and the configuration looks different there, I don't know, I don't use the cloud. Go to http://<DEVICE_IP>/ and then Settings -> Device name

[iMusicJJ]

Awesome!

There is no way to automatically detect every device that connects to MQTT. You have to input the device-ids into the list, manually?

[bieniu]

Currently, there is such an option (it was not there when the script was created) but implementing it would be a breaking change for all users, so I don't want to go in this direction.

[JapSeyz]

I see. Thanks for the swift replies. If you end up making a guide or change to allow automatic discovery without whitelisting,can you ping me, if you remember?

[bieniu]

Sure