Device unavailable after sometime and no turning back

[goliath2k]

Hi! First of all thanks for this great component. I'm not willing to use MQTT because I still want to use the shelly cloud, so this is just perfect.

I still only have 2 shelly, a Shelly 1 connected to a switch and a shelly 2 connected as a roller.

I installed shellyforhass with HACS and everything went fine for about a week. After that my Shelly2 just shows as unavailable although it's still connected to my network and I can control it in shelly cloud or browser with IP adress.

My network is managed by 2 Asus AC-5300 in AI Mode, right now my Shelly 1 who is working perfectly is connected to my main router (where my hassio is connected too) and the shelly 2 (unavailable) is connected to my mesh router. Could this be the problem? The SSID is the same, I can't guarantee if during the time both were working fine they were both connected to the same router, but if I had to guess I would say no because the distance of the shelly. Anyway, if this is the problem, I might try connect my raspberry (Hassio) to the router where the shelly is getting wi-fi signal.

Also I know there was some Hassio .x update and I'm not sure if it was after that.

[goliath2k]

Hi again, I forced the shellys to connect to the same node and no issues, so it seems that mesh systems act as AP for this problem. Any help setting proxy? I run the proxy.py on my windows and nothing happens, also test.py and nothing happens. The cursor just stays blinking.

[hakana]

The proxy.py should be running on the same lan as the Shelly. You need to update the script with the ip to your HA system. You also need to have routing and firewalls configure to allow UDP messages on port 5683.

[hakana]

Sounds the multicast not working between the different nodes. Check your settings in the nodes if you can see anything about IGMP settings or multicast.

[goliath2k]

Thanks @hakana I was on the same lan as shelly, but probably the problem was the configuration of ports. Just a noob question, the machine where I run the proxy must be kept on right? I tried on my workstation, but probably have to put it running on a server I have.

About the multicast I was reading the manual online, I will check as soon as I arrive at home. Anyway, right now is working fine just because both shellys are connected do my main router. Unfortunately in asus software is not possible to choose which node to connect unless I change the SSID (not desirable). I'll have to wait if it negotiates the other router, or next week when I'll had 2 more shellys that will be closer to the second node and almost sure they will connect to that one.

[goliath2k]

can't make it work... on my router and node I have: Enable multicast routing (IGMP Proxy): Enable Enable efficient multicast forwarding (IGMP Snooping): Enable UDP Proxy (Udpxy): 5683

Do I need to change the UDP_IP on proxy.py?

[goliath2k]

Well, something's weird on my network. Both shellys connected to the main router work just perfect, but even in that case the scripts test.py and proxy.py don't work. When I run them on cmd the cursor just keeps blinking. I already turned off the firewall from both router and computer (turned off access to internet first) and nothing happens. I have multicast active.

[hakana]

@goliath2k @jotacor Try to run "tcpdump -A udp port 5683" and see if you can see the messages sent from your Shellies.

[hakana]

We trying to solve this together with the Shelly team.

[goliath2k]

@hakana will try that today. Anyway I have some more details to share. Don't know if the component or shelly firmware can be improved in any way, but clearly there is some issue with my network. Will try some more things. My network is something like this:

https://drive.google.com/open?id=1PCw2Bv6P7jRxBMWXGuY8PdQ07E7Yd5_1

Both Asus (I said the wrong model above) have the same SSID, as a mesh should have, but the Shellys that connect trough the mesh node become unavailable to HASSIO (still totally fine in shelly cloud or even on Amazon Alexa).

I tried the following scenario with an old WiFi Powerline:

https://drive.google.com/open?id=1CCo2LltFbjcyqQ67mKUXgJkeXbXMpc7E

In this case I put this one with a different SSID so I can choose to where the shelly connects. I can have shellys connected to the router with one SSID and other to the WiFi's Powerline and all of them communicate to HASSIO with no problem. After that I made all Shelly connected only to the powerline and everythings fine.

I will now try 2 scenarios:

• put wireless backhaul on routers
• ethernet backhaul but eliminating powerline, in fact the powerline was a temporary solution, I already passed cable inside the wall, will try to crimp the terminals during the weekend.

[hakana]

Great, lot of people having the same problem. But for me it working and I have tried to get the problem here. I have tried to put different routers/switches but all working for me. Maybe there is problem with older routers mostly.

[goliath2k]

This ones are quite new, but could be a problem with the firmware in mesh mode, I will even try in AP mode, don't know if the powerline can in any way disturb the data sent. Is this model: https://www.asus.com/Networking/RT-AC88U/ Anyway, I will keep in touch with the tests I do, even if I found a solution to make it work, if it helps to improve the component or the shelly firmware or help any other user.

[diego065]

I have a "mesh" network and this problem happen to me too... Until the update with the last versione, I was having the devices disconnecting in different moment of the day, casually; after the update and rebooting, now I have 2 out of 5 shellies completely disappeared! No changes in network setting were made in the meantime Thanks

[hakana]

There have not been any changes in that part in the latest releases. Can you see any errors in the log?

[goliath2k]

I have a "mesh" network and this problem happen to me too... Until the update with the last versione, I was having the devices disconnecting in different moment of the day, casually; after the update and rebooting, now I have 2 out of 5 shellies completely disappeared! No changes in network setting were made in the meantime Thanks

@diego065 what mesh brand do you have? Just to see if a problem of Asus or affects other brands. Can you see those 2 missing shellys on your router page? Ar the connected to the main router or a node?

[diego065]

There have not been any changes in that part in the latest releases. Can you see any errors in the log?

In the normal log I have no messages other than the warning (You are using a custom integration for shelly which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant.)... is there a way to see a log only for this component? Actually both shellies re-appeared but the status is unavailable!

@diego065 what mesh brand do you have? Just to see if a problem of Asus or affects other brands. Can you see those 2 missing shellys on your router page? Ar the connected to the main router or a node?

I have the tplink deco P7... I can see both on the router page and they are working correctly within the app!

[diego065]

I did some test and I confirm that is something related to the network: when I switch off the second floor device and I reconnect the shellys to the "main" network, they magically re-appears on HA...

[goliath2k]

@diego065 the P7 are the powerline version right? Wonder if the Homeplug protocol as something to do with that as my mesh is Asus but they both are connected to each other with a powerline. Have to try to connect them with a cable.

[goliath2k]

@hakana one update Asus mesh with wi-fi backhaul doesn't work also. I disabled mesh and put one of the routers in AP mode, this way it totally works, I can even have 2 different shellys connected to 2 different AP with different SSID and it works, so it's something related with mesh systems. I have to search what router does different between mesh mode ou AP mode.

[JohanLeirnes]

Im having this issue aswell. Im using a mixture of Extenders, Powerlines and Aps Main router: Sophos UTM9 Connection: Shelly -> Devolo dLan Powerline 1200+ -> Linksys RE7000 -> Linksys EA9500 -> Sophos UTM9.

From what i can tell multicast is on everywhere, but some of the devices does not have the option to enable/disable it.

(the shelly is mounted in my garage about 300m away from my house thats why the path is so long)

[JohanLeirnes]

Also tcpdump -A udp port 5683 shows nothing so guess something is being blocked.

[hakana]

Also tcpdump -A udp port 5683 shows nothing so guess something is being blocked.

It you have a laptop and start to connect it to same network as Shelly, you can find out where the multicast is blocked.

[marchammer2003]

I'm having the same problem. My network consists of a single router where all my devices (including shelly's) is connected to. What is my options to avoid the "Unavailable" status in the future?

[Misiu]

@goliath2k I'm planning to buy TP_link Deco M5, but now You got me worried. I want to have mesh, because of large area I must cover. I want to use Shelly with HA and I'd like to be 100% sure that this will work. @diego065 were You able to solve this? Mesh networks are new and many brands often update the firmware. Maybe something changed during the last days? Maybe this should be reported to TP-link and Asus?

[diego065]

@Misiu sorry for delay... Actually even if I disconnected the tplink upstairs, the problem wasn't solve completely: the shellies sometimes get disconnect from HA for various periods of time (sometimes for seconds and sometimes for hours!). Of course when I can't see them on HA, they are correctly working on the shelly app so it's not a problem of the tplink itself! Actually I am really thinking to go back to rest component...

[Misiu]

@diego065 I have the exact same issue. I've installed HA on Raspberry Pi in my parent's house (vanilla setup). I've added 4 Shelly 2.5 that control roller shutters using ShellyForHASS, yesterday everything worked fine, but today two Shelly are shown as unavailable and two were working just fine. I logged in into the Shelly app and four Shelly devices were there. This is really annoying and frustrating. The best part is that I restarted Home Assistant a couple of times and that didn't help but after some time two Shelly devices that were missing shown in Home Assistant.

@hakana any news from Shelly team? @diego065 what rest component do You mean? Can You share a sample config? I think that I will give it a try. I wanted to use MQTT, but if I enable MQTT on Shelly I lose cloud connection, I want and I need both, but currently there is no way to have both.

[diego065]

Hey, this is my old config before this addon came out

• platform: rest name: "Bagno 1 piano" scan_interval: 5 resource: http://192.168.1.128/relay/0 body_on: 'turn=on' body_off: 'turn=off' is_on_template: '{{ value_json.ison == true}}' headers: content-type: application/x-www-form-urlencoded

• platform: rest name: "LED 1 piano" scan_interval: 5 resource: http://192.168.1.128/relay/1 body_on: 'turn=on' body_off: 'turn=off' is_on_template: '{{ value_json.ison == true}}' headers: content-type: application/x-www-form-urlencoded

[Misiu]

@diego065 thank You for sending config. Do You have a config for roller shutter mode? The biggest problem with rest (in my opinion) is the need to query Shelly devices every X seconds. CoAP or MQTT doesn't need to pool devices periodically.

I've posted a question on the Shelly Facebook page and contacted their support team. I'll update this thread with info (if I'll get any)

[diego065]

Unfortunately it's probably not possibile or at least the position won't be showed! For roller shutter you probably have to use the shell command... Let me know if they will reply or maybe if is possible to "fix" this addon!

[Misiu]

@diego065 if I get any reply I'll sure post it here. This is frustrating. 3 roller shutters are working fine and one is offline, but it was working the whole day, and now puff...it is gone form HA. @hakana did You figure something out with Shelly team?

[Misiu]

@diego065 I got this reply:

We check and can't find any issue form Shelly side. Please HASS creators to check why their plugin do not work correct.

and this one:

shelly always send CoAP messages, but maybe network setup filtering them. We can't do anything in that case.

but the devs didn't want any logs or didn't ask for any clarification. As I wrote, if this would be a router issue then all Shelly devices would be unavailable, but in my case, one or two devices are unavailable for some time and after a while, everything gets back to normal. This is confusing :(

[diego065]

Do you think that there should be different network settings in the shelly to make them works? Actually I disabled the "mesh" property on the shelly in the router page but nothing change!

[Misiu]

@diego065 I have problems with Shelly devices availability on a normal router. I don't have a mesh system, I want to buy one, but now I've started wondering if it makes sense.

Do You know if it is possible to log into Home Assistant when the shelly device got unavailable and when it got available again? I have some not used Shelly devices (about 10) so I can connect them and observe on more sensors.

The worst part is that Shelly support said that everything works and they didn't even propose a way to log anything to check which fault is it: Shelly device, the router, ShellyForHASS or something else.

@hakana can we log devices availability using Home Assistant?

[diego065]

If you look and the device properties you can see how often the device is unavailable.. I have a screenshot of mine but I can’t post image in here... maybe send me your email address in private and I will share it with you... however it happens quite often: about 5-6 times/hour, sometimes for a couple of minutes and sometimes even more than 1 hour, I don’t know depending on what!

[Misiu]

Did You tried igmp_fix: true in configuration.yaml? I have a tp-link router running openwrt at my home and I didn't have any issues, but my parents have tp-link with stock firmware and I had (and still have) problem there). Did You tried logging CoAP using test.py?

[diego065]

how to do that (the test.py..)? What "igmp_fix" does?

[Misiu]

@diego065 take a look at the readme

igmp_fix	Enable sending out IP_ADD_MEMBERSHIP every minute

what operating system are You using? DO You know how to run a python script? test.py allows You to check if this isn't an issue with Your router, basically, it scans for CoAP messages. They should be send every 15 seconds by all shelly devices.

[Misiu]

@hakana I think that I found something. After looking at logs I found this:

2019-10-24 19:44:20 ERROR (Thread-2) [pyShelly] Error http GET: http://192.168.0.30/settings [Errno 113] Host is unreachable Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/pyShelly/init.py", line 209, in http_get conn.request("GET", url, None, headers) File "/usr/local/lib/python3.7/http/client.py", line 1244, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1290, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1239, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output self.send(msg) File "/usr/local/lib/python3.7/http/client.py", line 966, in send self.connect() File "/usr/local/lib/python3.7/http/client.py", line 938, in connect (self.host,self.port), self.timeout, self.source_address) File "/usr/local/lib/python3.7/socket.py", line 727, in create_connection raise err File "/usr/local/lib/python3.7/socket.py", line 716, in create_connection sock.connect(sa) OSError: [Errno 113] Host is unreachable Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/pyShelly/init.py", line 209, in http_get conn.request("GET", url, None, headers) File "/usr/local/lib/python3.7/http/client.py", line 1244, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1290, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1239, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output self.send(msg) File "/usr/local/lib/python3.7/http/client.py", line 966, in send self.connect() File "/usr/local/lib/python3.7/http/client.py", line 938, in connect (self.host,self.port), self.timeout, self.source_address) File "/usr/local/lib/python3.7/socket.py", line 727, in create_connection raise err File "/usr/local/lib/python3.7/socket.py", line 716, in create_connection sock.connect(sa) OSError: [Errno 113] Host is unreachable

Shellies are available in App (Official Shelly App), I see them on the router page, but for some reason HA can't do a request.

[Misiu]

@hakana another clue. I added a ping sensor that pings the Shelly devices. For reasons unknown to me two devices were offline (maybe wifi issue, can't tell why):

It looks like pyShelly is trying to query /settings endpoint, but when it won't get any reply it shows the entity as unavailable. The wors part is that even when the device will get back to online (based on my ping graph) pySHelly won't query it again (maybe it is querying it, but the entity stays unavailable until I restart HA). Is there a way to tell PyShelly to re-query the device and show the entity as available?

[Misiu]

@diego065 @goliath2k please add ping sensor and check if Shelly devices are available and reachable form HA. Sample config (put this into the configuration.yaml file):

binary_sensor:
  - platform: ping
    host: 192.168.0.30
    name: Shelly kitchen
    scan_interval: 30

  - platform: ping
    host: 192.168.0.181
    name: Shelly bedroom
    scan_interval: 30    

Please post your results here. It will be very helpful.

[hakana]

@Misiu Shelly4Hass show the device unavailable when not got any status up update since 60 sec. You can change this using the unavailable_after_sec param.

I'm working with a new version that are more stable. I will also add polling when COAPnot working. But I have limited time so it will take some time ;(

[Misiu]

@hakana thank You for the reply. I'll try adding unavailable_after_sec param. If I understand correctly coap is used only to listen to broadcast messages (similar to test.py), but then You use REST calls to get info from the device.

Would it be possible to show the entity (device) as offline when REST requests fail, but still query that device (/settings endpoint) every X seconds/minutes? This way when the endpoint finally responds the entity (device) will be shown as online. This approach is used for example in MQTT sensors.

I'll try to run test.py in my test environment and will plug/unplug the router and the Shelly devices.

One more question: is ShellyForHASS enabling the entity when the device will get online? Simple use case: -I have Shelly 2.5 added as an entity in HA -I disconnect Shelly -it is shown as device unavailable -I connect the device back -What ShellyForHASS is doing???

Will ShellyForHASS show the device again (entity should be available again)? Can ShellyForHASS show the device as offline when it is disconnected?

[diego065]

This is what my log look like... @Misiu , yours is really really better!! I just added the ping sensor.. I will post the results tomorrow!

[diego065]

Here the results... Look like the device is always available on ping but shelly is still often unavailable like the picture above!

[Misiu]

@hakana I used openHAB some time ago and I just got info that they have official binding for Shelly - https://www.openhab.org/addons/bindings/shelly/

If someone has errors and problems with CoAP please try openHAB. I'm not suggesting to migrate from HA, but I'd like to ask people to check if CoAP communication will work there. If everything will work maybe it will be easier to figure out what is wrong and we will be able to fix it.

I like HA and I'm amazed by the work of @hakana, but like everyone here, I want this component to work perfectly and be added as an official one. This way the whole HA community could help.

@hakana please add patreon account, I think that not only I would like to support you 🙂

[chemelli74]

@goliath2k , @Misiu , @diego065 can you test with latest plugin 0.1.6b7 and latest firmare v.1.5.7 ?

Thank you,

Simone

[chemelli74]

Hi all, just to inform that after 3 weeks of inactivities on very old thread, I'll proceed to close them. If later one someone verify that the issue is still there, a new issue need to be opened.

Thank you,

Simone

[chemelli74]

No update for 21 days, closing

Simone