kevinvincent / ha-wyzesense

A Home Assistant Component to interface with the WYZE Sense hub and sensor system
368 stars 97 forks source link

Sensors Stop Responding - Requires Reboot #114

Open cheechie opened 4 years ago

cheechie commented 4 years ago

I was running hassio on a pie and the wyze sensors would just stop responding after a day or sometimes 3 days. A reboot would fix the problem. Recently I have switched to running home assistant in a VM (Proxmox), but i still get the same issue. A reboot of home assistant fixes it every time.

Let me know if I can provide any logs to help.

eregev commented 4 years ago

+1. Exact same scenario. Thought migrating to a Debian VM might help but it still requires that reboot frequently.

johnwarne commented 4 years ago

Same for me. I've tried using previous versions of the component and keep needing to restart for the component to work again. It is AWESOME when it works but unfortunate when it doesn't. Here's what my last log on 0.0.7 says about the error:

Log Details (ERROR)
Logger: custom_components.wyzesense.wyzesense_custom 
First occured: 11:15:30 AM (1 occurences) 
Last logged: 11:15:30 AM

[Errno 110] Operation timed out

I'm on a HassOS 3.11 on a VirtualBox VM.

photinus commented 4 years ago

Can you put your logs into Debug?

logger: default: info logs: custom_components.wyzesense: debug ~Will

On Fri, Mar 6, 2020 at 10:18 AM John Warne notifications@github.com wrote:

Same for me. I've tried using previous versions of the component and keep needing to restart for the component to work again. It is AWESOME when it works but unfortunate when it doesn't. Here's what my last log on 0.0.7 says about the error:

Log Details (ERROR) Logger: custom_components.wyzesense.wyzesense_custom First occured: 11:15:30 AM (1 occurences) Last logged: 11:15:30 AM

[Errno 110] Operation timed out

I'm on a HassOS 3.11 on a VirtualBox VM.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kevinvincent/ha-wyzesense/issues/114?email_source=notifications&email_token=AC362MOG2AY4ZBLKIHG7R3TRGE5A3A5CNFSM4LC7JNG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOCKJZQ#issuecomment-595895526, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC362MIM53ECNFHX6Q4TYYTRGE5A3ANCNFSM4LC7JNGQ .

cheechie commented 4 years ago

I am going to try this in my config until it gets sorted out. And have it reboot everyday.

restart HA to keep it fresh

https://community.home-assistant.io/t/the-ability-to-schedule-a-reboot/30144

eregev commented 4 years ago

@cheechie I've had mine rebooting for other reasons nightly. Seems like wyze gets through most of the day but by evening it's always in need of a reboot. Too bad hourly restarts aren't feasible for the time being...

johnwarne commented 4 years ago

Can you put your logs into Debug? logger: default: info logs: custom_components.wyzesense: debug ~Will

You got it. I've got a log for a timeframe during which the component was working then failed. Where would you like me to send it?

swingking commented 4 years ago

FYI, my system is also having this problem. I'm running HA on Ubuntu 18.04 in Docker container. It used to work fine but then I had a server power outage and ever since I've had issues. The only difference from the other comments is sometimes I have to stop the container, pull the controller for 10 sec, replace and then restart the container. I've turned on logging so if it happens again I'll upload the logs.

photinus commented 4 years ago

@johnwarne can you post it to Pastebin and share a link here?

photinus commented 4 years ago

btw, I'm pretty sure what is happening is the Home Assistant supervisor is restarting HA for some reason and we're not closing the connection the USB dongle. Then when it tried to re-initialize the connection it times out waiting on the initialization stage.

johnwarne commented 4 years ago

Sure thing. I've removed everything except the component's logs: https://pastebin.com/Q8ETakPQ

photinus commented 4 years ago

@johnwarne Can you try updating? You're on 0.0.4 from the debug logs, 0.0.7 fixes a lot of random issues.

johnwarne commented 4 years ago

Apologies, I've been bouncing around between 0.0.5, 0.0.6, and 0.0.7. In HACS it says I'm currently on 0.0.6: https://www.dropbox.com/s/b3khmm9185t6mvk/Screenshot%202020-03-06%2014.16.32.png?dl=0. I'll update from 0.0.6 to 0.0.7 and will update this thread if/when I get another failure.

photinus commented 4 years ago

Thanks @johnwarne , also please note the color of the LED on the dongle when it fails?

johnwarne commented 4 years ago

@photinus The component has errored again. The light on the dongle is blue. Here's a link to the debug log:

https://pastebin.com/e4BA9CyP

kevinvincent commented 4 years ago

@johnwarne See this information on this error in virtualbox here: https://github.com/kevinvincent/ha-wyzesense/issues/20#issuecomment-595829838 Basically it involves switching your virtual machine to use a different (more stable) usb host controller This is unfortunately an issue where the OS drops the connection with the USB device.

grantalewis commented 4 years ago

EDIT (about 2 hours later): Just applied a HACS update and the problem seems to be resolved.

= = = = = = = = = =

Apologies if the above already discusses the problem I'm having.

This morning I noticed that all of my Wyze sensors had stopped responding. I had added two sensors yesterday, but they were functioning fine when I went to bed. To troubleshoot, I restored to an earlier Snapshot which seemed to fix the problem... until I rebooted at which time all sensors went offline once again.

I'm Running Home Assistant 0.106.5 on a Mac mini, Ubuntu 18.04.4 LTS, Docker. Supervisor version is 209.

I installed the Wyze Sense Component through HACS.

Previously-working configuation.yaml entry:

binary_sensor:
  - platform: wyzesense
    device: "/dev/hidraw0"

Under Supervisor | System | Hardware | serial I see an entry for

/dev/serial/by-id/usb-0658_0200-if00

~$ dmesg | grep hidraw  
[    2.923925] hidraw: raw HID events driver (C) Jiri Kosina
[    2.987907] hid-generic 0003:1A86:E024.0001: hiddev0,hidraw0: USB HID v1.00 Device [HID 1a86:e024] on usb-0000:00:06.0-4/input0
[    3.940468] appleir 0003:05AC:8242.0002: input,hiddev1,hidraw1: USB HID v1.11 Device [Apple Computer, Inc. IR Receiver] on usb-0000:00:06.0-5/input0
[    5.224273] hid-generic 0003:05AC:820A.0003: input,hidraw2: USB HID v1.11 Keyboard [HID 05ac:820a] on usb-0000:00:06.0-6.1/input0
[    5.459073] hid-generic 0003:05AC:820B.0004: input,hidraw3: USB HID v1.11 Mouse [HID 05ac:820b] on usb-0000:00:06.0-6.2/input0

Wyze USB dongle is solid blue.

Word of warning: I'm not very familiar with Docker. It's running/working, but it confuses the heck out of me. I'm happy to try to issue Docker commands, but I'll need some firm hand-holding.

Logs GREPd for any mention of wyze are in the accompanying .zip file.

Any suggestions on how to get things back up and running?

wyze_log.txt.zip

RonSpawnson commented 4 years ago

I'm also frequently experiencing this. Happens once or twice per week

ghost commented 4 years ago

I'm also experiencing this about 1-2 times a week. I have 9 contact sensors and 3 motion sensors paired with it. Restarting HA doesn't fix it, removing/reinserting the dongle doesn't fix it. Only rebooting the whole host fixes it.

Running Hass.io installed on Ubuntu LTS.

johnwarne commented 4 years ago

@johnwarne See this information on this error in virtualbox here: #20 (comment) Basically it involves switching your virtual machine to use a different (more stable) usb host controller This is unfortunately an issue where the OS drops the connection with the USB device.

Between switching to the USB 3.0 host controller and going back to 0.0.7 of the component, this seems to have solved my stability problems for the past several days (knocks on wood). I'd gone down to USB 2.0 since that helped with stability in the past, but perhaps with newer versions of the component that's not applicable. Thanks, @kevinvincent !

RoldyBuffalo commented 4 years ago

So what is the solution to this? or a workaround perhaps? I reboot my host (not hass) and it fixes the issue. I also figured out reinstalling the component and rebooting hass (not host) fixes it as well.

Would really like to know how I can prevent my usb dongle from disconnecting every time supervisor updates, or worse at random like it seem to be doing about once every 2-3 days without manually rebooting the host. I run a vpn tunnel from my host as well (I know, I know, I need at least one more pi) but it's lightweight in the essence the ssh just needs to be kept open to keep the tunnel open. Nifty little failsafe.

Anyway, I am experiencing all the issues described in this thread. The dongle disconnecting, not reconnecting, and the sensors remaining in their off position. This is only resolved with the methods I mentioned. Is using usb 3.0 instead of 2.0 the trick? Curious if it is, as on my rpi4b it happens with both usb 3.0 and usb2.0 BUT I am using powered usb3.0 hub as instructed by several guides for rpi4b and usb's. Something about their power circuitry only having a set amount of voltage across the usb lanes. Google it if you're curious.

cheechie commented 4 years ago

So what is the solution to this? or a workaround perhaps? I reboot my host (not hass) and it fixes the issue. I also figured out reinstalling the component and rebooting hass (not host) fixes it as well.

Would really like to know how I can prevent my usb dongle from disconnecting every time supervisor updates, or worse at random like it seem to be doing about once every 2-3 days without manually rebooting the host. I run a vpn tunnel from my host as well (I know, I know, I need at least one more pi) but it's lightweight in the essence the ssh just needs to be kept open to keep the tunnel open. Nifty little failsafe.

Anyway, I am experiencing all the issues described in this thread. The dongle disconnecting, not reconnecting, and the sensors remaining in their off position. This is only resolved with the methods I mentioned. Is using usb 3.0 instead of 2.0 the trick? Curious if it is, as on my rpi4b it happens with both usb 3.0 and usb2.0 BUT I am using powered usb3.0 hub as instructed by several guides for rpi4b and usb's. Something about their power circuitry only having a set amount of voltage across the usb lanes. Google it if you're curious.

I have not seen a solution yet. I am running it in a VM in proxmox. I have to power of my HA VM and then uplug my USB dongle and then plug it back in and reboot. I might go back to hassio on the pie, because then I just had to reboot and the sensors would come back online.

I sure hope this issue can be resolved, otherwise I will have to look for other sensors to use instead of wyze sensors.

tteck commented 4 years ago

I updated the bridge firmware from .30 to .33 fixed the problem for me.

eregev commented 4 years ago

@tteck, upgraded by plugging back into a wyze cam? I switched to the 3.0 VirtualBox driver and haven't had issues yet (although it often takes between 1-5 days for the issue to show back up).

tteck commented 4 years ago

@tteck, upgraded by plugging back into a wyze cam? I switched to the 3.0 VirtualBox driver and haven't had issues yet (although it often takes between 1-5 days for the issue to show back up).

Yes, I had to purchase a Wyze cam (also picked up a spare starter kit) to see what firmware was on my bridge. Firmware was .30 updated to .33 on the new bridge (kept the old bridge at .30 just in case) re-scanned my sensors (20) It's been 3 weeks with no problems.

ghost commented 4 years ago

Interesting. I've never even plugged the Sense Bridge into a Wyze Cam. Will give it a shot and see if the firmware update fixes it!

cheechie commented 4 years ago

I updated the bridge firmware from .30 to .33 fixed the problem for me.

I just updated my bridge firmware from 0.30 to 0.33. My HA would not recognize any of my sensors with the new firmware. I had to add all 24 of my wyze sensors again buy scanning and pushing the button with the pin. Hopefully this works!

The firmware update is in camera settings ->Device Info ->Bridge Firmware Version

swingking commented 4 years ago

@kevinvincent, my Wyze component stopped responding this afternoon and I have the debug logs. I've uploaded the relevant section to pastebin: https://pastebin.com/jiDDv4Cs

The wyze logs just seem to stop at a "Trying to parse:" message. FYI, the LED on the dongle is still blue. My system is a Ubuntu 18.04 system with HA running in a Docker container.

[Edit] I probably should mention that the system was running for almost exactly three days before it stopped handling events.

Thanks!

RoldyBuffalo commented 4 years ago

ugh, if I must update my firmware, I suppose that's the fix. No bother other then re-pairing my sensors. I will report once the firmware is updated on progress.

Edit

Curiously, when I plugged the bridge into the camera, it either auto updated almost instantly, or it was already on 0.33 firmware for the bridge. I'm willing to bet the former. I will attempt to run the system for a few days without reboot, maybe the update (or not) fixed it?

Edit_2

Alright, it seems like it DID in fact update the bridge without me manually doing it once I plugged it into the camera and rebooted the camera, it applied the update within seconds. Secondly, the sensors all were unresponsive when I plugged it back into hass (which is why I think it updated,) I ran the wyzesense_scan call for every sensor (it did take about 20 minutes to go through them all) 16 sensors in total. The sensors are now responding normally, only time will tell if this solved the disconnect issues. Thanks for the tips!

bschatzow commented 4 years ago

I do not have a camera as I purchased the sensors to use with HA. So far I have not had the disconnect Issues as others above. Is there a way to update firmware (if needed) without the camera?

RonSpawnson commented 4 years ago

My bridge firmware is stuck on updating and never finishes. I'm thinking maybe it's because I have rstp firmware on camera. Anyone been able to successfully update firmware via camera with official rstp firmware?

I also second the question on whether there is any way to update bridge firmware without camera.

RoldyBuffalo commented 4 years ago

My bridge firmware is stuck on updating and never finishes. I'm thinking maybe it's because I have rstp firmware on camera. Anyone been able to successfully update firmware via camera with official rstp firmware?

Try to open the wyze app, open the front page with all your cameras, and toggle off for 10 seconds, then on the camera from the power button on that page. once it boots all back up, you should see your bridge with the new firmware. I am running rtsp and mine updated successfully.

I also second the question on whether there is any way to update bridge firmware without camera.

I do not know of another way to do this, but either of the cameras work. They are not terrible cameras, I've used mine in a pinch when we lost our cat. I placed the camera outside in pan-mode with the litter box, opened it up in hass and watched for 3 hours till she came back. Without the camera idk what I would have done, it was freezing out. I know, specific case, but for 20 dollars and the ability to add rtsp firmware (even though it needs updated) and include it into hass, but really my cameras are place holders until we get a more stable, and robust system in place. I originally had the bridge hooked up to the camera, and it piped into alexa with some magic with smart things. This was gross, so I wanted a system to integrate it into alexa, without alot of b.s. DrZzzz's video got me started. 6 months later I'm damn near completely local control now. Couldn't have done more then half my automations without these freaking awesome little sensors. motion lighting, door chimes, security, mail call, they are the backbone of my system. I have faith that this update, hopefully fixed the issues I was having.

I will say as a final word RonSpawnson. My bridge didn't update immediately, I HAD to reboot the camera with the bridge installed for it to finish the update. you can check the current version in the app in the firmware page, check once it reboots it might update on it's own like mine did.

don't forget this as well

...Secondly, the sensors all were unresponsive when I plugged it back into hass (which is why I think it updated,) I ran the wyzesense_scan call for every sensor (it did take about 20 minutes to go through them all) 16 sensors in total. The sensors are now responding normally, only time will tell if this solved the disconnect issues. ...

RonSpawnson commented 4 years ago

Thanks @VarenDerpsAround - I was able to finally upgrade my bridge firmware. To give some additional information in case anyone else runs into the issue I did - I tried your suggestions and tried multiple different sequences of upgrading, resetting, power cycling in app and physically, and none worked.

To resolve - I restored my camera to the latest non-rstp firmware (v4.9.5.98) and then the bridge firmware upgrade finally worked for me right away! For reference, the rstp firmware version I was using which was not allowing upgrades was v4.28.4.41.

I'll monitor my situation just like the others who recently upgraded the bridge firmware. If any of us see this issue crop up again, let's share that information in this issue. If a sufficient amount of time goes by without any of us experiencing the issue (1-2 weeks), I believe this confirms that this issue can be resolved via firmware upgrade, and perhaps the README can be updated with this information which can close this issue. Let's stay tuned for a couple weeks to confirm.

cheechie commented 4 years ago

I updated the firmware on my bridge 3 days ago, and it has been working flawlessly since then. I've had no issues, I hope it continues.

cheechie commented 4 years ago

I updated the firmware on my bridge 3 days ago, and it has been working flawlessly since then. I've had no issues, I hope it continues.

Well it all stopped working this morning. Upgrading the firmware for the bridge did not fix the issue.

tteck commented 4 years ago

Strange, you and I are both running Proxmox and mine is still going strong. Nearly 4 weeks. I guess that I just got lucky, and that never seems to happen.

In proxmox for my USB pass through I used USB Vendor/Device ID and selected 1a86:e024 instead of USB port I also checked use USB3

Hate that it didn't fix your issue.

PeterPlatt commented 4 years ago

I'm seeing this issue now. I picked up a refurbished Wyze cam. I'm attempting the upgrade. Current Bridge version is 0.0.0.30 upgrading to 0.0.0.33. I'll post later if this works as well.

RonSpawnson commented 4 years ago

Sad to report that I also experienced the sensors not responding issue again this morning even though I updated my bridge to version 0.33 a few days back. Running on raspberry pi 4. Seems there is still another issue at play here sadly.

cheechie commented 4 years ago

Strange, you and I are both running Proxmox and mine is still going strong. Nearly 4 weeks. I guess that I just got lucky, and that never seems to happen.

In proxmox for my USB pass through I used USB Vendor/Device ID and selected 1a86:e024 instead of USB port I also checked use USB3

Hate that it didn't fix your issue.

That's exactly my settings.
usb

PeterPlatt commented 4 years ago

I'm seeing this issue now. I picked up a refurbished Wyze cam. I'm attempting the upgrade. Current Bridge version is 0.0.0.30 upgrading to 0.0.0.33. I'll post later if this works as well.

Well, the upgrade helped and now I got a new Wyze cam toy. My setup is a Rasberry pi 4 and I have about 20+ sensors registered. I had to go around to each one and re-pair. No other changes needed. I also setup the reboot every night just for safe measure.

mconlow commented 4 years ago

Is there an understanding of why sensors lose connection to the bridge, or why the bridge loses connection to the OS? Maybe this is my lack of understanding of how this add on works, but I'd think that once it is working, if you don't update any of the firmwares, it would keep working?

Like others I have a camera that has been flashed to the rtsp firmware so it will be a bit of a chore to get it back on standard firmware, move the bridge onto it, update the firmware, then flash back to the rtsp.

tteck commented 4 years ago

I hope everyone will give us a recap of the status of their Wyze Sense integration since updating their firmware. For me, it's been running for over a month with no problems. The only restarts are because of core updates.

ghost commented 4 years ago

So far so good for me. ~19 days without a hiccup.

RonSpawnson commented 4 years ago

I want to update with my experience. After firmware update it worked correctly for me for 4 days, after which I thought it broke. Since then it has been chugging along perfectly for another 13 days. I'm beginning to lose confidence that my issue after first 4 days was actually a repro of the sensor issue or something else entirely.

At this point I can say that the firmware update has either fixed it entirely for me, or at the very least has significantly lowered the frequency (used to occur at least once or twice a week for me). I'll update again if I experience a repro of the sensor loss issue. If I do not make any further comments, it has continued working flawlessly for me.

cheechie commented 4 years ago

I hope everyone will give us a recap of the status of their Wyze Sense integration since updating their firmware. For me, it's been running for over a month with no problems. The only restarts are because of core updates.

Ever since I updated my firmware, I've been running for 13 days without issue.

Phara0h commented 4 years ago

I'm running 0.33 and was running fine for about 3 hours then stoped updating.

Phara0h commented 4 years ago

Here is a the messages up to the point it stopped working.

custom_components.wyzesense.wyzesense_custom] <=== Received: Packet: Cmd=5332, Payload= 2020-04-01 03:54:15 DEBUG (Thread-2) [custom_components.wyzesense.wyzesense_custom] ===> Sending: Packet: Cmd=53FF, Payload=ACK(5332) 2020-04-01 03:54:15 DEBUG (Thread-2) [custom_components.wyzesense.wyzesense_custom] Sending: b'aa555332ff0283' 2020-04-01 03:54:15 DEBUG (Thread-2) [custom_components.wyzesense.wyzesense_custom] ===> Sending: Packet: Cmd=5333, Payload=b'0000017134bbce16' 2020-04-01 03:54:15 DEBUG (Thread-2) [custom_components.wyzesense.wyzesense_custom] Sending: b'aa55530b330000017134bbce1603d5' 2020-04-01 03:54:15 DEBUG (Thread-2) [custom_components.wyzesense.wyzesense_custom] Trying to parse: b'55aa5333ff0284' 2020-04-01 03:54:15 DEBUG (Thread-2) [custom_components.wyzesense.wyzesense_custom] Received: b'55aa5333ff0284' 2020-04-01 03:54:15 DEBUG (Thread-2) [custom_components.wyzesense.wyzesense_custom] <=== Received: Packet: Cmd=53FF, Payload=ACK(5333) 2020-04-01 03:54:35 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.1830159120] Client exceeded max pending messages [2]: 512 2020-04-01 04:38:48 WARNING (Thread-5) [pychromecast.socket_client] [Living Room:8009] Heartbeat timeout, resetting connection 2020-04-01 04:39:13 ERROR (Thread-5) [pychromecast.socket_client] [Living Room:8009] Failed to connect to service Chromecast-Ultra-5e1a971623ba139ab40c72085277cbec._googlecast._tcp.local., retrying in 5.0s

Phara0h commented 4 years ago

@kevinvincent debugged it further, called wyzesense.scan when it stopped working and got this error.

Logger: homeassistant.components.websocket_api.http.connection.1814187792 Source: custom_components/wyzesense/wyzesense_custom.py:106 Integration: websocket_api (documentation, issues) First occurred: 10:46:02 PM (2 occurrences) Last logged: 10:46:06 PM

[Errno 32] Broken pipe Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 134, in handle_call_service connection.context(msg), File "/usr/src/homeassistant/homeassistant/core.py", line 1234, in async_call await asyncio.shield(self._execute_service(handler, service_call)) File "/usr/src/homeassistant/homeassistant/core.py", line 1261, in _execute_service await self._hass.async_add_executor_job(handler.func, service_call) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/config/custom_components/wyzesense/binary_sensor.py", line 151, in on_scan result = ws.Scan() File "/config/custom_components/wyzesense/wyzesense_custom.py", line 530, in Scan self._DoSimpleCommand(Packet.EnableScan()) File "/config/custom_components/wyzesense/wyzesense_custom.py", line 393, in _DoSimpleCommand self._DoCommand(pkt, cmd_handler, timeout) File "/config/custom_components/wyzesense/wyzesense_custom.py", line 379, in _DoCommand self._SendPacket(pkt) File "/config/custom_components/wyzesense/wyzesense_custom.py", line 331, in _SendPacket pkt.Send(self.__fd) File "/config/custom_components/wyzesense/wyzesense_custom.py", line 106, in Send ss = os.write(fd, pkt) BrokenPipeError: [Errno 32] Broken pipe

RonSpawnson commented 4 years ago

Update - I unfortunately indeed can confirm this happened for me again, even with 0.33 firmware. I have not restart it, so if there's anything I can do to help diagnose the issue while I'm in this state, please let me know.

RyanWoirol commented 4 years ago

Mine survives anywhere from 30 minutes to a few hours before I need to restart but sometimes reinstall the service. I can always tell when it goes out because the LED goes from blue to yellow.

tteck commented 4 years ago

Mine survives anywhere from 30 minutes to a few hours before I need to restart but sometimes reinstall the service. I can always tell when it goes out because the LED goes from blue to yellow.

I think you may have a different problem. The issue we're discussing, the led stays blue. At least mine would.