Frequent disconnects - Githubissues

VorlonCD commented 1 year ago

Hello!

So I sit here watching my blueiris screen most of the day and since day one of using the bridge, I've seen frequent disconnections. Maybe every 10-20 minutes.

I realize this may be a Wyze issue.

At first I was just living with it assuming the devices had crappy wifi but now I'm not so sure.

When it happens, both of my Wyze Cam v3's seem to go out at the same time.
If I watch the app at the same time as Blueiris streaming window, the app is still working and showing xx KB/s changing, but if I stream in the bridge web interface it goes out at the same time.
Both have latest firmware, one has 2 bars and the other full 3 bars of Wifi, 15 feet from the router
Issue has persisted through multiple versions of the bridge for the last few months since I first installed
I am on Windows 11 22h2 running the latest version of Docker + all WSL2 updates. (Latest ubuntu, apt update upgrade, etc)
Issue has persisted through a fresh reinstall of Windows and later an uninstall/reinstall of docker/distro/wsl/etc. Delete/recreate docker container and image.
Still happening with 1.9
I even changed network adapter since the issue first started.
I'm not 100% sure of this fact: I had the bridge running on docker+nvidia Jetson, and I believe I noticed the same thing.
No vpn, no 3rd party antivirus/firewall/etc.
It looks like this in the log when it happens:

[WyzeBridge][WARNING][OfficeCam] Stream did not receive a frame for over 20s
[py.warnings][WARNING][SunBackCam] WARNING: Audio pipe closed
[RTSP][OFFICECAM] ❌ '/officecam' stream is down
[RTSP][OFFICECAM] 📕 Client stopped reading
[WyzeBridge][WARNING][SunBackCam] Stream did not receive a frame for over 20s
[WyzeBridge][INFO][WyzeBridge] 🎉 Connecting to WyzeCam V3 - sunbackcam on 10.0.1.234 (1/3)
[WyzeBridge][INFO][WyzeBridge] 🎉 Connecting to WyzeCam V3 - officecam on 10.0.1.189 (1/3)
[wyzecam.iotc][DEBUG][SunBackCam] Connect via IOTC_Connect_ByUIDEx
[wyzecam.iotc][DEBUG][OfficeCam] Connect via IOTC_Connect_ByUIDEx
[RTSP][SUNBACKCAM] 📕 Client stopped reading
[RTSP][SUNBACKCAM] ❌ '/sunbackcam' stream is down
[wyzecam.tutk.tutk_ioctl_mux][DEBUG][SunBackCam] Now listening on channel id 0
[wyzecam.tutk.tutk_ioctl_mux][DEBUG][SunBackCam] SEND <K10000ConnectRequest code=10000 resp_code=10001> <TutkWyzeProtocolHeader prefix=b'HL' protocol=1 code=10000 txt_len=0> b''
[wyzecam.iotc][INFO][OfficeCam] AV Client Start: chan_id=0 expected_chan=0
[wyzecam.tutk.tutk_ioctl_mux][DEBUG][OfficeCam] Now listening on channel id 0
[wyzecam.tutk.tutk_ioctl_mux][DEBUG][OfficeCam] SEND <K10000ConnectRequest code=10000 resp_code=10001> <TutkWyzeProtocolHeader prefix=b'HL' protocol=1 code=10000 txt_len=0> b''
[wyzecam.tutk.tutk_ioctl_mux][DEBUG][SunBackCam] RECV <TutkWyzeProtocolHeader prefix=b'HL' protocol=36 code=10001 txt_len=17>: b'\x03`<9Z^80\x89rl\xa5\xe1\x07\xc6T\x19'
[wyzecam.tutk.tutk_protocol][DEBUG][SunBackCam] Sending response: <K10008ConnectUserAuth code=10008 resp_code=10009>
[wyzecam.tutk.tutk_ioctl_mux][DEBUG][SunBackCam] SEND <K10008ConnectUserAuth code=10008 resp_code=10009> <TutkWyzeProtocolHeader prefix=b'HL' protocol=1 code=10008 txt_len=30> b'\x7f\x1eT,\x08u\xf5n\xd6%\x19l\x1e\x1d\xbc\x10c870\x01\x01\x075511956'
[wyzecam.tutk.tutk_ioctl_mux][DEBUG][OfficeCam] RECV <TutkWyzeProtocolHeader prefix=b'HL' protocol=36 code=10001 txt_len=17>: b'\x03\x1c|\xdb+\xee\x06\xe6\x04-\n\xab\xa0\xdc\xa1\xddB'
[wyzecam.tutk.tutk_protocol][DEBUG][OfficeCam] Sending response: <K10008ConnectUserAuth code=10008 resp_code=10009>

I've attached the full log.

Thanks!

Wyze bridge docker disconnect issue.zip

alan5ab commented 1 year ago

How many cameras are being recorded by blue iris? Move the bridge to a dedicated ubuntu machine and most likely it will work fine.

VorlonCD commented 1 year ago

How many cameras are being recorded by blue iris? Move the bridge to a dedicated ubuntu machine and most likely it will work fine.

6 total cameras, 2 are from the Wyze bridge. I might have to try a dedicated machine, but BI is running on the same machine as Docker and has a 16 core threadripper with 128 gb ram and a really fast nvme drive and a 1080 TI card wtih 12 GB ram - I wouldnt have thought it would be any kind of resource issue having docker on the same machine.

mrlt8 commented 1 year ago

hmm, very strange! Stream did not receive a frame for over 20s means that the bridge is having issues pulling the video frames from the camera. Could this be some kind of interference on the wifi causing the cams to drop temporarily?

VorlonCD commented 1 year ago

I dont think so? Its a beefy router and one of the two cameras is really close with 3 bar signal according to the app. And both feeds go out at the same time. What I thought was an unrelated issue may not be - both docker containers I have completely and permanently loose network connectivity every few days. A restart of docker fixes. Reinstall of docker and WSL2 did not fix, and using a different network card didnt fix.

mrlt8 commented 1 year ago

Any chance you can test the bridge on another machine to rule out any issues with your current setup?

xxsteven69xx commented 1 year ago

Same issue I think

2022-11-23 13:19:57 wyze-bridge | 2022/11/23 19:19:57 [RTSP][PORCH] ✅ '/porch' stream is UP! (3/3) 2022-11-23 13:19:57 wyze-bridge | 2022/11/23 19:19:57 [RTSP][SPICEWOOD1] ✅ '/spicewood1' stream is UP! (3/3) 2022-11-23 13:19:58 wyze-bridge | 2022/11/23 19:19:58 [RTSP][GAURD] ✅ '/gaurd' stream is UP! (3/3) 2022-11-23 13:19:58 wyze-bridge | 2022/11/23 19:19:58 [RTSP][SPICEWOOD2] ✅ '/spicewood2' stream is UP! (3/3) 2022-11-23 13:19:59 wyze-bridge | 2022/11/23 19:19:59 [RTSP][SPICEWOOD1] 📖 New client reading 2022-11-23 13:19:59 wyze-bridge | 2022/11/23 19:19:59 [RTSP][PORCH] 📖 New client reading 2022-11-23 13:19:59 wyze-bridge | 2022/11/23 19:19:59 [RTSP][FRONTYARD] 📖 New client reading 2022-11-23 13:19:59 wyze-bridge | 2022/11/23 19:19:59 [RTSP][SPICEWOOD2] 📖 New client reading 2022-11-23 13:19:59 wyze-bridge | 2022/11/23 19:19:59 [RTSP][GAURD] 📖 New client reading 2022-11-23 13:20:18 wyze-bridge | 2022/11/23 19:20:18 [WyzeBridge] ⏰ Timed out connecting to swbedroom (20s). 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [RTSP][PORCH] 📕 Client stopped reading 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [RTSP][PORCH] ❌ '/porch' stream is down 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [RTSP][FRONTYARD] ❌ '/frontyard' stream is down 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [RTSP][FRONTYARD] 📕 Client stopped reading 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [RTSP][SPICEWOOD1] ❌ '/spicewood1' stream is down 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [RTSP][SPICEWOOD1] 📕 Client stopped reading 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [RTSP][GAURD] ❌ '/gaurd' stream is down 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [RTSP][GAURD] 📕 Client stopped reading 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [RTSP][SPICEWOOD2] ❌ '/spicewood2' stream is down 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [RTSP][SPICEWOOD2] 📕 Client stopped reading 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [porch] Stream did not receive a frame for over 20s 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [frontyard] Stream did not receive a frame for over 20s 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [spicewood1] Stream did not receive a frame for over 20s 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [gaurd] Stream did not receive a frame for over 20s 2022-11-23 13:22:41 wyze-bridge | 2022/11/23 19:22:41 [spicewood2] Stream did not receive a frame for over 20s

mrlt8 commented 1 year ago

Any chance you have have a Ubiquiti network?

VorlonCD commented 1 year ago

@mrlt8 Switching to linux machine fixed. I think it was a docker issue or maybe WSL2 in Windows. There has recently been a big new update for both WSL and Docker for Windows, but I havent tried that new release yet.

No Ubiquiti. But I wonder if there could still be some issue on the wyze-bridge side anyway?

On the Linux machine, the weaker signal camera still goes out maybe once an hour, but it never affects the other camera.

On my windows machine, every time the weaker signal camera went out the OTHER camera would also go out a few seconds later. Could there be some async/blocking issue going on in the code?

heezy01 commented 1 year ago

i'm running Unraid, docker v1.9.2. i have a 5 cam setup, 1 v3 Pro and 4 v3's. the v3 Pro has been running great for me at HD100. i'm currently running the v3's at HD30.

i'm using iSpy 64 on a Windows 10 machine, just to view the rtsp feeds, no recording. i get frequent disconnects on the v3's. it seems like the framerate isn't smooth as it gets a little choppy. it's like the video pauses for a few seconds, then the time on the bottom right of the video speeds up to catch up to the current frame.

wifi is good. i have an orbi mesh system. only one camera has a 2 bar wifi signal in the wyze app. the rest have 3 bars.

2022/11/26 15:11:392022/11/26 15:11:392022/11/26 15:11:39 [RTSP][BACKYARD-2-V3] 📕 Client stopped reading[RTSP][BACKYARD-2-V3] 📕 Client stopped reading[RTSP][BACKYARD-2-V3] ❌ '/backyard-2-v3' stream is down [tee @ 0x55f3b130a680] All tee outputs failed. av_interleaved_write_frame(): Broken pipe 2022/11/26 15:11:41 [Backyard 2 v3] Stream did not receive a frame for over 20s

chris-congdon commented 1 year ago

I'm having this same issue trying to run the container on a Windows 11 machine. It seems like the container just intermittently loses network connectivity, however my host machine connectivity is fine.

Here is from the bridge logs:

wyze-bridge  | 2022/12/04 17:12:33 [RTSP][DISCONNECTED] ❌ '/disconnected' stream is down
wyze-bridge  | 2022/12/04 17:12:33 [RTSP][BACK-YARD-CAM] ❌ '/back-yard-cam' stream is down
wyze-bridge  | 2022/12/04 17:12:34 [RTSP][V3-PRO] ❌ '/v3-pro' stream is down
wyze-bridge  | 2022/12/04 17:12:34 [RTSP][DRIVEWAY] ❌ '/driveway' stream is down
wyze-bridge  | 2022/12/04 17:12:34 [Disconnected] Stream did not receive a frame for over 20s
wyze-bridge  | 2022/12/04 17:12:34 [Back Yard Cam] Stream did not receive a frame for over 20s
wyze-bridge  | 2022/12/04 17:12:34 [V3 Pro] Stream did not receive a frame for over 20s
wyze-bridge  | 2022/12/04 17:12:34 [Driveway] Stream did not receive a frame for over 20s

And I had a ping running from inside the container at the same time:

[1670173710.340992] 64 bytes from 8.8.8.8: icmp_seq=290 ttl=38 time=12.2 ms
[1670173711.346762] 64 bytes from 8.8.8.8: icmp_seq=291 ttl=38 time=17.0 ms
[1670173712.345073] 64 bytes from 8.8.8.8: icmp_seq=292 ttl=38 time=14.2 ms
[1670173714.332756] no answer yet for icmp_seq=293
[1670173715.372841] no answer yet for icmp_seq=294
[1670173716.412841] no answer yet for icmp_seq=295
[1670173717.452887] no answer yet for icmp_seq=296
[1670173718.492908] no answer yet for icmp_seq=297
[1670173718.936724] 64 bytes from 8.8.8.8: icmp_seq=293 ttl=38 time=5605 ms
[1670173718.936810] 64 bytes from 8.8.8.8: icmp_seq=294 ttl=38 time=4604 ms
[1670173718.936904] 64 bytes from 8.8.8.8: icmp_seq=295 ttl=38 time=3564 ms
[1670173718.940055] 64 bytes from 8.8.8.8: icmp_seq=296 ttl=38 time=2527 ms
[1670173718.940142] 64 bytes from 8.8.8.8: icmp_seq=297 ttl=38 time=1487 ms
[1670173718.940278] 64 bytes from 8.8.8.8: icmp_seq=298 ttl=38 time=447 ms
[1670173719.505565] 64 bytes from 8.8.8.8: icmp_seq=299 ttl=38 time=11.6 ms
[1670173720.510184] 64 bytes from 8.8.8.8: icmp_seq=300 ttl=38 time=14.5 ms

I was just pinging google DNS as an example here, but pinging the IP of the camera I see the same drops.

mrlt8 commented 1 year ago

interesting. I'll have to try to replicate the issue on a windows box.

StoneLegion commented 1 year ago

Same issues really strange bug....

64 bytes from 8.8.8.8: icmp_seq=52 ttl=37 time=29.3 ms
64 bytes from 8.8.8.8: icmp_seq=53 ttl=37 time=26.7 ms
64 bytes from 8.8.8.8: icmp_seq=54 ttl=37 time=31.4 ms
64 bytes from 8.8.8.8: icmp_seq=55 ttl=37 time=28.1 ms
64 bytes from 8.8.8.8: icmp_seq=56 ttl=37 time=31.1 ms
64 bytes from 8.8.8.8: icmp_seq=57 ttl=37 time=35.4 ms
64 bytes from 8.8.8.8: icmp_seq=58 ttl=37 time=5342 ms
64 bytes from 8.8.8.8: icmp_seq=59 ttl=37 time=4278 ms
64 bytes from 8.8.8.8: icmp_seq=60 ttl=37 time=3248 ms
64 bytes from 8.8.8.8: icmp_seq=61 ttl=37 time=2215 ms
64 bytes from 8.8.8.8: icmp_seq=62 ttl=37 time=1175 ms
64 bytes from 8.8.8.8: icmp_seq=63 ttl=37 time=135 ms
64 bytes from 8.8.8.8: icmp_seq=64 ttl=37 time=43.9 ms
64 bytes from 8.8.8.8: icmp_seq=65 ttl=37 time=31.2 ms

The High MS where it froze up and did the whole disconnect thing.

2022-12-20 18:13:53 wyze-bridge  | 2022/12/20 23:13:53 [RTSP][GARAGE] 📖 New client reading 
2022-12-20 18:13:53 wyze-bridge  | 2022/12/20 23:13:53 [RTSP][FRONTDOOR] 📖 New client reading 
2022-12-20 18:13:53 wyze-bridge  | 2022/12/20 23:13:53 [RTSP][SIDEDOOR] 📖 New client reading 
2022-12-20 18:13:53 wyze-bridge  | 2022/12/20 23:13:53 [RTSP][BACKYARD] 📖 New client reading 
2022-12-20 18:18:09 wyze-bridge  | 2022/12/20 23:18:092022/12/20 23:18:09 [RTSP][FRONTDOOR] 📕 Client stopped reading
2022-12-20 18:18:09 wyze-bridge  |  [RTSP][FRONTDOOR] ❌ '/frontdoor' stream is down
2022-12-20 18:18:09 wyze-bridge  | 2022/12/20 23:18:09 [RTSP][SIDEDOOR] ❌ '/sidedoor' stream is down
2022-12-20 18:18:09 wyze-bridge  | 2022/12/20 23:18:09 [RTSP][SIDEDOOR] 📕 Client stopped reading
2022-12-20 18:18:09 wyze-bridge  | 2022/12/20 23:18:09 2022/12/20 23:18:09 [RTSP][BACKYARD] 📕 Client stopped reading
2022-12-20 18:18:09 wyze-bridge  | [RTSP][BACKYARD] ❌ '/backyard' stream is down
2022-12-20 18:18:09 wyze-bridge  | 2022/12/20 23:18:09 [RTSP][GARAGE] ❌ '/garage' stream is down2022/12/20 23:18:09
2022-12-20 18:18:09 wyze-bridge  |  [RTSP][GARAGE] 📕 Client stopped reading
2022-12-20 18:18:10 wyze-bridge  | 2022/12/20 23:18:10 [FrontDoor] Stream did not receive a frame for over 20s
2022-12-20 18:18:10 wyze-bridge  | 2022/12/20 23:18:10 [SideDoor] Stream did not receive a frame for over 20s
2022-12-20 18:18:10 wyze-bridge  | 2022/12/20 23:18:10 [BackYard] Stream did not receive a frame for over 20s
2022-12-20 18:18:10 wyze-bridge  | 2022/12/20 23:18:10 [Garage] Stream did not receive a frame for over 20s

chris-congdon commented 1 year ago

Just to update I moved this to an Ubuntu machine and it's been working great. My Windows 11 machine is where my BlueIris server is running so would be great to be able to run from there as well, so may revisit this in the future..

StoneLegion commented 1 year ago

I think I found a solution it might be one or the other...

1) I went from something.15 to something .12 version of docker. 2) when I uninstalled .15 then installed .12 during the options I told it to use the open something virtualization vs I think the windows version?

Either way that ended up solving my problem enough that I'm happy as I'm running out. If someone needs more testing, etc I can do it in the new year. I'm going to guess it was something bugging using the windows virtualization more then the version but who knows...

This all on Windows 11.

mrlt8 commented 1 year ago

Seems to be a similar issue on MacOS with newer versions of docker #626

As @KaneHart confirmed, v4.12.0 seems to be the latest version working of Docker Desktop on Mac/Windows.

juched78 commented 1 year ago

Was seeing similar 20s drop messages and constant restarts inside blue iris. Running all this on a Debian Bullseye LXC container under proxmox. Downgrading to 1.9.2 seems to help, but still seeing some restarts, not close to as as common as with the newer docker container.

Dropped back to 1.8.8 and seeing a dramatic drop in disconnects and timeouts. I will watch for a bit and then go back to 1.10.1 and maybe try 1.9.2 for longer and see where it comes back. Not changing anything else now in my system, same docker, same network, no reboots except for docker container version.

maxfield-allison commented 1 year ago

I'd be very interested to see wifi channel utilization on these. sounds like the same behavior from #241

juched78 commented 1 year ago

Well, things have settled and now 1.10.1 is showing a disconnect on 1 camera in 45 minutes... so much more stable. I did stop the uptime-kuma poll of the web page, as I was wondering if that could be causing a conflict. Now it is just blue iris as a client and is more stable.

So, maybe it was my network/wifi getting calm, or it was the removal of my uptime check. Will monitor now.

VorlonCD commented 1 year ago

@mrlt8 FYI I tried Docker Windows again (v4.15.0) again with bridge v1.10.1. It may have been a little better with how frequently it disconnects but just as it did before it seems to take down dockers internet connectivity within a day.

It may also be a clue that whenever the camera with weaker wifi goes out the one with strong wifi also blinks and comes back before they both come back.

v1.9.0 running directly on a Linux machine was stable. I still saw occasional disconnects on the camera with the weaker wifi connection but it wasnt bringing the other closer camera down at the same time. I just installed v1.10.1 and will try it out for a while.

On Windows, about once a day, I noticed I cant see either of my wyze cameras, and I also have a second container (QBittorrent) that looses its internet connectivity at the same time. A restart of docker fixes for a day or so. Without bridge running at the same time, the other container was up over a week without trouble. I tend to think its a bug with docker windows or maybe the underlying WSL engine. However, when it dies, I CAN go into the WSL Linux console and ping anything I want so it seems to be isolated to docker. Feels like some kind of resource leak that gets worse over time.

mrlt8 commented 1 year ago

Thanks for the feedback! will try to see if something is causing issues with the latest version of docker.

Liang-KS commented 1 year ago

i found this thread by searching the disconnecting issue.

I've fixed mine by removing the wyze key id and api key....using solely the username/password.

the reason why I put the api tokens is I thought in this way I don't need to login with my username/password.

mrlt8 / docker-wyze-bridge

Frequent disconnects #613