search

mrlt8 / docker-wyze-bridge

WebRTC/RTSP/RTMP/LL-HLS bridge for Wyze cams in a docker container
GNU Affero General Public License v3.0
2.54k stars 155 forks source link

Performance issues with more than 3 cams #286

Open CB954 opened 2 years ago

CB954 commented 2 years ago

Anything more than 2 cams I start having performance issues with V3 cams. Anything past 3 cams the Wyze app won't connect to v3 cameras and when the rtsp stream works it is choppy and stops for V3 cams. - The V2 cam keeps working...

I have 3 V3s and 1 V2.

All cams are on the latest firmware.

I have tried: IGNORE_RES=1 NET_MODE=LAN FRESH_DATA=true

mrlt8 commented 2 years ago

Could this be a network/wifi issue?

rugene76 commented 2 years ago

I have 10 cameras and am experiencing the same problem - streams starting and stopping. Unfortunately, it makes the bridge unusable. I also see the start/stop problem in the native Wyze app, so I wonder if they are having a problem on their side. I was hoping to tie the bridge into frigate but it doesn't look that will work due to the flakiness of the camera stream.

CB954 commented 2 years ago

Could this be a network/wifi issue?

I do not think so, as I do not have the issue when I stop the container. I have 1 V2 and 3 V3s. With the container is running, the connection via RTSP is slow and extremely laggy. Interestingly, when the container is running, cameras won't even load in the Wyze app. As soon as I stop this container, the cameras work flawlessly in the Wyze app. It's almost like Wyze throttles how many streams can be running simultaneously. If I limit the container to only two cameras I do not have this issue.

rugene76 commented 2 years ago

I've done some experimenting and have found that more than 4 or 5 camera causes issues in general for wyze cam streams. No matter where they come from. I've also tried their RTSP firmware load and got similar results even with a local stream. I contacted wyze to discuss but they gave the usual "reboot and retry" BS.

Filtering down to one camera helps, but after running a few minutes, I get a "stream is down" with AV_ER_DATA_NOREADY.

I run on a 3 node ASUS ZEN xt8 mesh router so I doubt I have a router issue. Internet connection speed is about 400/50mbs down/up on T-mobile internet. I did however bind each camera to a mesh node to be sure they had the best connection. That might of helped....

In any case, I'm not sure what is causing the connection issues when lot's of cameras are connected. Even with 10 camera's the total bandwidth is modest. I can run multiple TVs streaming netflix 4k, et al, with no problem.

Next up I'm gonna try the dafang hacks on github and see how that works.

Anybody have any ideas?

mrlt8 commented 2 years ago

@CB954 Can you stream from multiple cameras in the app without any issue? + > Add Device Group > Camera Group.

@rugene76 Pretty sure it's the wifi chip/antenna on the cameras that cause it to hog the 2.4ghz channel. WAN speed shouldn't matter if you're streaming over LAN mode. You might also want to turn off some non-standard wifi options on your 2.4 bands like airtime fairness.

I've noticed more connection drops when channel utilization is higher (channel capacity in asus' wifi radar?), which you might be able to check with something like sudo iw wlan0 scan | egrep 'SSID|primary channel|utilisation'

CB954 commented 2 years ago

@CB954 Can you stream from multiple cameras in the app without any issue? + > Add Device Group > Camera Group.

@mrlt8 Yes, no issue streaming all cameras with a camera group. However, as soon I start the bridge, the camera group will no longer stream.

mrlt8 commented 2 years ago

Could you see if v1.2.0 improves/resolves this issue?

CB954 commented 2 years ago

Could you see if v1.2.0 improves/resolves this issue?

@mrlt8 Nope, same issue. It seems to really effect the V3 my V2 will connect while the bridge is running. I see lots of frame drops when running the bridge.

rugene76 commented 2 years ago

I redeployed on docker, thinking I pulled the latest version, and so far things seem much more stable when I read from one camera....but I'm a little confused in that the version listed in the logs says I now have version 1.1.1 Where else would I look on portainer to verify the version?

mrlt8 commented 2 years ago

@rugene76 the version number printed in the log should be most accurate.

rugene76 commented 2 years ago

hmmm, I'm wondering what I did wrong in Portainer. I deleted the stack and redeployed it from scratch. When the container starts up using the env FRESH_DATA=true , I get wyze-bridge ver 1.1.0 and rtsp v0.17.17

RyanEwen commented 2 years ago

I've done some experimenting and have found that more than 4 or 5 camera causes issues in general for wyze cam streams. No matter where they come from. ... I run on a 3 node ASUS ZEN xt8 mesh router so I doubt I have a router issue. Internet connection speed is about 400/50mbs down/up on T-mobile internet. In any case, I'm not sure what is causing the connection issues when lot's of cameras are connected. Even with 10 camera's the total bandwidth is modest. I can run multiple TVs streaming netflix 4k, et al, with no problem.

I also have the XT8 and also had stuttering issues in the Wyze app itself. It turned out to be WiFi Agile Multiband and Protected Management Frames on the 2.4Ghz band. Once disabled, things cleared up quite nicely in the Wyze app. I can stream 4+ cams at once no problem (I have 6 cams).

However, wyze-bridge did not benefit from this. The feeds are still unusable for me. Seems like I get a frame every 2 seconds, if I can stay connected.

Edit: Filtering down to even a single cam doesn't seem to help.

rugene76 commented 2 years ago

Ryan, thanks for the info, I'll try it out. In general I'm experiencing quirkiness with the xt8 on other things so I appreciate your insights.

RyanEwen commented 2 years ago

Ryan, thanks for the info, I'll try it out. In general I'm experiencing quirkiness with the xt8 on other things so I appreciate your insights.

FWIW I'm using the latest beta firmware: 9.0.0.4.386_46980

tetris-rom commented 2 years ago

Seems like I’m running into the same problem. Anything over 4 cameras on, and I’m seeing this in the logs for most cameras.

2022/06/12 11:15:30 [Camera] WARNING: Frame not available yet 2022/06/12 11:16:02 [TSP] [Camera] Client stopped reading 2022/06/12 11:16:07 [RTSP][Camera] New client reading 2022/06/12 11:16:08 [TSP] [Camera] Client stopped reading 2022/06/12 11:16:08 [TSP] [Camera]*'/camera' stream is down 2022/06/12 11:16:09 [Camera] Stream did not receive a frame for over 20s

Troubleshooting done:

Tested using following models:

Is there a way to ignore cameras that we don’t want to have rtsp enabled for? This would make the container usable with rtsp streams for 4 cameras, until this bug is solved.

cheme75 commented 2 years ago
  • Able to load and see all 6 streams at one time

I have 2 beta db cams I’m ignoring by just using the filtering as shown in the instructions for the env variables - I’m listing just the 2 cams and db I want to include.

tetris-rom commented 2 years ago

Thank you for calling that you are able to filter by cameras. I missed that section when skimming through the read me. I’m able to filter for the four cameras that I want/need most and see the streams. It would still be good to see if the bug can be fixed to allow more than 4 cameras to be added.

McCloudS commented 1 year ago

Was pulling out my hair with this exact same issue. Even a single camera with Wyze Bridge running would kill the Wyze app completely. Turn off Wyze Bridge, everything worked fine.

I had replaced my router and it had by default enabled QoS and some sort of multimedia prioritization preference. Similar to the comments above, make sure you don't have any QoS type settings enabled. Once that was disabled, 5+ cameras with no delay or choppiness.

gitloon commented 1 year ago

I'm getting this as well, but i try to filter out various cameras and now its happening with just 2 cams even. Getting that No frame for 20s issue. Thing is this was all working fine for me, then I had to reformat my Windows 11 machine. Resetting up and now I have this issue making it unusable which gives me a sad cause man what a cool thing this was for my cams. Not sure if its wins11 or my machine.. or i didn't have latest perhaps before the reformat. I want to go back in time lol. it was working so well. If anyone comes up with any other ideas please do comment!

gitloon commented 1 year ago

I have to go down to simply 1 cam for it not to disconnect. it is steady as a rock with only one cam.. :(

gitloon commented 1 year ago

Nevermind, that just reconnected too.. no frame for over 60s (i bumped up from default 20s) it lasted longer than with 2 or more. just not sure what the issue is at this point. the Wyze app doesn't stutter at all.. will set up Blue Iris shortly again, (everything was working before until i reformatted and reinstalled win11, as noted above) and see if it has problems as well.

RyanEwen commented 1 year ago

I don't think it matters how many cameras are enabled in the bridge. It seems to be an issue depending how many cameras are on the network. Since I got more than 4, I cannot use the bridge at all. :(

mrlt8 commented 1 year ago

@gitloon is your host running the bridge connected over wifi? Is there another machine you could test the bridge on?

@RyanEwen Could be a router issue? Might want to double check the setting on your router - IGMP, multicast routing, airtime fairness, etc.

gitloon commented 1 year ago

hrm, i didn't try completely disconnecting a cam.. i'll try that just to confirm. I too have 4 cams going.

Well even with 3 cams going I am seeing this issue. Not sure whats up.

@mrlt8 It is not on wifi it is hardlined with a 1G connect to my router. Like i said, i half blame something here, because i don't know what is different now vs what I was using successfully for months before the reformat. :/ only thing i can think of was perhaps i wasn't using "latest" but I've tried downgrading to no avail.

RyanEwen commented 1 year ago

@mrlt8 I do suspect it's unrelated to wyze-bridge.

Here's my WiFi settings: https://imgur.com/a/Izcst9v

Earlier in this thread I mentioned that I had issues even in the Wyze app until I turned off WiFi Agile Multiband and Protected Management Frames. I also turned off Smart Connect so that I could control these settings per-band. At least the app works half decently now.

I am planning on replacing my ASUS XT8s with something else over the winter as I find they are flaky in other ways as well. I'll report back when I've done that in case it makes a difference.

gitloon commented 1 year ago

FWIW Blue Iris is back running, 4 cameras, all connected via RTSP and are doing as well as normal. the 2.4ghz congetstion at my condo is stupid, but here we are :D They are definitely working at least... and as i mentioned before, the wyze app is working. i set NET_MODE=P2P and it still always said LAN mode. is there a way to force it from the wyze api?

mrlt8 commented 1 year ago

@RyanEwen First, I'd try to set the wifi channel to either 1, 6, or 11 - whichever is least congested and has the least overlap with other networks. Other setting should be under the professional tab and I believe LAN/IPTV..?

@gitloon p2p is essentially the same as LAN except it goes over the internet (e.g. streaming direct from the camera from a remote location) and the tutk library will try to go from LAN > P2P > relay, but since your bridge is on the same LAN as the cameras, it's streaming direct from them locally. Might want to take a look at your channel utilization as well, could be some neighbor congesting the channel you're on. Could it be a windows related firewall setting? I believe @cheme75 had some networking issue on windows.

cheme75 commented 1 year ago

Fwiw, I got 2 v2 cams and 2 db cams original install went to relay and nothing I could find in win had any affect to change it. Trying to specify LAN always resulted in errors. I had some glitch when updating docker after windows update/reboot, pc crashed during docker update process, so I uninstalled docker, rebooted, then did a fresh install, then pulled the bridge to reinstall the container and bingo, it came back up in LAN mode as the default and has remained. Weird since nothing else changed.

RyanEwen commented 1 year ago

@mrlt8 the imgur link has a screenshot of the Professional tab as well as the General tab, but I've been playing around with the Professional settings since the screens were uploaded. I've tried those channels and others in the past, but it didn't help unfortunately.

New things I've tried:

I think the Preamble Type and/or Channel Bandwidth changes do actually help, but not enough to stream 4+ cameras at once. Before those changes I couldn't even stream a single one at once, so it's a move in the right direction at least. All 6 cameras appear to have solid connections and the router claims they are all linked at 72 mbps.

Curious how the Wyze app can handle all 6 at once without blinking an eye. You would think that the network traffic etc would be the same?

gitloon commented 1 year ago

So.. i believe in my case using docker-desktop is the culprit. I've switched to just running docker inside a Ubuntu 22 WSL and everything with 4 cameras is smooth as silk. Not sure if docker-desktop has changed recently but that is definitely the issue. Since removing docker-desktop, as a side note, my Blue Iris install is working even better :D (though could be anectdotal.) Thanks to @cheme75 for the idea of uninstalling, which in turn led me to just not use docker-desktop.

mrlt8 commented 1 year ago

@RyanEwen I could be wrong, but I believe 20 Mhz channel width will have less overlap with neighboring wifi. I believe you can scan nearby networks on the WiFi Radar tab to find a good channel for your environment.

These are based on what I have setup for a family member on an AX Asus router streaming 5+ cams:

Wireless > Professional Tab

LAN > IPTV Tab

Is that 6 cameras streaming in the group view?

@gitloon very weird! I've also noticed lots of changes with docker desktop on MacOS, though it seems to function the same as before - at least for now.

RyanEwen commented 1 year ago

@mrlt8 thanks! It's not perfect but my 6 camera feeds are connecting and working through wyze-bridge + webrtc camera card in Home Assistant!

The setting that appears to help the most is Multicast Rate(Mbps): OFDM 6 I'll have to test more to know for sure, but it seems like moment I set that one back to Auto everything gets flaky again.

mrlt8 commented 1 year ago

Awesome! I believe multicast rate will set the minimum rate required before getting kicked off, so you may need to play around with that.

Please let me know if you find any other setting that could potentially help! It might actually be a good idea to add some documentation for some of the router setting that could help.

gitloon commented 1 year ago

with 4 cameras using the sweet sweet web UI (awesome start!) I get some buffering issues and instead of picking back up, it just spins until I fully refresh the browser tab. Is there a setting to allow a longer buffer of some sort? (when i choose HLS on a single camera it shows up just fine, and if it buffers, it picks back up. But when in the 4 camera mode one or two of the cams will end up spinning within like say 30 mins or so.)

This might be a different thread/issue so if so i'll open a new one, but since we were talking about 4 cams and performance...

mrlt8 commented 1 year ago

Do you get the same issue with RTSP? I've noticed videojs will sometimes hang with the HLS stream in certain browsers.

gitloon commented 1 year ago

Hrm, now that you say it, it appears its is pulling from the low latency HLS no matter which button i push on the bottom under each video. I'll dig into the code more and see how to force it (or maybe my browser or adblocks are stopping some functionality) because the only "button" that seems to do anything is the HLS one, which opens a new singular video of just that camera. RMTP and RTSP doesn't seem to change anything (via inspect from chrome).

mrlt8 commented 1 year ago

As far as I know, none of the browsers have native support for RTSP/RTMP, so you'd need to view them in third-party media player or NVR.

gitloon commented 1 year ago

okay, yeah that's making more sense now :D thanks! I'll swing through all the browsers, i do see the videojs now in the inspect window.. so that is likely it. thanks again.

RyanEwen commented 1 year ago

Bit of an update.. I got all 6 streams coming in silky smooth to the WebRTC card in HA / HACS.. although I didn't use wyze-bridge to do it. Instead I used this: https://github.com/gtxaspec/wz_mini_hacks

Now I have RTSP streams coming from each cam itself. That project also allows you to use Wyze cams as USB webcams, SSH into them, RTMP stream from them, run them offline, use wired Ethernet adapters with them, and many other things. Pretty cool!

Best part is that it isn't a firmware and doesn't do anything permanent to the cameras. You just drop the files onto the SD card and reboot. The Wyze firmware and features go unaltered, and you can remove the SD card any time to go back to stock.

digitalpixies commented 1 year ago

Bit of an update.. I got all 6 streams coming in silky smooth to the WebRTC card in HA / HACS.. although I didn't use wyze-bridge to do it. Instead I used this: https://github.com/gtxaspec/wz_mini_hacks

Now I have RTSP streams coming from each cam itself. That project also allows you to use Wyze cams as USB webcams, SSH into them, RTMP stream from them, run them offline, use wired Ethernet adapters with them, and many other things. Pretty cool!

Best part is that it isn't a firmware and doesn't do anything permanent to the cameras. You just drop the files onto the SD card and reboot. The Wyze firmware and features go unaltered, and you can remove the SD card any time to go back to stock.

I believe it's considered a firmware soft mod. Wyze firmware was designed to first boot off of the sdcard and the wz mini hack just adds and modifies a few things. I use it and it works well. I use the "USB gadget" support to get my rpi frigate+home assistant combo working without relying on wifi or extra Ether net dongles

rugene76 commented 1 year ago

I'm gonna have to try the wyze firmware approach - I currently use wyze bridge. What do you mean by "USB gadget"?

btw, I run HA and frigate (7 cameras) on ubuntu with virtualbox for HA and docker for wyze-bridge and frigate.

digitalpixies commented 1 year ago

The Wyze camera (when connected to a pc or router via USB) enumerates itself as a USB cdc. Probably better described on their page. https://github.com/gtxaspec/wz_mini_hacks#features

USB gadget support, connect the camera directly to a supported router to get an internet connection, no USB Ethernet Adapter required, using USB CDC_NCM.

Basically when I connect it to my rpi, I can do a peer to peer tcpip communication. If your having performance issues, using this method of communicating to the Wyze cam should avoid any wifi bandwidth issues while still treating the Wyze as another rstp stream (and not having the need to invest in a USB Ether net poe dongle)