2.9.0 suddenly causes poor connection to cameras

[vipergts450]

Hello,

I upgraded to 2.9.0 yesterday and suddenly the connections from the bridge to my two cameras is very unstable. The log has these messages every couple of seconds. I have ON_DEMAND=False set in my config but that did not help. Nothing about my configuration changed since the prior version which had been rock solid.

Possible related to this person's issue: https://github.com/mrlt8/docker-wyze-bridge/issues/1186#issuecomment-2106408697

2024-05-13T16:13:03.196702946Z  * Serving Flask app 'frontend'
2024-05-13T16:13:03.196970160Z [WyzeBridge] 📚 Using 'auth' from local cache...
2024-05-13T16:13:03.197045726Z  * Debug mode: off
2024-05-13T16:13:03.198131787Z [WyzeBridge] 📚 Using 'user' from local cache...
2024-05-13T16:13:03.199037258Z [WyzeBridge] 📚 Using 'cameras' from local cache...
2024-05-13T16:13:03.199659505Z [WyzeBridge] 🪄 FILTER ALLOWING: 2 of 3 cams
2024-05-13T16:13:03.200873864Z [WyzeBridge] [+] Adding cam1 [WYZE_CAKP2JFUS]
2024-05-13T16:13:03.203321704Z [WyzeBridge] WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
2024-05-13T16:13:03.203474466Z  * Running on all addresses (0.0.0.0)
2024-05-13T16:13:03.203525782Z  * Running on http://127.0.0.1:5000
2024-05-13T16:13:03.203571620Z  * Running on http://172.23.0.2:5000
2024-05-13T16:13:03.203615671Z [WyzeBridge] Press CTRL+C to quit
2024-05-13T16:13:03.215488053Z [WyzeBridge] [+] Adding cam2 [WYZECP1_JEF]
2024-05-13T16:13:03.216297232Z [WyzeBridge] starting MediaMTX 1.1.1
2024-05-13T16:13:03.217069413Z [WyzeBridge] 🎬 2 streams enabled
2024-05-13T16:13:04.205027178Z [WyzeBridge] 172.23.0.1 - - [13/May/2024 16:13:04] "GET /api/sse_status HTTP/1.1" 200 -
2024-05-13T16:13:04.218120477Z [WyzeBridge] 🎉 Connecting to WyzeCam V3 - cam1 on 192.168.2.52
2024-05-13T16:13:04.245859903Z [WyzeBridge] 🎉 Connecting to WyzeCam Pan - cam2 on 192.168.2.50
2024-05-13T16:13:06.272797673Z [cam1] 📡 Getting 180kb/s HD stream (H264/20fps) via LAN mode (WiFi: 92%) FW: 4.36.11.8391 🔒
2024-05-13T16:13:06.273624367Z [cam1] 🔊 Audio Enabled - ALAW > AAC/16,000Hz
2024-05-13T16:13:06.351398248Z [cam2] 📡 Getting 180kb/s HD stream (H264/15fps) via LAN mode (WiFi: 97%) FW: 4.10.9.3006 🔒
2024-05-13T16:13:06.352678972Z [cam2] 🔊 Audio Enabled - MULAW > AAC/8,000Hz
2024-05-13T16:13:06.370007139Z [cam1] WARNING: Skipping wrong frame_size at start of stream [frame_size=1]
2024-05-13T16:13:06.446627324Z [cam2] WARNING: Skipping wrong frame_size at start of stream [frame_size=1]
2024-05-13T16:13:06.980112380Z [WyzeBridge] ✅ '/cam2 stream is UP! (3/3)
2024-05-13T16:13:07.438466197Z [WyzeBridge] 📖 New client reading from cam2
2024-05-13T16:13:08.033675802Z [WyzeBridge] ✅ '/cam1 stream is UP! (3/3)
2024-05-13T16:13:08.153190291Z [WyzeBridge] 📖 New client reading from cam2
2024-05-13T16:13:08.448102057Z [WyzeBridge] 📖 New client reading from cam1
2024-05-13T16:13:08.615706720Z [WyzeBridge] 📕 Client stopped reading from cam2
2024-05-13T16:13:09.649123751Z [WyzeBridge] 📖 New client reading from cam1
2024-05-13T16:13:10.017194505Z [WyzeBridge] 📕 Client stopped reading from cam1
2024-05-13T16:13:10.422995827Z [WyzeBridge] 📖 New client reading from cam1
2024-05-13T16:13:11.232538143Z [WyzeBridge] 📕 Client stopped reading from cam1

Config lines:
version: '2.4'
services:
    wyze-bridge:
        container_name: wyze-bridge
        restart: unless-stopped
        image: mrlt8/wyze-bridge:latest
        ports:
            - 1936:1935 # RTMP
            - 8555:8554 # RTSP
            - 8888:8888 # HLS
            - 8889:8889 #WebRTC
            - 8189:8189/udp # WebRTC/ICE
            - 8002:5000 # WEB-UI
        environment:
            # [OPTIONAL] (Can be set in the WebUI):
            - WYZE_EMAIL=xxxx
            - WYZE_PASSWORD=xxxx
            # [OPTIONAL] IP Address of the host to enable WebRTC e.g.,:
            - WB_IP=192.168.2.4
            - WB_API=xxxx
            # [OPTIONAL] Wyze API credentials:
            - API_ID=xxxx
            - API_KEY=xxxx
            - ENABLE_AUDIO=True
            - AUDIO_CODEC=AAC
            - FILTER_NAMES=cam1, cam2
            - SUBSTREAM=False
            - NET_MODE=LAN
            - ON_DEMAND=False

Could this be a port mapping issue that arose with the latest update? I don't see any log lines that suggest the bridge is losing the feed from the cameras, but seemingly clients connecting externally to the bridge to retrieve the feeds have this constant connect/disconnect problem.

The choppy stream, timeout when fetching snapshot on the WebUI (the auto refresh sometimes pulls a new snapshot image and sometimes does not for 15-20min), and general misbehavior seems to point to something between the front-end and the mediamtx backend, but I can't seem to figure out what exactly.

[1fastt2]

I am experiencing the same thing. I reverted back to 2.8.3 which has been solid. I did confirm I have the same issues with 2.9.1 as well.

[vipergts450]

This doesn't have anything to do with the new FFMPEG option -use_wallclock_as_timestamps I presume? Could this option cause issues with NVRs? Possibly better config option could be -live_flv according to the Frigate forums. However, this is possibly also the entirely wrong rabbithole to go down.

[mrlt8]

Could you try the latest dev build to see if it helps?

@vipergts450 it's a timestamp issue with the raw h264 - the cameras return a timestamp together with the raw h264, but there doesn't seem to be a way to pass that info on to ffmpeg so use_wallclock_as_timestamps seems to help keep everything in sync.

[vipergts450]

On the dev branch as of this morning. Will report back in a bit.

[vipergts450]

It's certainly more stable than yesterday, which is a big help. However, something's still going on with the timestamping or keyframing on my Cam v3 (oddly not happening with my very old OG Cam Pan). Do these cams handle the stream keying differently?

2024-05-14T12:00:09.784351760Z [cam1] [video] super slow
2024-05-14T12:00:09.785032712Z [cam1] WARNING: clear buffer
2024-05-14T12:01:54.293497646Z [cam1] WARNING: Audio pipe closed
2024-05-14T12:01:54.994294824Z [WyzeBridge] 📕 Client stopped reading from cam1
2024-05-14T12:01:55.452390762Z [cam1] [Exception] Did not receive a frame for 39s
2024-05-14T12:01:59.528215199Z [WyzeBridge] ❌ '/cam1' stream is down
2024-05-14T12:01:59.528750219Z [WyzeBridge] 🎉 Connecting to WyzeCam V3 - Cam1 on 192.168.2.52
2024-05-14T12:02:01.743734438Z [cam1] 📡 Getting 180kb/s HD stream (H264/15fps) via LAN mode (WiFi: 93%) FW: 4.36.11.8391 🔒
2024-05-14T12:02:01.744719617Z [cam1 🔊 Audio Enabled - ALAW > AAC/16,000Hz
2024-05-14T12:02:01.844483893Z [cam1] WARNING: Skipping wrong frame_size at start of stream [frame_size=1]
2024-05-14T12:02:02.107081553Z [WyzeBridge] ✅ '/cam1 stream is UP! (3/3)
2024-05-14T12:02:10.485513162Z [WyzeBridge] 📖 New client reading from cam1

Once the stream came back up on its own, it's stable again for the moment. It looks like the stream was up for almost exactly 30 minutes before resetting.

Update: happened again after exactly 30 minutes just as before. So looks like there's a 30 min cycle for resets on the Cam v3 at least, maybe as a counter rolls over? My Cam Pan stream has not reset at all since the container went online.

[mrlt8]

hmmm, thank you for the data point! I wonder if the camera's might be running some kind of clock sync every 30 minutes that might be throwing off our av sync checks....

[vipergts450]

Updating again. Since this morning, the Cam V3 has done the 30 min reset consistently, but it's now fast enough that the NVR it's connected to doesn't "notice" the camera going down every single time (only maybe 3 times total all day did it notice the disconnect and send me a warning email) and quickly resumes when it comes back up. The old Cam Pan has never dropped. So the two firmwares seem to do keyframing/timestamping/time sync a bit differently. This current dev version seems to be a reasonable stopgap fix in the interim as compared to 2.9.0-2.9.1 in my opinion until a more permanent fix is discovered.

[vipergts450]

Update: Stream for the Cam v3 finally went kaput exactly 9 hours, 21min, 25 seconds after starting this morning and did not recover. That's about 33,685 seconds or ~15 min after a signed 16-bit integer counter would roll over past 32768. Not sure if coincidental or related some way how the new FFMPEG counter works. It did not recover again as it had throughout the day and remains permanently down.

2024-05-14T20:31:21.276990825Z [WyzeBridge] ✅ '/cam1 stream is UP! (3/3)
2024-05-14T20:31:29.356919469Z [WyzeBridge] 📖 New client reading from cam1
2024-05-14T20:50:45.169324471Z [cam1] [video] super slow
2024-05-14T20:50:45.170020970Z [cam1] WARNING: clear buffer
2024-05-14T20:51:24.367778977Z [cam1] [video] super slow
2024-05-14T20:51:51.152894206Z [cam1] [video] super slow
2024-05-14T20:52:14.384863744Z [WyzeBridge] 📕 Client stopped reading from cam1
2024-05-14T20:52:15.210762472Z [cam1] WARNING: Audio pipe closed
2024-05-14T20:52:15.774693428Z [cam1] Stream stopped

That's the log from the last "normal" reset I saw all day until the final stream ended.

The Cam Pan (cam2) has stayed up all day and never had a "blip".

[vipergts450]

Today's dev release has kept both streams up since starting up 4 hours ago.

[1fastt2]

I can also confirm that the latest DEV build is solid in regards to the original issue of the thread. I still have zero success with V4 cameras but I don't want to get off topic of this thread.

[mrlt8]

Unfortunately, it seems like the audio seems to drift behind on the dev branch.

[vipergts450]

How about trying the live_flv flag?

This demuxer is used to demux FLV files and RTMP network streams. In case of live network streams, if you force format, you may use live_flv option instead of flv to survive timestamp discontinuities.

Also, this issue with the 13hr 15min bug I mentioned a few days ago. Maybe I did my math wrong when I said 9 hours?

That thread references using -correct_ts_overflow 0 as an input flag potentially.

https://github.com/bluenviron/mediamtx/issues/622

And later in https://github.com/bluenviron/mediamtx/issues/922

Looks like it was fixed in MediaMTX in 0.18.4?

[mrlt8]

Thanks @vipergts450, I will have to take a look at those.

I've added the -re flag for the audio which seems to help keep a stable connection and the audio in sync for a couple of hours now.

Would appreciate any feedback!

[vipergts450]

I am pulling the dev now as of this writing and will report back.

[mrlt8]

@vipergts450 Any feedback on the last dev build?

[vipergts450]

@mrlt8 I have a full day under my belt on this latest dev build and have received no disconnects from the NVRs connected to the stream, and the logs only give the occasional [cam1] [video] super slow. Cam 2 has never been an issue through all these dev builds (older firmware, older camera). I have not paid attention to the audio delays since I don't make heavy use of that, so unfortunately I am a bad data point for this test for audio sync.