WyzeCam v3 Pro video "lags behind" after awhile...

[carldanley]

Hey again! I've been running the dev branch until your changes get merged in. I noticed this morning that the camera timestamps were hours behind. It almost seems like something is going on where the video is being queued up and sent "slowly"... and after awhile, the timestamps begin to get further and further behind. I mention the timestamps because it's part of the actual video being received.

I can restart the bridge and the streams are instant. An hour goes by, they're behind a couple minutes. A couple hours go by, the streams are now behind 20-30 min, etc.

Hope I'm describing this in a way that makes sense? My first inclination was to check bitrate but I'm not sure. My gut's telling me that it has something to do with the larger video size falling behind but that's a shot in the dark?

Any way I can help, let me know!

[mrlt8]

Hmm, I believe some users had a similar issue with the doorbell when the camera switched between night mode.

Could you try power cycling the camera or disabling the audio in the bridge?

[carldanley]

I tried doing both things, I even added a completely new camera today (another v3 Pro) and made sure everything was setup fresh. Today, video was not lagged after this morning's restart but after awhile - it was actually audio that was lagging behind quite a bit - it seems like the two get out of sync with each other. I'm not really sure how to provide you additional detail that would be useful but anything you need, let me know.

[carldanley]

I was poking around a bit and saw a couple of the old issues regarding FPS for day/night mode. The WyzeCam v3 Pro definitely has 2 different rates; not sure how I can account for this with FIX_FPS. I think this is the issue but I want to test.

I am definitely getting this:

[mrlt8]

hmm, will look into the audio issue.

How is the video delay doing? I wonder if it's similar to that other issue where it only shows up when night vision is active?

[carldanley]

I just checked everything again.

I've tested this on the 4x v3 pro's I have running (have 2 more coming in the mail) and it seems to stand true that when they switch from day/night automatically, they run into the problem and start lagging behind on the Wyze timestamp (and video). If the cameras do not switch from night/day automatically, everything seems to work perfectly.

This makes me feel that the difference in FPS is causing the issue - can I just force the FPS to be 15 in all cases? Even if the camera sends 20, will it be an issue?

[EDIT] I can use the wyze bridge api to /stop and /start the cameras and thing's work perfectly again.

[mrlt8]

You could try FORCE_FPS_CAM_NAME=15 with your cam name. Be sure to comment out FIX_FPS or it will try to use the fps set in the camera parameter.

[carldanley]

Alright, just set them up to force fps, will report back after some time and see if I notice any difference.

[IceDraken]

I am getting the same delay overtime with my Wyze V3's not even pro. After a day, the cameras are off by roughly 15 minutes. If I restart the RTSP simple server... it corrects it.

[mrlt8]

@IceDraken do you know if the delay starts/worsens when switching to night mode?

I believe some people were having issues with insufficient memory causing a delay.

I'll try to see if we might be able to have ffmpeg drop frames to keep up.

The camera also returns the time stamp with each frame, so we may be able to monitor that to see if it starts to fall behind.

[IceDraken]

My homeassistant that i use this in has 10Gigs of Ram... 4 CPU's... virtualized. 64gig HD... it's CPU use is around 5% and the memory is only around 50%... I don't know though if that memory is a good reflection of how much memory is dedicated to the bridge.

What I can do @mrlt8 is confirm at 8PM... it's on the dot. I'll check again at 8AM... I doubt it's night mode, as I want to check again than at 8PM tomorrow as I had a delay this AM and I rebooted it, came home from work around 4:30 and had a delay of 15min. But I'll investigate time frame and see if I can be helpful.

[carldanley]

Checking in - the force FPS env worked to force the FPS in both modes (day/night) but still noticed 2 cameras off (1 by 4 minutes and 1 by 12 minutes). It’s definitely something about swapping day/night

[mrlt8]

@carldanley Is that with the forced fps of 15, and are the other two cameras also falling behind?

I've noticed that even with the same camera model and firmware, some cameras will have a different FPS in the camera settings for some reason.

@IceDraken any FPS mismatch warning in your log? Power cycling the cameras can sometimes help with weird issues like this.

[IceDraken]

@mrlt8 ... a lot of waiting for keyframe... frame not available... all the "relay mode" messages... I don't see FPS mismatch, but i run with the FPX_FIX option.

[mrlt8]

Can you try to run the bridge without FPS fix?

[IceDraken]

@mrlt8 I was having it without the FPS_FIX yesterday... so I enabled it today... if you want, i can turn it off... but it's already night time. I just figured I would leave it on and see how far off it is again at 8AM when it's sunny again.

[carldanley]

Is that with the forced fps of 15, and are the other two cameras also falling behind?

Yes, that's with the forced 15 - I put it on all of the cameras.

[IceDraken]

@mrlt8 8AM, off 15 minutes... FPS_FIX turned off around midnight. all cameras are now throwing: WARNING: FPS param mismatch (avRecv FPS=20)

[IceDraken]

8PM... Off again though @mrlt8 I mean, they did switch to night view already. Do you want me to check in the AM and right before sunset to confirm "day" keeps time properly?

[mrlt8]

I pushed some changes to the dev branch that should reduce the sleep time between frame requests; this was previously added to reduce CPU usage but could be the potential cause for the drift...

Would appreciate it if you guys could test out the dev image and see if this helps.

[carldanley]

Excellent! Will pull latest dev now and give it a go - I was also working on a tool which used OCR to detect the timestamps on the wyze cameras and restart them if there was too much of a lag

[carldanley]

Not sure if this is helpful for you but here are some metrics of latest release versus dev - it doesn't seem like that much more CPU usage for my 4x wyzecam pro v3's:

before:

after:

I, of course, need to wait longer to compare apples to apples but it doesn't really look like any bad jumps going on.

[carldanley]

As I suspected, the increase in CPU isn't actually there - if anything, it seems more stable.

RAM stayed basically the same.

That said, cameras "feel" better, 3 / 4 are within 1 second of each other. 1/4 is behind about 5 seconds.

tl;dr - going to let it run for awhile and will report back tomorrow

[mrlt8]

Awesome, thank you for the data!

[carldanley]

Been about 7 hours and here are some preliminary results:

Only 1 is off by about 4 minutes.

[mrlt8]

Awesome, I'll try to see if there is anything else we might be able to do.

btw, do you still have the forced fps set to 15?

[carldanley]

I do have forced fps set to 15 still; want me to remove it and test for a bit?

Another update, I restarted the one camera that was behind 4 minutes when I posted last time. All cameras were in sync - it's been 14 hours and only 1 camera is behind 2 minutes. Whatever you did is definitely an improvement! Things definitely feel better!

Appreciate your hard work on this ;)

[mrlt8]

Yeah, could you try removing the forced fps and see if it makes a difference?

Is it always the same camera that falls behind?

[mrlt8]

Pushed an update to the dev branch that should drop the frames if the timestamp from the camera is older than 15 seconds. May cause issues if the clock on the camera or bridge are out of sync, so we might have to sync the camera's clock with the bridge on startup.

[carldanley]

see if it makes a difference?

Absolutely, will drop the env var.

Is it always the same camera that falls behind?

No, the first time it was my garage camera (I was thinking it might have been signal strength related). The second time, it was a different camera inside the house that has good RSSI.

Pushed an update to the dev branch

Excellent, will pull this down and test overnight.

[carldanley]

Also, small update, I've had this container running for 35 hours:

This is the current CPU / Memory usage:

It seems you were right that it has increased the CPU load. It did fix a LOT for me; I think it would be worth letting the user opt-in based on an env-variable value; I would 100% opt-in for these changes ;)

[mrlt8]

Thanks for the data! Will try to see if there's anything we can do about that!

Does anyone have any suggestions on benchmarking/monitoring docker resources?

[carldanley]

My entire cluster runs Kubernetes so I can leverage helm charts to deploy a Prometheus stack alongside Grafana - which makes it really easy to get charts like the ones you see above. Without orchestration like Kubernetes, you can still install Prometheus and Grafana -- it will just be a bit more manual for a setup process. This will allow you to scrape metrics over a timeseries and do fun things with the data.

Less exciting, there is always docker stats -- you just won't get anything fancy or historical timeseries data - it's like top or htop.

Not sure if that's any help?

[mrlt8]

Thanks @carldanley! I've been using portainer which has a similar view.

I've been trying to look at what the wyze app does for the sleep interval between frames and it seems like they do something like:

sleep_interval_ms = (1000 / max(fps, 10)) - 10

if product_model == "HL_CAM3P": # V3 Pro
    sleep_interval_ms = sleep_interval_ms + 10

essentially ((1000/20)-10)/1000 = 0.04 which should be the same as 1/(20+5)

Not sure why they're adding back the 10 ms for the v3 pro, but I think we can ignore that. Please let me know if this causes the cameras to drift and or extra latency!

[carldanley]

Update: 23 hours running!

Drift:

I also started wondering if maybe they were out of sync on the Wyze app and could confirm they are pretty much all within < 3 seconds of each other.

The timestamp closest to real time from the ones above was the 23:01:54 one.

CPU:

Memory:

I'll pull latest dev and give it a test!

[carldanley]

For document's purpose, on latest dev, the timestamps were this on restart:

[carldanley]

@mrlt8 you want me try this update? ^

[mrlt8]

That would be great, thanks! It's mostly for the other cameras that are using other FPS, but it's always good to have more data!

Been running it for a while and memory and CPU seem pretty stable for now.

[IceDraken]

is this change only for the V3 Pro's @mrlt8... because mine that are off by like 15 minutes after going from day to night, and night back to day are all just regular V3's.

[mrlt8]

@IceDraken should work for all cameras. Please let me know if the latest changes help!

[IceDraken]

I been looking to see how I can get into the dev branch on the homeassistant add-on, but I am not 100% familiar with HACS yet. Is it easy and I am just missing something?

[mrlt8]

You can add the edge repo which should have the dev branch:

https://github.com/mrlt8/edge-repo

[carldanley]

@mrlt8 Dev branch is spamming 0 quite a bit - not sure if you're using this logging:

[IceDraken]

Finally, got it installed on the HomeAssistant so I can view usages and try it out... Sorry for being a bit new at all this.

[mrlt8]

@carldanley whoops, that was some debug logging for the audio. should be removed now.

[carldanley]

Getting a couple weird issues with latest dev. Container starts up, tried to connect to Wyze camera and then gets to 2/3 and spits out:

Unrecognized option 'fps_mode'.
Error splitting the argument list: Option not found

Then FFMPEG stops and the camera tries to restart. It was working previously - not sure if it's related to something else.

[carldanley]

As a side note, things were running really well with the cameras - things were mostly stable and they were within a few seconds of each other. Going to switch to latest so cameras are back up and running - give me the word for when to test again.

[mrlt8]

Will remove that. ffmpeg was complaining about -vsync is deprecated. Use -fps_mode but I don't think that was helping anyways?

[carldanley]

It's a dev branch for a reason ;) Have at it! I can add the -fps_mode if you need. Either way, there is no rush - totally understand you've got other stuff going on!!

[carldanley]

I can also lock my container to a dev sha - that would probably help.

[mrlt8]

Hey @carldanley could you see if you notice any memory issues on the latest dev branch? CPU seems stable, but I'm noticing a slight memory increase after running it for a couple of days.