Frame send time * msec too slow, throttling maxfps

[Jsalas424]

Describe Your Environment

• v1.34.26
• Installed via PPA
• Ubuntu 18.04 in a Proxmox VM with 3 sockets x 2 cores and 3 GiB of RAM for just the 1 camera
• Firefox

If the issue concerns a camera

• Amcrest IP2M-841W configred with instructions from here: https://wiki.zoneminder.com/Amcrest#IP2M-841B.2FIP2M-841W
• 30fps
• 1920x1080
• ZoneMinder Source Type: RSTP

Describe the bug

Logs are filled with error 'Frame send time * msec too slow, throttling maxfps' when viewing the stream via Firefox or ZMNinja

To Reproduce

Expected behavior

Debug Logs

Level 9 Debug logs for zms_m1: https://pastebin.com/uP63ByXe

[welcome[bot]]

Thanks for opening your first issue here! Just a reminder, this forum is for Bug Reports only. Be sure to follow the issue template!

[Jsalas424]

Updated post with level 9 debug logs for zms_m1: https://pastebin.com/uP63ByXe

[alabamatoy]

I believe this is a result of the browser and the associated internet connection being too slow when viewing the monitor live. The error is not, I believe, between the camera and ZM, but between ZM and the browser client. Please correct me if I am wrong. I see this on mine when one of my users connects from a very slow Internet service.

Also, I always set my Amcrest cameras to 10FPS in the camera setup since I dont need NFL-replay-grade frame rate. 10FPS seems to work quite well and reduces the load on the server dramatically. I have a server very similar in proc power to yours (but mine is not virtualized) running 8 cameras, and it runs around 50% CPU or less.

[connortechnology]

Agreed that is exactly what that message means. However it also includes jpeg encoding time. So if a system was too overloaded it might also spit this out.

Unfortunately I think the logic that we use to deal with this situation is flawed. We divide the max fps by 1.5 which is kindof a lot. Also, we never increase the fps again so a temporary blip will permanently reduce the fps.

We could try reducing the fps by a smaller amount. We could also add another time check that doesn't include the jpeg encoding so as to tell us whether it is general system slowness or bandwidth that are the issue.

[connortechnology]

What is interesting here is that each frame gets progressively slower. WHY?

A better approach might be to scale the image or otherwise reduce the size/complexity of the image. This would save cpu as well as bandwidth.

[alabamatoy]

Also, we never increase the fps again so a temporary blip will permanently reduce the fps

"Permanently" for the session, or truly permanently?

[Jsalas424]

Thanks for chiming in! I monitor this system with Grafana so I can provide figures for CPU loads. Here is the past 30 days of data.

Ignore that big spike in "/" directory storage usage, I forgot to turn off debugging.

Is Zoneminder single threaded or multithreaded? If single threaded than maybe I'm overloading the CPU otherwise these figures do not corroborate that hypothesis.

I'm wondering if this is actually a write issue to disk. These are backed by ZFS volumes on spinning rust, so maybe its a write issue? But then I would expect to see this drop only during writes of my data rather than constantly.

The camera is connected via wifi in the same room with an unobstructed line of site. I'm using a Ubiquiti Dream Machine as my access point so I wouldn't expect the connection to be the issue.

[connortechnology]

@alabamatoy for the session.

This issue would have to be either cpu or bandwidth/latency. We take the time, then encode the jpeg in ram, then print it out to the socket, take the time again. If the amount of time is greater than the desired fps, we decrease the fps so that we wait less between sending frames.

Your graph shows pretty low cpu use and encoding jpeg's is not a time consuming operation. So I'm assuming network problems.

[Jsalas424]

I live in an apartment complex, 2.4GHz interference is definitely a reality and I'm assuming the wireless NIC in that Amcrest isn't the best, but there is only 30ft or so between the camera and the AP. I guess the only way to confirm is to hardwire it. I'll try to do that this weekend and report back.

[Jsalas424]

What I do find odd though is that "sometimes" the bandwidth seems fine as you can see in the graphs I provided. I'm getting 10Mbps at some points and at others it slows to a couple kbps. Maybe I just don't know enough about wireless n

[connortechnology]

as already stated, the issue is not with the camera. It is between your browser and ZM. wifi is inherently unstable. So wiring the camera in will improve your available wifi bandwidth but I suspect the issues will persist.

[Jsalas424]

as already stated, the issue is not with the camera. It is between your browser and ZM. wifi is inherently unstable. So wiring the camera in will improve your available wifi bandwidth but I suspect the issues will persist.

I just reread that and saw the issue in my logic. So would a speed test between my end device (laptop on LAN wifi) and my ZM VM be provide more valuable information?

[Jsalas424]

Or am I being dense here and are you specifically referring to just the browser and not the connection between the end devices and the ZM VM? In that case I should have stated that when I generated these error logs, I was viewing my stream via ZMNinja not through my Firefox browser, although I can recreate this through Firefox if that is of value.

[Jsalas424]

Here I'm connecting from client to Zoneminder VM and getting 73 Mbits/s throughput tested with iperf3

[Jsalas424]

This is solved and was due a 2.4GHz band wifi interference issue

The Amcrest IP2M-841W only uses 2.4GHz making it prone to signal interference is crowded environment, for example my apartment building in a major city.

I noticed that the network adapter for my ZM VM had increased usage right when I went on vacation:

and then tanked when I got back:

My AP (Unifi Dream Machine) broadcasts 3 wifi channels as seperate VLANs. My main and guest channel broadcasts both on the 2.4GHz and 5GHz channels, but only at 2.4GHz for the camera channel. I turned off 2.4GHz on my main wifi channel since I know my main devices all support 5GHz anyways, and it fixed it!