Can't remove Buffer underrun

[newburns]

No matter what I do, I cannot get rid of my buffer underrun issue. This is a CentOS x86_64 installation and all cameras are recording 30 fps or more at 704x480 resolution. The setting is an ffmpeg stream on record mode. I tried adjusting the buffer settings of each camera, but to no avail.

[newburns]

All of the cameras do work

[knight-of-ni]

You have 18 cameras, recording continuously, at 30fps. I'd say your hardware can't keep up. Why are you recording at 30 fps??

Temporarily set all your cameras to Monitor mode. Now, put one camera back to record mode and observe the system. If no errors appear then repeat, one camera at a time, until the errors reappear. You have just found the limits of your hardware.

[newburns]

I am in a VM on Proxmox. I have 3 zoneminder installs running. I have more resources I can allocate to ZM, but which resource is it missing? I'm only at 40%-60% CPU usage? It's not a hard drive I/O issue because I'm running ZFS. I give it RAM, but I don't know how to allocate it for use.

htop only shows 1600MB used for RAM.

I have more resources than that to give to this, but I'm not sure what to do with the extra resources.

Also, I'm recording 30fps because it is a famous tunnel in US that we have to keep record of activity for so many days for incidents.

[knight-of-ni]

My guess is your video storage medium cannot keep up. Remember, zoneminder transcodes everything to jpg's, so the disk i/o requirements are that of 18 mjpeg streams, not h264 streams. Follow the test I proposed to determine what the limits are.

I still don't understand why you are running 30fps. Unless you signed a contract with your customer to deliver that then your application does not require it. Have you tried 15fps? It is barely noticeable to the human eye, but tremendously reduces the load on your system. It will also double the number of days of recordings you have, which usually makes the customer happy.

[newburns]

I don't know where your tests are.

Here are some Bonnie++ Results WHILE the 3 Zoneminder servers were running, one of which is 18 cameras @ 30fps.

Initial Bonnie++ test before server deployment showed that I can get above 500MB/s constant. The zoneminder is only pushing 30MB/s max.

Could it be something other than Disk I/O. The NIC and Disk are not close to their limits yet

[knight-of-ni]

Despite the graphs, you are still getting buffer over runs. You are hitting a limit somewhere. Going back to my original suggestion, you need to temporarily scale back the load on the system to determine what the bottleneck is (or isn't). We can't get any further until you collect that data.

[newburns]

OK. I will turn cameras to monitor until there are no overruns. Do I report back how many cameras I can effectively record without overruns? Is that the only data that needs to be collected?

What is the normal camera usage for Zoneminderand at what fps?

[sixdeuce062]

I feel your pain on the 30fps but i usally most i have been able to get is 28 cameras at vga before i hit my harware limit unless you are using a motherboard with QPI or an equivilent also vi box isnt a first choice for hitting hardware limits in my opinion On Dec 4, 2013 4:54 PM, "newburns" notifications@github.com wrote:

OK. I will turn cameras to monitor until there are no overruns. Do I report back how many cameras I can effectively record without overruns? Is that the only data that needs to be collected?

What is the normal camera usage for Zoneminderand at what fps?

— Reply to this email directly or view it on GitHubhttps://github.com/ZoneMinder/ZoneMinder/issues/261#issuecomment-29849637 .

[knight-of-ni]

Don't think of a surveillance system like you would a cpu. If you buy a computer rated at 1 GHz, one can reasonably expect the machine to run continuously at its maximum rated speed. Instead, think of a surveillance system like you would a car.... you might be able to drive your car 100mph, but in nearly all cases you don't ever want to do that, and you certainly don't want to drive it that fast continuously. Yes, it is technically possible to do 30 fps, and there are reasons to do it. However those reasons are few and far between.

Unfortunately, Zoneminder's current recording method is extremely inefficient. For small businesses or home systems, these inefficiencies are a good trade off for getting a nice folder full of easy-to-view jpeg's. A lot of people like that "feature". Once you step into the realm of larger systems with higher resolutions or high frame rates, those inefficiencies really start to show their ugly heads. You are going to have much more difficulty getting a zoneminder system to run that many cameras at 30fps than you would commercial software.

All one needs for any typical surveillance system is 5-7 fps because it only takes 1 frame to catch the incident. Now, you do have moving cars so we can get into bumping the frame rate a little, but lets not worry about that until we get past the buffer overruns.

Back to the issue.... We are trying to figure out what is causing the bottleneck so report back what ever you observe. Generally speaking, you need to scale the system back and then experimentally determine the threshold at which the problem reoccurs. Only after we know the threshold can we talk about what to do about it.

[kylejohnson]

I'm on board with @knnniggett here. I've run many ZFS systems and unless the ARC and L2ARC are properly tuned, you generally end up with a system slower than software raid on linux using mdadm.

Out of curiosity, how much RAM does your ZFS server have, and how many TBs of data is it serving? Are you using a L2ARC device? If so, what is it?

[sixdeuce062]

yes i agree alot of CPU is wasted converting everything to JPEG its great for home and maybe small business but coming from a enterprise angle and even bring enterprise quality to home and business recording at the metadata level (pure non muxed stream) is the best and then converting recorded data as you need is the best. regrettably i have only been able to do this by fat fingering it in at the command line. i am working on scripts for this as the CPU usage is almost non existent per stream (1.3mega h.264 1.5kbps stream 0.3%CPU on a i5 quad core) i was thinking about using some of the original ZM scripts but going to metadata recording is instead of jpeg storage is like writing a new ZM

[kylejohnson]

Take a look at #39

[newburns]

Turns out, I was out of CPU power. Overnight, load would hit the lower 30. I only have 12 cores allocated, so it was hanging with certain spikes. Both RAM and CPU would spike and hang. I have reduce all the cameras to 5fps @ CIF resolution (352x240). I increased the camera count to 31. There is no issue recording now. I can't do too much with the changes of resolutions because this is a production environment. In a week or two, I may be able to bump everything to 15fps and 2CIF resolution. I would like to at least double everything.

I am having a new problem in montage view. Two errors show up constantly

[Thu Dec 05 12:02:28 2013] [error] [client **_] ERR [socketsendto( /var/lib/zoneminder/sock/zms-852916s.sock ) failed: No such file or directory], referer: http://******/zm/index.php?view=montage&group=0

[Thu Dec 05 12:02:28 2013] [error] [client **_] PHP Warning: mysqlpconnect(): Too many connections in /usr/share/zoneminder/www/includes/database.php on line 32, referer: http://******/zm/index.php?view=montage&group=0

[newburns]

Also, I was able to view 18 without a problem. Now I can't view more than 5 or 6

[sixdeuce062]

i always have to restart ZM when i play with large changes like that

[newburns]

@kylejohnson My ZFS system is running "disabled-sync", so there was no need to tune the ARC. I have 24gb RAM and Dual E5520 for processing, but it's only a 6 - 3TB drive - RAID10 (mirror) ZFS, so not a lot of drives for IOPs. Again, this is a virtual hosting environment using Proxmox, so there are 3 Zoneminder systems running at all times, along with some Windows and Linux VMs

@sixdeuce062 This was just after a restart. I've gotten this error quite a bit before. Any way to tackle this? [Thu Dec 05 12:02:28 2013] [error] [client **_] ERR [socketsendto( /var/lib/zoneminder/sock/zms-852916s.sock ) failed: No such file or directory], referer: http://******/zm/index.php?view=montage&group=0

[newburns]

Also, here are my relevant APACHE settings:

StartServers 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 500 MaxClients 500 MaxRequestsPerChild 4000 StartServers 4 MaxClients 600 MinSpareThreads 25 MaxSpareThreads 150 ThreadsPerChild 25 MaxRequestsPerChild 100

[sixdeuce062]

I normally try to avoid testing large scale loads on VM's. Whenever i have that problem It usually fixes itself after a restart of ZM

Dell poweredge 2950 quad core xeon with 16g of ram and using a FSB i get 28 on monitor at 30fps using VGA before its crashes if i delete all of them and add them back in they wont show up till i restart ZM i am guessing its a cache issue some where

[knight-of-ni]

@newburns The 6 camera stream limitation is a client browser limitation and is documented in the FAQ: http://www.zoneminder.com/wiki/index.php/FAQ#I_have_several_monitors_configured_but_when_I_load_the_Montage_view_in_FireFox_why_can_I_only_see_two.3F_or.2C_Why_don.27t_all_my_cameras_display_when_I_use_the_Montage_view_in_FireFox.3F

Note that, last time I tried it, Chrome showed the same symptoms, but I do not know how to change it in that browser.

Fix that first. Then provide us more information surrounding the socket_sendto error. That can be caused by a lot of things so what we need to know is how many cameras can you open up in live view from a single client before you get those errors? none? 1? 6? 12?

[newburns]

This is what I had originally. I changed to 500 to be safe and this is what I can see in Montage with 500 set on Firefox and the httpd.conf from above

This is the /var/log/httpd/error_log that I get

[Fri Dec 06 08:34:01 2013] [error] [client **_] ERR [socketsendto( /var/lib/zoneminder/sock/zms-535273s.sock ) failed: No such file or directory], referer: http://******_/zm/index.php?view=montage&group=0 [Fri Dec 06 08:34:01 2013] [error] [client **] PHP Warning: mysqlpconnect(): Too many connections in /usr/share/zoneminder/www/includes/database.php on line 32, referer: http://******/zm/index.php?view=montage&group=0

I wouldn't need more than 500 max server connections would I? Also, this seems to be a database connection limit, and not an httpd APACHE limitation

[kylejohnson]

@newburns The connection limit is (probably) in the php configuration, not in the mysql configuration.

[knight-of-ni]

I'm going to assume this issue has been resolved and will close. If this is not the case, please re-open and provide new information describing the current state of the system.