search

ZoneMinder / zoneminder

ZoneMinder is a free, open source Closed-circuit television software application developed for Linux which supports IP, USB and Analog cameras.
http://www.zoneminder.com/
GNU General Public License v2.0
4.84k stars 1.19k forks source link

(FreeBSD)(CLANG) Sendfile7 support is broken #3505

Closed abishai closed 1 year ago

abishai commented 1 year ago

I try to build ZM 1.36.19 and after recent changes in https://github.com/ZoneMinder/zoneminder/blob/1.36.19/src/zm_sendfile.h it doesn't compile

/usr/ports/multimedia/zoneminder/work/zoneminder-1.36.19/src/zm_sendfile.h:29:17: error: no matching function for call to 'sendfile'
  ssize_t err = sendfile(in_fd, out_fd, *offset, size, nullptr, &size, 0);
                ^~~~~~~~
/usr/include/sys/socket.h:705:5: note: candidate function not viable: no known conversion from 'size_t *' (aka 'unsigned long *') to 'off_t *' (aka 'long *') for 6th argument
int     sendfile(int, int, off_t, size_t, struct sf_hdtr *, off_t *, int);

I can revert the offending argument to off_t back, but is it a viable solution?

connortechnology commented 1 year ago

Um.. so I think we can get away with just casting ... ssize_t and off_t are basically the same.

So how about this?

#elif HAVE_SENDFILE7_SUPPORT
  off_t sent;
  int err = sendfile(in_fd, out_fd, *offset, size, nullptr, &sent, 0);
  if (err && errno != EAGAIN)
    return -errno;
  return static_cast<ssize_t>(sent);
#else
abishai commented 1 year ago 
--- src/zm_sendfile.h.orig  2022-06-12 09:18:29 UTC
+++ src/zm_sendfile.h
@@ -17,7 +17,7 @@ extern "C" {

 /* Function to send the contents of a file. Will use sendfile or fall back to reading/writing */

-ssize_t zm_sendfile(int out_fd, int in_fd, off_t *offset, size_t size) {
+ssize_t zm_sendfile(int out_fd, int in_fd, off_t *offset, off_t size) {
 #ifdef HAVE_SENDFILE4_SUPPORT
   ssize_t err = sendfile(out_fd, in_fd, offset, size);
   if (err < 0) {

^ My idea was to change argument, but if you want to keep it, casting works as well. I'll apply yours variant then. Thank you!

connortechnology commented 1 year ago

The issue is that we were abusing the stack variable size. Which really.... we shouldn't do. So I think this looks better. Does it compile and work there? If so I'll merge it.

abishai commented 1 year ago

The code compiles, but something is broken in zms. Monitor live view is constantly refreshes.

The logs is filled with

2022-06-18 13:18:57 | web_php |   | 3124 | ERR | Unable to get semaphore. | includes/functions.php | 1907
-- | -- | -- | -- | -- | -- | -- | --
2022-06-18 13:18:57 | zms_m1 |   | 273810 | WAR | Unable to send stream frame: Broken pipe | zm_monitorstream.cpp | 412
2022-06-18 13:18:56 | web_php |   | 3124 | ERR | Unable to get semaphore. | includes/functions.php | 1907

I'm building with DHAVE_SENDFILE=0 Without it I'm receiving zmc crash by signal 11 on startup, but I set this knob before. I was unable to solve this issue since I ported 1.30 to FreeBSD. I remember ZM has a fallback if sendfile is not available. Maybe it's broken now? Or flickering is another regression? It looks like zms sends one frame and than reloads and start to send initial frame one more time and so on.

The last version that works is v1.36.12. Probably, I should try bisect or revert some obvious places to see if it helps.

abishai commented 1 year ago

Probably, it's not zms, but zmc.

Found locked packet when trying to free up video packets. This basically means that decoding is not keeping up.

It looks like zmc provides broken video to all subsystems.

Now I have the following suspicious places:

  1. Semaphore error (looks like this is the source of the problem)
  2. Policy issue. Content Security Policy: The page’s settings observed the loading of a resource at eval (“script-src”). A CSP report is being sent.
  3. Incorrect link generation to zms while behind reverse proxy. http instead of https. I believe this one was solved a long time ago.
abishai commented 1 year ago

Semaphore issue was simple enough - for semaphore support I need php74-sysvsem PHP extension. Stream is working now. On the other side, I see a lot of 

"Found locked packet when trying to free up video packets. This basically means that decoding is not keeping up." warning messages.

and 

Analysis daemon for 8 Monitor-8 needs restarting, time since last analysis 66 seconds (1655563844-1655563778)

for every monitor in Record state (with or without analysis enabled)

"Locked packets" are filling logs very fast. Monitors are ffmpeg 5MP with 6 FPS 6000 VBR in passthrough mode. I see, the severity of such messages was bumped recently from Debug to Warning. Who is not keeping up here?

Analysis issue is the most troublesome - it forces zmc to reload every minute. I think this is not FreeBSD specific problem.

Also, content policy should be relaxed to unblock eval() in js if it's used. 

script-src 'unsafe-eval' 'self'

solves script error.

The problem with http:// link is tricky. I didn't find exact place.

I see the following in the logs:

Setting to streaming: [https://***](https://***)
Loading mixed (insecure) display content “http://***” on a secure page
Image stream has loaded! starting streamCmd for 563970

So, it detects correct URL with https:// but after that tried to use http:// Something very confusing.

connortechnology commented 1 year ago

My experience with 5MP cams is that we cannot decode, and do motion detection on them fast enough.

Yes you do need semaphore support.

Can we please leave this discussion to the issue at hand? Being sendfile implementation. What is the status of that? Is it working with SENDFILE=1?

abishai commented 1 year ago

I agree, should I create separate issues for the problems discovered? Issues with record only monitors is rather serious.

As for sendfile implementation:

  1. With the patch, ZoneMinder compiles.
  2. Without SENDFILE=0 zmc crashes on strartup with signal 11. I think, we need stack trace, right?
badrpc commented 1 year ago

I think sendfile on FreeBSD can only work with sockets. When I built it with sendfile support, I wasn't able to watch recorded events with the following error in logs:

zms_e95.log:12/01/22 22:11:12.762041 zms_e95[101577].WAR-zm_eventstream.cpp/1091 [Unable to send raw frame 1: Socket operation on non-socket 356482 remaining]
zms_e95.log:12/01/22 22:11:12.762191 zms_e95[101577].ERR-zm_eventstream.cpp/730 [Can't send /var/db/zoneminder/events/3/2022-11-29/95/00001-capture.jpg: Socket operation on non-socket]

ERRORS section of the manual lists

     [ENOTSOCK]     The s argument is not a socket.

Here s is the second argument of sendfile.

connortechnology commented 1 year ago

@abishai what is the state of this ? I there something I can merge to master?

abishai commented 1 year ago

@connortechnology With your patch, ZM compiles and works with SENDFILE=0. According @badrpc it looks like SENDFILE must be 0 on FreeBSD.

As for 'supplemental' issues mentioned in https://github.com/ZoneMinder/zoneminder/issues/3505#issuecomment-1159478730 record one is too serious and blocks my migration from 1.36.12 But maybe it was fixed in latest versions of ZM.

Probably, I'm the only user of ZM on FreeBSD, at least nobody complains that we have outdated 1.36.12 in port tree.

I think, we can formally close this issue and hope that latest version of ZM operates better than 1.36.19.

VVD commented 1 year ago

@abishai, I'm installed 1.36.12 today! It have a lot of Deprecated warnings with php 8.1. Begin to update port to 1.36.32, got same build error, create patch, want to create issue… and found this one. :-D

My patch is:

--- src/zm_sendfile.h.orig
+++ src/zm_sendfile.h
@@ -17,7 +17,7 @@ extern "C" {

 /* Function to send the contents of a file. Will use sendfile or fall back to reading/writing */

-ssize_t zm_sendfile(int out_fd, int in_fd, off_t *offset, size_t size) {
+ssize_t zm_sendfile(int out_fd, int in_fd, off_t *offset, ssize_t size) {
 #ifdef HAVE_SENDFILE4_SUPPORT
   ssize_t err = sendfile(out_fd, in_fd, offset, size);
   if (err < 0) {

Let's try together update port!

abishai commented 1 year ago

Is 1.36.32 has deprecated warnings fixed?

Can you check that Record mode works as well? I've experienced watchdog issue for every monitor in Record mode in 1.36.19. But, as I doubt that this one FreeBSD specific, maybe it's fixed in 1.36.32

Analysis daemon for 8 Monitor-8 needs restarting, time since last analysis 66 seconds (1655563844-1655563778)
connortechnology commented 1 year ago

I would expect those to either be fixed, or the issue is outside ZM.

php8.1 deprecation warnings are simply turned off in 1.36.32.

connortechnology commented 1 year ago

So this patch compiles fine here.... I'd be happy to merge it...

hjf commented 1 year ago

I compiled 1.36.32 with this patch and I don't see any issues so far, save for icons not visible in Web UI. Recordings and montage work fine, and no errors in the log. I sent an email to @abishai with the ports tree patch

badrpc commented 1 year ago

@hjf can you watch recorded events?

hjf commented 1 year ago

@hjf can you watch recorded events?

yes when watching as MP4. it acts weird as MJPEG, but i'm not sure if it's my setup.

abishai commented 1 year ago

@connortechnology May I ask you another offtopic question here ? Some of locales in lang looks suspicious. For example, en_gb.php contains LC_TIME=en_GB.utf8 . I think this is incorrect, it should be UTF-8.

abishai commented 1 year ago

nm, it's fixed already.

abishai commented 1 year ago

@hjf Received the patch, will try it. Thanks!

VVD commented 1 year ago

Is 1.36.32 has deprecated warnings fixed?

I don't see them anymore.

Can you check that Record mode works as well? I've experienced watchdog issue for every monitor in Record mode in 1.36.19. But, as I doubt that this one FreeBSD specific, maybe it's fixed in 1.36.32

I'm use Modect with Camera Passthrough.

After I install 1.36.12 and configure 3 cams I got #3414, then added ram 16GB+swap 16GB, but OOM was almost after every alarm. Then it disappeared and I don't know what helped. It work fine for 3 days with a lot of alarms even with 4GB of ram without use of swap. Now I update to 1.36.32 and got same on one cam - every alarm = OOM. Set Maximum Image Buffer Size (frames) to 200 and OOM replaced with messages in log:

Jan 14 18:14:45 cctv zmwatch[78559]: WAR [Analysis daemon for 2 Cam-2 needs restarting, time since last analysis 48 seconds (1673709285-1673709237)]
Jan 14 18:14:46 cctv zmc_m2[96548]: WAR [zmc_m2] [Unable to get starting packet lock]
Jan 14 18:14:46 cctv zmc_m2[96548]: ERR [zmc_m2] [ALARM but no event]
Jan 14 18:14:57 cctv zmdc[78510]: WAR ['zmc -m 2' has not stopped at 23/01/14 18:14:57 after 10 seconds. Sending KILL to pid 96548]
Jan 14 18:18:14 cctv zmc_m2[98348]: WAR [zmc_m2] [You have set the max video packets in the queue to 200. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval 21 is larger than this setting.]
Jan 14 18:18:16 cctv zmc_m2[98348]: WAR [zmc_m2] [Found locked packet when trying to free up video packets. This basically means that decoding is not keeping up.]

Analysis FPS 2, FPS from cam 10, video stream from cams 2048x1536 h264.

Top look like this most time:

Mem: 1951M Active, 260K Inact, 1277M Laundry, 693M Wired, 392M Buf, 20M Free
Swap: 16G Total, 841M Used, 15G Free, 5% Inuse, 4096B In, 3840K Out

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root          3 155 ki31     0B    48K RUN      0  58,2H 244,15% [idle]
96282 www           4  20    0   687M   358M select   0   7:25  20,12% /usr/local/bin/zmc -m 3
96266 www           4  20    0   655M   362M select   0   7:26  19,05% /usr/local/bin/zmc -m 1
98348 www           4  20    0  3389M  2455M select   1   1:48  15,86% /usr/local/bin/zmc -m 2
abishai commented 1 year ago

Analysys FPS (and other ZM FPS settings) are very dangerous. I think they should be unset...

abishai commented 1 year ago

Is ajax request works for you in monitor view ? (Request that loads monitor alarm state and fps). It throws HTTP=500, something is wrong with semaphores again, however I have semaphore php extensions.

hjf commented 1 year ago

I have had OOM but I think they are related to my server not being able to compress h.264 fast enough. but I tried to revert back to JPEG and the MJPEG recordings don't play. there is a segfault in logs.

abishai commented 1 year ago

Well, looks like 1.36.12 is the last one that works under FreeBSD. 1.36.32 has the same issue that stopped updating to 1.36.19

Any attempt to write stream produces zmc crash. While monitor in detect mode and detection not triggered everything looks OK. The fasted way to test is just put monitor in record mode.

I've tried to lower FPS and resolution, so it's not 'decoding not keeping up' and single zmc instance produces 0.3 load (= less than 1 core)


1/14/23, 6:50:22 PM GMT+3 | zmdc |   | 12729 | ERR | 'zmc -m 6' exited abnormally, exit status 11 | zmdc.pl | -
-- | -- | -- | -- | -- | -- | -- | --
1/14/23, 6:50:22 PM GMT+3 | zmc_m6 |   | 583650 | ERR | Signal address is 0x125, from 0x27f012 | zm_signal.cpp | 80
1/14/23, 6:50:22 PM GMT+3 | zmc_m6 |   | 583650 | ERR | Got signal 11 (Segmentation fault), crashing | zm_signal.cpp | 50
1/14/23, 6:49:54 PM GMT+3 | zmc_m6 |   | 569575 | WAR | Found locked packet when trying to free up video packets. This basically means that decoding is not keeping up. | zm_packetqueue.cpp | 156
1/14/23, 6:49:54 PM GMT+3 | zmc_m6 |   | 569575 | WAR | You  have set the max video packets in the queue to 120. The queue is full.  Either Analysis is not keeping up or your camera's keyframe interval 0  is larger than this setting. | zm_packetqueue.cpp | 139
1/14/23, 6:48:57 PM GMT+3 | zmdc |   | 12729 | WAR | 'zmc -m 6' has not stopped at 23/01/14 18:48:57 after 10 seconds. Sending KILL to pid 31612 | zmdc.pl | -
1/14/23, 6:48:46 PM GMT+3 | zmc_m6 |   | 582186 | WAR | Unable to get starting packet lock | zm_monitor.cpp | 2692
1/14/23, 6:48:46 PM GMT+3 | zmwatch |   | 12864 | WAR | Analysis daemon for 6 Monitor-6 needs restarting, time since last analysis 47 seconds (1673711326-1673711279) | zmwatch.pl | -
1/14/23, 6:44:07 PM GMT+3 | zmc_m6 |   | 101695 | WAR | Found locked packet when trying to free up video packets. This basically means that decoding is not keeping up. | zm_packetqueue.cpp | 156
1/14/23, 6:44:07 PM GMT+3 | zmc_m6 |   | 101695 | WAR | You  have set the max video packets in the queue to 120. The queue is full.  Either Analysis is not keeping up or your camera's keyframe interval 0  is larger than this setting. | zm_packetqueue.cpp | 139
1/14/23, 6:43:22 PM GMT+3 | zmdc |   | 12729 | ERR | 'zmc -m 6' exited abnormally, exit status 11 | zmdc.pl | -
1/14/23, 6:43:22 PM GMT+3 | zmc_m6 |   | 581980 | ERR | Signal address is 0xbd, from 0x27f012 | zm_signal.cpp | 80
1/14/23, 6:43:22 PM GMT+3 | zmc_m6 |   | 581980 | ERR | Got signal 11 (Segmentation fault), crashing | zm_signal.cpp | 50
1/14/23, 6:43:21 PM GMT+3 | zmc_m6 |   | 276253 | WAR | Found locked packet when trying to free up video packets. This basically means that decoding is not keeping up. | zm_packetqueue.cpp | 156
1/14/23, 6:43:21 PM GMT+3 | zmc_m6 |   | 276253 | WAR | You  have set the max video packets in the queue to 120. The queue is full.  Either Analysis is not keeping up or your camera's keyframe interval 0  is larger than this setting. | zm_packetqueue.cpp | 139
1/14/23, 6:42:41 PM GMT+3 | zmdc |   | 12729 | ERR | 'zmc -m 6' exited abnormally, exit status 11 | zmdc.pl | -
1/14/23, 6:42:41 PM GMT+3 | zmc_m6 |   | 581790 | ERR | Signal address is 0xdd, from 0x27f012 | zm_signal.cpp | 80
1/14/23, 6:42:41 PM GMT+3 | zmc_m6 |   | 581790 | ERR | Got signal 11 (Segmentation fault), crashing | zm_signal.cpp | 50

signal 11 is always comes after 'locked packed' warning. Probably, memory corruption after zmc memory exhausted.

VVD commented 1 year ago

Analysys FPS (and other ZM FPS settings) are very dangerous. I think they should be unset...

Recommendations got this #3414.

Is ajax request works for you in monitor view ? (Request that loads monitor alarm state and fps). It throws HTTP=500, something is wrong with semaphores again, however I have semaphore php extensions.

Montage preview? Or what?

php81-ctype-8.1.14             The ctype shared extension for php
php81-gd-8.1.14                The gd shared extension for php
php81-intl-8.1.14              The intl shared extension for php
php81-opcache-8.1.14           The opcache shared extension for php
php81-pdo-8.1.14               The pdo shared extension for php
php81-pdo_mysql-8.1.14         The pdo_mysql shared extension for php
php81-pecl-APCu-5.1.22         APC User Caching
php81-session-8.1.14           The session shared extension for php
php81-sockets-8.1.14           The sockets shared extension for php

signal 11 is always comes after 'locked packed' warning. Probably, memory corruption after zmc memory exhausted.

Where do you get this log? What file name?

Oh, from menu Log:

1/14/23, 7:01:36 PM GMT+3 | zmc_m2 |   | 100074 | WAR | Found locked packet when trying to free up video packets. This basically means that decoding is not keeping up. | zm_packetqueue.cpp | 156
-- | -- | -- | -- | -- | -- | -- | --
1/14/23, 7:01:35 PM GMT+3 | zmc_m2 |   | 100074 | INF | Cam-2: 4700 - Capturing at 10.02 fps, capturing bandwidth 50062bytes/sec Analysing at 1.60 fps | zm_monitor.cpp | 1681
1/14/23, 7:01:34 PM GMT+3 | zmc_m2 |   | 100074 | WAR | You  have set the max video packets in the queue to 200. The queue is full.  Either Analysis is not keeping up or your camera's keyframe interval 21  is larger than this setting. | zm_packetqueue.cpp | 139
abishai commented 1 year ago

Montage preview? Or what?

When you click on monitor name ?view=watch

VVD commented 1 year ago

Montage preview? Or what?

When you click on monitor name ?view=watch

I see video, but apache spam php errors in error_log:

[Sat Jan 14 19:19:27.173855 2023] [php:warn] [pid 99226] [client MYIP:10718] PHP Warning:  fopen(/usr/local/www/zoneminder/includes/functions.php.sem.590549): Failed to open stream: Permission denied in /usr/local/www/zoneminder/includes/functions.php on line 2239, referer: http://SERVER/zm/index.php?view=watch&mid=1
[Sat Jan 14 19:19:27.173914 2023] [php:error] [pid 99226] [client MYIP:10718] PHP Fatal error:  Uncaught TypeError: flock(): Argument #1 ($stream) must be of type resource, bool given in /usr/local/www/zoneminder/includes/functions.php:2242\nStack trace:\n#0 /usr/local/www/zoneminder/includes/functions.php(2242): flock(false, 2)\n#1 /usr/local/www/zoneminder/ajax/stream.php(28): sem_acquire(false, 1)\n#2 /usr/local/www/zoneminder/index.php(269): require_once('/usr/local/www/...')\n#3 {main}\n  thrown in /usr/local/www/zoneminder/includes/functions.php on line 2242, referer: http://SERVER/zm/index.php?view=watch&mid=1

Ofc process with owner www can't open /usr/local/www/zoneminder/includes/functions.php.sem.590549:

ls -l /usr/local/www/zoneminder/includes/
total 352
-rw-r--r--  1 root  wheel   7367 Nov 18 22:21 Control.php
-rw-r--r--  1 root  wheel  23610 Nov 18 22:21 Event.php
-rw-r--r--  1 root  wheel  21270 Nov 18 22:21 Filter.php
-rw-r--r--  1 root  wheel  14208 Nov 18 22:21 FilterTerm.php
-rw-r--r--  1 root  wheel   1162 Nov 18 22:21 Frame.php
-rw-r--r--  1 root  wheel   7225 Nov 18 22:21 Group.php
-rw-r--r--  1 root  wheel    565 Nov 18 22:21 Group_Monitor.php
-rw-r--r--  1 root  wheel  23994 Nov 18 22:21 Monitor.php
-rw-r--r--  1 root  wheel    564 Nov 18 22:21 MontageLayout.php
-rw-r--r--  1 root  wheel  16434 Nov 18 22:21 Object.php
-rw-r--r--  1 root  wheel   4234 Nov 18 22:21 Server.php
-rw-r--r--  1 root  wheel   1954 Nov 18 22:21 Snapshot.php
-rw-r--r--  1 root  wheel   5129 Nov 18 22:21 Storage.php
-rw-r--r--  1 root  wheel   1464 Nov 18 22:21 User.php
-rw-r--r--  1 root  wheel   1686 Nov 18 22:21 Zone.php
drwxr-xr-x  2 root  wheel   1024 Jan 14 16:58 actions
-rw-r--r--  1 root  wheel  17024 Nov 18 22:21 auth.php
-rw-r--r--  1 root  wheel   9610 Jan 14 16:49 config.php
-rw-r--r--  1 root  wheel  27654 Nov 18 22:21 control_functions.php
drwxr-xr-x  2 root  wheel    512 Jan 14 16:58 csrf
-rw-r--r--  1 root  wheel  11257 Nov 18 22:21 database.php
-rw-r--r--  1 root  wheel  77700 Nov 18 22:21 functions.php
-rw-r--r--  1 root  wheel   2212 Nov 18 22:21 lang.php
-rw-r--r--  1 root  wheel  16647 Nov 18 22:21 logger.php
-rw-r--r--  1 root  wheel   6211 Nov 18 22:21 session.php

chown www /usr/local/www/zoneminder/includes fixed this error, but WHY it try to create files in dir with php files?!

abishai commented 1 year ago

Probably some temp directory or config is not set. On another way, php issues are not so serious. But zmc crash is. All I know version 1.36.12 works, 1.36.32 (and as I know for sure - 1.36.19) has zmc issue. So we can't update the port.

VVD commented 1 year ago

/usr/local/www/zoneminder/includes/functions.php:

// The following works around php not being built with semaphore functions.
if ( !function_exists('sem_get') ) {
  function sem_get($key) {
    return fopen(__FILE__ . '.sem.' . $key, 'w+');
  }
  function sem_acquire($sem_id) {
    return flock($sem_id, LOCK_EX);
  }
  function sem_release($sem_id) {
    return flock($sem_id, LOCK_UN);
  }     
}
abishai commented 1 year ago

That FILE is obviously wrong. It must point to temporary directory. I'd created separate issue if not exists. But this is merely a fallback code. You can try to find extent ion that provides sem_get and install it.

abishai commented 1 year ago

Patch against current port tree https://pastebin.com/bzGxWYNB We don't have much options left. Only identify the commit that introduced zmc issue with git bisection. Now we have rather broad dimentions, like 1.36.12 < x <= 1.36.19

VVD commented 1 year ago

ZM_DIR_TEMP is the good replace:

--- includes/functions.php.orig
+++ includes/functions.php
@@ -2236,7 +2236,7 @@
 // The following works around php not being built with semaphore functions.
 if ( !function_exists('sem_get') ) {
   function sem_get($key) {
-    return fopen(__FILE__ . '.sem.' . $key, 'w+');
+    return fopen(ZM_DIR_TEMP . '/sem.' . $key, 'w+');
   }
   function sem_acquire($sem_id) {
     return flock($sem_id, LOCK_EX);

Patch against current port tree https://pastebin.com/bzGxWYNB

  1. /var/cache/zoneminder/events => /var/db/zoneminder/events Check this too: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=267304#c3
  2. Require USE_PHP= intl.
VVD commented 1 year ago

Installed devel/php81-sysvsem, restarted apache, all work fine now.

VVD commented 1 year ago

My patch is: https://pastebin.com/bRCug87r

badrpc commented 1 year ago

Regarding sendfile: As I have said in https://github.com/ZoneMinder/zoneminder/issues/3505#issuecomment-1335877465 the way zoneminder uses sendfile is not supported on FreeBSD. zms uses zm_sendfile to send data back to fcgiwrap and the file descriptor there is not a socket. I used a hacky[1] approach to disable sendfile and got a working (except the OOM problem) 1.36.32 package. I'll try to find time to disable it properly.

Regarding sem_get, the php80-sysvsem (or php81-/php82- to match the version of php) package has solved the problem for me. The port must be modified to list it as a dependency, but in the meantime I just installed it manually.

The 1.36.32 runs out of memory for me too. Easily kills 32G machine on first alarm. It looks like the one reported in https://github.com/ZoneMinder/zoneminder/issues/3614, I'm not sure how to trace this one down.

  1. Hacky patch to disable sendfile 
    
--- work/zoneminder-1.36.32/src/zm_sendfile.h.orig  2022-11-18 19:21:30.000000000 +0000
+++ work/zoneminder-1.36.32/src/zm_sendfile.h   2022-12-01 22:39:22.542680000 +0000
@@ -5,32 +5,33 @@
extern "C" {
#endif

-#ifdef HAVE_SENDFILE4_SUPPORT -#include <sys/sendfile.h> -#elif HAVE_SENDFILE7_SUPPORT +//#ifdef HAVE_SENDFILE4_SUPPORT +//#include <sys/sendfile.h> +//#elif HAVE_SENDFILE7_SUPPORT

include <sys/types.h>

include <sys/socket.h>

-#include <sys/uio.h> -#else +//#include <sys/uio.h> +//#else

include

-#endif +//#endif

/ Function to send the contents of a file. Will use sendfile or fall back to reading/writing /

ssize_t zm_sendfile(int out_fd, int in_fd, off_t *offset, size_t size) { -#ifdef HAVE_SENDFILE4_SUPPORT

abishai commented 1 year ago

@badrpc sendfile is already disabled in port config via cmake option, no need to patch is furhter if it compiles - it will always fallback to read/write.

badrpc commented 1 year ago

Sorry, that's probably a recent update and I didn't notice it. Need to update my tree I suppose. Sorry again.

abishai commented 1 year ago

No, it was from the beginning. Under FreeBSD ZM never used sendfile. Any attempt to turn it on would crash zms. It's just stopped compile at all, so we were fixing compilation issue, not sendfile itself. My, @VVD or @connortechnology patch is enough to fix sendfile problem. We have zmc issue though. We need to find when it turned into memory hog.

badrpc commented 1 year ago

Hmm.... That (disabled sendfile) doesn't match my experience. I can indeed see that ports Makefile sets -DHAVE_SENDFILE=0 but I'm pretty sure that it was compiling with HAVE_SENDFILE7_SUPPORT and then zms process was reporting zms_e95.log:12/01/22 22:11:12.762191 zms_e95[101577].ERR-zm_eventstream.cpp/730 [Can't send /var/db/zoneminder/events/3/2022-11-29/95/00001-capture.jpg: Socket operation on non-socket]. I will check and report back.

Regarding the memory usage of zmc, yes I agree. But it might be worth opening another issue for that as this one starts to have multiple parallel threads and becomes harder to follow.

connortechnology commented 1 year ago

So I'd like to get to the bottom of all this. I used to have a freebsd vm,m but it's long gone. I'm downloading 13.1. Can you guys do me a favour and document the steps needed to go from a fresh install to getting zm installed? Doesn't have to be perfect, and we can stick it all in the official docs so that FreeBSD can become a more supported platform.

abishai commented 1 year ago

Simply installed or you need to play with building as well ? I'll try to describe the process, but from my experience it scares Linux folks =/

Port has step-by-step installation procedure documented step-by-step https://github.com/freebsd/freebsd-ports/blob/main/multimedia/zoneminder/files/pkg-message.in

The only trick is to switch fresh FreeBSD installation to latest packaged editing /etc/pkg/FreeBSD.conf

FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}

and install necessary software with

pkg install mysql57-server fcgiwrap nginx zoneminder

and several PHP dependencies I forgot to include

pkg install php81-intl  devel/php81-sysvsem

You can setup ZM according documentation after that. (The latest version in port 1.36.12). The process should be nearly the same as in every distro and specific daemon enabled and start are mentioned in documentation.

If you want to build it under FreeBSD

After you've got everything running, stop ZM and delete it

pkg delete zoneminder

Install port collection with 

portsnap auto

ZM build recipe will be in /usr/ports/multimedia/zoneminder

Try to build it with 

make

make will pull necessary build dependencies and compile everything. (If it try to compile something big like llvm, you can install it from packages).

make clean

will clean all built material

After that, apply my patch https://pastebin.com/bzGxWYNB on top, this will bump port to 1.36.32. Try to make clean it to verify everything still builds.

Useful make commands

make clean - cleans workdirs
make - compiles
make install - build + install
make deinstall - removes installed
make extract - extract sources to workdir
make patch - extract sources and apply patches (specific to FreeBSD - from files/) - useful if you want to edit something. (make will proceed with patched sources) if you type make after.

Please notice, that make here is not cmake or any other linux make, it's port collection make, that executes recipe from Makefile

abishai commented 1 year ago

(I can prepare everything if you can provide ssh access)

VVD commented 1 year ago

fcgiwrap nginx

I can try to add description how to configure with apache. But better discuss this in IRC or in https://bugs.freebsd.org/bugzilla/.

abishai commented 1 year ago

I've come to idea of serial tests using exactly the same dependencies I have in production (it's very stable on 1.36.12). I copy everything and begin step by step version bump of ZM alone in test environment.

This, we'll find when bug occurs (or not if bug in dependency or how ZM works with updated dependency).

As for now, 1.36.14 doesn't have the problem, Logs are green.

Version 1.36.18 - broken. zmc doesn't sigfault, but record and mocord modes are broken. They throws 'locked package' and restarting every minute. Version 1.36.16 similar to 1.36.18

Monitors 3 and 5 are in mocord mode and restaring Monitor 6 in record and restarting as well. Monitors 1, 2, 4 are in modect and works while they are not actually recording.

2023-01-15 15:13:57 | zmdc |   | 22213 | WAR | 'zmc -m 5' has not stopped at 23/01/15 15:13:57 after 10 seconds. Sending KILL to pid 22950 | zmdc.pl | -
-- | -- | -- | -- | -- | -- | -- | --
2023-01-15 15:13:56 | zmwatch |   | 22339 | WAR | Analysis daemon for 5 Monitor-5 needs restarting, time since last analysis 62 seconds (1673784836-1673784774) | zmwatch.pl | -
2023-01-15 15:13:56 | zmdc |   | 22213 | WAR | 'zmc -m 5' has not stopped at 23/01/15 15:13:56 after 10 seconds. Sending KILL to pid 22950 | zmdc.pl | -
2023-01-15 15:13:55 | zmwatch |   | 22339 | WAR | Analysis daemon for 3 Monitor-3 needs restarting, time since last analysis 51 seconds (1673784835-1673784784) | zmwatch.pl | -
2023-01-15 15:13:45 | zmwatch |   | 22339 | WAR | Analysis daemon for 5 Monitor-5 needs restarting, time since last analysis 51 seconds (1673784825-1673784774) | zmwatch.pl | -
2023-01-15 15:13:07 | zmc_m6 |   | 609334 | WAR | Found locked packet when trying to free up video packets. This basically means that decoding is not keeping up. | zm_packetqueue.cpp | 136
2023-01-15 15:13:07 | zmc_m6 |   | 609334 | WAR | Found locked packet when trying to free up video packets. This basically means that decoding is not keeping up. | zm_packetqueue.cpp | 136
2023-01-15 15:13:07 | zmc_m6 |   | 609334 | WAR | Found locked packet when trying to free up video packets. This basically means that decoding is not keeping up. | zm_packetqueue.cpp | 136

Will narrow the way further. /I will update this post/

abishai commented 1 year ago

1.36.15 doesn't look good, first compilation failure in zm_time, fixedf with https://github.com/ZoneMinder/zoneminder/commit/df414fc028aebddd0105c7405dbccfe61dc7458a Now I have linker issue because of curl I disabled long time ago in cmake... Going to investigate it. As 1.36.16 compiles normally, I suppose 1.36.15 was buggy...

I compiled it, but after database update it pretends to be 1.37.16 instead of 1.36.15. And zmc is broken fully. No monitors starts, zmc segfaults.

abishai commented 1 year ago 
abishai@observer-test:~ % doas -u www zmc -m 6
01/15/23 16:55:49.342352 zmc_m6[570458].ERR-zm_monitor.cpp/804 [Can't open memory map file /dev/shm/zm.mmap.6: No such file or directory]

01/15/23 16:55:49.342445 zmc_m6[570458].WAR-zmc.cpp/241 [Couldn't connect to monitor 6]

01/15/23 16:55:49.342513 zmc_m6[570458].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]

01/15/23 16:55:49.342568 zmc_m6[570458].ERR-zm_signal.cpp/80 [Signal address is 0x60, from 0x23831e]

I see. We don't have /dev/shm . We have tmpfs for that and can mount it to directory if needed. A lot of FreeBSD users do this for /tmp to speed things. I suspect 1.36.15 is development branch then? In 1.36.xxx we can set shm directory in ZM config

abishai commented 1 year ago

OK, as I can't test 1.36.15, we can assume that the issue was introduced somewhere between 1.36.14 and 1.36.16. If monitor writes, frame queue is overflowed for some reason. In 1.36.16 zmc is killed by watchdog, in 1.36.32 things become even worse, it exhausts the memory and crashes.

Any idea of suspicious places to look? Probably, I'd try to fix 1.36.16 to work like 1.36.14 rolling back random cpp files.

nm about /dev/shm. CMake reverted ZM_PATH_MAP to defaults because of my mistake in local patch. I still don't know why 1.36.15 updated database to 1.37.16 and should I test it for zmc bug.

connortechnology commented 1 year ago

I don't think there is much point in testing each version, but if you are going to do so, use git bisect.

/dev/shm is not hard coded. It gets the value from /etc/zm.conf or other files in /etc/zm/conf.d. Generally these files are pre-populated by cmake build system.

ANyways, I will get my build system running today and likely have it all figured out soon.

I am concerned about this lack of curl. We really need it for modern features.