search

OpenIPC / majestic

Majestic Community edition integration kit
MIT License
32 stars 5 forks source link

majestic segfaults after some time on hi3518ev100 + ov9712 #171

Open zerog2k opened 1 month ago

zerog2k commented 1 month ago

Issue description

Majestic crashes (segfault) after some time (minutes to hours) streaming rtsp h264. (No crashes if streaming only mjpeg.)

Another observation is that when majestic dies, even though watchdog is enabled, the camera never resets. OS stays up and available but majestic is gone. (I'm certain watchdog was working properly on majestic/openipc versions from at least as recent as February. Majestic did crash, but I believe watchdog was keeping this from being a terminal issue.)

might be similar to #159

Expectations

no crash

Steps to reproduce

  1. startup rtsp stream from rtsp://CAMERAIP/?stream=0
  2. after some time majestic coredumps

Information to attach

majestic.4731.11.1720275013.core.zip majestic.yaml.zip

root@openipc-garage:~# majestic
14:00:02  <       majestic> [    main] main@237                      Majestic Lite for HiSilicon, version master+49b91be, built on 2024-07-04
14:00:03  <       majestic> [watchdog] watchdog_start@94             Detected Hisilicon Watchdog
14:00:03  <       majestic> [watchdog] watchdog_start@115            Watchdog timeout set to 60 seconds
14:00:03  <       majestic> [   httpd] start_web_server@379          Set server path: /var/www
14:00:03  <       majestic> [   httpd] new_http_server@367           HTTP server started on :::80
14:00:03  <       majestic> [    main] init_sdk@66                   Loading main configuration: /etc/majestic.yaml
14:00:03  <       majestic> [     sdk] try_sensor_config@3292        SENSOR=ov9712
14:00:03  <       majestic> [     sdk] find_sensor_config@3260       matched sensor config: ov9712_i2c_dc_720p.ini
14:00:03  <       majestic> [     sdk] start_sdk@386                 Using /etc/sensors/ov9712_i2c_dc_720p.ini sensor configuration file
14:00:03  <       majestic> [     sdk] start_sdk@391                 App was built with MPP version: Hi3518_MPP_V1.0.B.0.
14:00:03  <       majestic> [     sdk] start_sdk@396                 Current MPP version: HI_VERSION=Hi3518_MPP_V1.0.B.0
14:00:03  <       majestic> [     sdk] start_sdk@424                 sensor ov9712
14:00:03  <       majestic> [     sdk] start_sdk@432                   dev [1280x720]@0x0 15fps, BGGR
14:00:03  <       majestic> [  sensor] try_to_load@19                trying to load /usr/lib/sensors/libsns_ov9712_i2c_dc.so
14:00:03  <       majestic> [     hal] hisi_free_mem_ex@753          Free MMZ mem before allocation: 19580KB
14:00:04  <       majestic> [     sdk] dump_vb_configuration@2190    VB configuration:
14:00:04  <       majestic> [     sdk] dump_vb_configuration@2193      [0]: 1413120 x 6
14:00:04  <       majestic> [     hal] hisi_free_mem_ex@753          Free MMZ mem after allocation: 11288KB
14:00:04  <       majestic> [     sdk] init_sensor@2385              Sensor driver loaded
14:00:04  <       majestic> [     sdk] log_venc_chn@1250             h264 1280x720@15fps 1024kbit 15gop
14:00:04  <     iq_monitor> [     sdk] monitor_thread@313            Loading IQ profile /etc/sensors/iq/default.ini
14:00:04  <       majestic> [     sdk] create_vpss_chn@995             venc_chn: 0, vpss_grp: 0, vpss_chn: 0
14:00:04  <       majestic> [     sdk] start_video@1933                venc took 3400KB of MMZ [675]
14:00:04  <       majestic> [     sdk] log_venc_chn@1250             MJPEG 1280x720@5fps 50q
14:00:04  <       majestic> [     sdk] create_vpss_chn@995             venc_chn: 1, vpss_grp: 0, vpss_chn: 1
14:00:04  <       majestic> [     sdk] start_mjpeg@1391                mpeg_enc took 916KB of MMZ [900]
14:00:04  <       majestic> [     osd] init_osd@97                   OSD initialized
14:00:04  <       majestic> [     sdk] start_sdk@517                   OSD took 176KB of MMZ
14:00:04  <       majestic> [     sdk] start_sdk@544                 HiSilicon SDK started
14:00:04  <       majestic> [     hal] hisi_free_mem_ex@753          Free MMZ mem finally: 2632KB
14:00:04  <       majestic> [    rtsp] rtsp_init@28                  RTSP server started on port 554
14:00:04  <       majestic> [    gpio] set_gpio@26                   set_gpio(76, 0)
14:00:04  <       majestic> [    gpio] set_gpio@26                   set_gpio(79, 1)
14:00:04  <       majestic> [    gpio] set_gpio@26                   set_gpio(73, 0)
14:00:04  <       majestic> [   night] setup_gpio_listener@189       Starting monitor for hardware sensor
14:00:04  <       majestic> [    gpio] set_gpio@26                   set_gpio(76, 0)
14:00:04  <       majestic> [    gpio] set_gpio@26                   set_gpio(79, 0)
14:00:12  <       majestic> [  client] RtspClient_before@403         RTSP OPTIONS rtsp://10.0.3.33/stream=0 CSeq=1 from 10.0.0.4:65254
14:00:12  <       majestic> [  client] RtspClient_before@403         RTSP OPTIONS rtsp://10.0.3.33/stream=0 CSeq=2 from 10.0.0.4:65254
14:00:12  <       majestic> [  client] RtspClient_before@403         RTSP DESCRIBE rtsp://10.0.3.33/stream=0 CSeq=3 from 10.0.0.4:65254
14:00:12  <       majestic> [  client] RtspClient_before@403         RTSP SETUP rtsp://10.0.3.33/stream=0/video CSeq=4 from 10.0.0.4:65254
14:00:12  <       majestic> [  client] RtspClient_setup@314          TCP RTSP SETUP done, stream_id=0 from 10.0.0.4:65254
14:00:12  <       majestic> [  client] RtspClient_before@403         RTSP PLAY rtsp://10.0.3.33/stream=0 CSeq=5 from 10.0.0.4:65254
14:00:34  <       majestic> [   night] set_night_mode@115            Change mode to NIGHT
14:00:34  <       majestic> [    gpio] set_gpio@26                   set_gpio(76, 1)
14:00:34  <       majestic> [    gpio] set_gpio@26                   set_gpio(79, 0)
14:00:34  <       majestic> [    gpio] set_gpio@26                   set_gpio(73, 1)
14:00:34  <       majestic> [    gpio] set_gpio@26                   set_gpio(76, 0)
14:00:34  <       majestic> [    gpio] set_gpio@26                   set_gpio(79, 0)
<< after a few minutes >>
Segmentation fault (core dumped)
root@openipc-garage:~# cat /proc/umap/h264e

[H264E] Version: [Hi3518_MPP_V1.0.B.0 ], Build Time[Nov 19 2015, 16:37:03]

-----MODULE PARAM--------------------------------------------------------------
    ContiBuffer  H264eVBSource  H264eRcnEqualRef  H264eMiniBufMode
              0              1                 0                 0
-----CHN ATTR------------------------------------------------------------------
     ID  MaxWidth MaxHeight  PicWidth PicHeight   profile   MainStr   RefMode   BufSize   ByFrame   MaxStrCnt  PoolId
      0      1280       720      1280       720        mp       Yes        1X    691200       Yes         200       1

-----PICTURE INFO--------------------------------------------------------------
  ID     EncdStart   EncdSucceed    Lost    Disc    Skip  BufLeak   PSkip  Recode      RlsStr  UnrdStr   GetVBFail
   0          9215          9215   23328       0       0    23328       0       0        9129       86           0

-----STREAM BUFFER-------------------------------------------------------------
     ID     Base        RdTail      RdHead      WrTail      WrHead      DataLen     BufFree
     0      0xc3600000  0xa8f40     0xa8d40     0xa8d40     0xa8d40     0           448

-----RefParam INFO--------------------------------------------------------------
     ID    EnPred      Base   Enhance     EnIDR
      0       Yes         1         0       Yes

-----ROI INFO------------------------------------------------------------------
     ID     index    bAbsQp    Qp     width    height    startx    starty

-----Syntax INFO---------------------------------------------------------------
     ID   slcsplt   slcmode   slcsize
      0       No        N/A       N/A

-----Inter & Intra prediction INFO---------------------------------------------
     ID   profile  HWsize  VWsize  P16x16   P16x8   P8x16    P8x8   MvExt  I16x16    Inxn    Ipcm
      0        mp       5       2     Yes     Yes     Yes     Yes     Yes     Yes     Yes     Yes

-----Syntax INFO--------------------------------------------------------------
     ID   profile   EntrpyI   EntrpyP  Itrans  ptrans QMatrix   POC   DblkIdc   alpha    beta
      0        mp     cabac     cabac     4x4     4x4     No      2         0       0       0
Flole998 commented 1 month ago

I am seeing the same thing. The watchdog seems to be broken on other Chipsets aswell, looks like it gets disabled on crash, dmesg suggests that.

Flole998 commented 1 month ago

The watchdog has been effectively disabled on all HiSilicon platforms. If you want to enable it again, re-add the nodeamon=1 parameter for the insmod wdt command in load_hisilicon. For me that at least causes it to automatically reboot.

skilurius commented 3 weeks ago

I got access to a device for testing (HI3518CV100+IMX138), but unfortunately could not reproduce the rtsp crash. I tried three sessions with 1hour runtime each: mpv rtsp://root:12345@192.168.1.10/stream=0

root@openipc-hi3518cv100:~# cat /proc/umap/h264e

[H264E] Version: [Hi3518_MPP_V1.0.B.0 ], Build Time[Nov 19 2015, 16:37:03]

-----MODULE PARAM--------------------------------------------------------------
    ContiBuffer  H264eVBSource  H264eRcnEqualRef  H264eMiniBufMode
              0              1                 0                 0
-----CHN ATTR------------------------------------------------------------------
     ID  MaxWidth MaxHeight  PicWidth PicHeight   profile   MainStr   RefMode   BufSize   ByFrame   MaxStrCnt  PoolId
      0      1280       720      1280       720        mp       Yes        1X    691200       Yes         200       1

-----PICTURE INFO--------------------------------------------------------------
  ID     EncdStart   EncdSucceed    Lost    Disc    Skip  BufLeak   PSkip  Recode      RlsStr  UnrdStr   GetVBFail
   0         61869         61869   30934       0   30934        0       0       0       61869        0           0

-----STREAM BUFFER-------------------------------------------------------------
     ID     Base        RdTail      RdHead      WrTail      WrHead      DataLen     BufFree     
     0      0xc2e00000  0x6c2c0     0x6c2c0     0x6c2c0     0x6c2c0     0           692160      

-----RefParam INFO--------------------------------------------------------------
     ID    EnPred      Base   Enhance     EnIDR
      0       Yes         1         0       Yes

-----ROI INFO------------------------------------------------------------------
     ID     index    bAbsQp    Qp     width    height    startx    starty

-----Syntax INFO---------------------------------------------------------------
     ID   slcsplt   slcmode   slcsize
      0       No        N/A       N/A

-----Inter & Intra prediction INFO---------------------------------------------
     ID   profile  HWsize  VWsize  P16x16   P16x8   P8x16    P8x8   MvExt  I16x16    Inxn    Ipcm
      0        mp       5       2     Yes     Yes     Yes     Yes     Yes     Yes     Yes     Yes

-----Syntax INFO--------------------------------------------------------------
     ID   profile   EntrpyI   EntrpyP  Itrans  ptrans QMatrix   POC   DblkIdc   alpha    beta
      0        mp     cabac     cabac     4x4     4x4     No      2         0       0       0
Flole998 commented 3 weeks ago

Try leaving it running for longer. I am seeing this on

# ipcinfo -cs
hi3516ev300
imx335

and the camera has an uptime of about 20 hours now. So it does happen eventually, but it is not really plannable.

zerog2k commented 3 weeks ago

anyone from majestic dev team get a chance to analyze the coredump? with closed source and no symbols, it would be exceedingly hard for us to guess what is going on, and possibly theorize ways to reproduce it more reliably

skilurius commented 3 weeks ago

The core dump is unfortunately missing some function pointer. However, the problem seems to be the presumed buffer leak, it moves the datastream after a while which leads to the segmentation fault. The H264 buffer size was slightly increased on the current release, please check it and report back.

zerog2k commented 2 weeks ago 
root@openipc-hi3518ev100:~# majestic
20:33:56  <       majestic> [    main] main@237                      Majestic Lite for HiSilicon, version master+d3f2c7a, built on 2024-08-08
20:33:57  <       majestic> [watchdog] watchdog_start@94             Detected Hisilicon Watchdog
20:33:57  <       majestic> [watchdog] watchdog_start@115            Watchdog timeout set to 60 seconds
20:33:57  <       majestic> [   httpd] start_web_server@383          Set server path: /var/www
20:33:57  <       majestic> [   httpd] new_http_server@371           HTTP server started on :::80
20:33:57  <       majestic> [    main] init_sdk@66                   Loading main configuration: /etc/majestic.yaml
20:33:57  <       majestic> [     sdk] try_sensor_config@3276        SENSOR=ov9712
20:33:57  <       majestic> [     sdk] find_sensor_config@3244       matched sensor config: ov9712_i2c_dc_720p.ini
20:33:57  <       majestic> [     sdk] start_sdk@380                 Using /etc/sensors/ov9712_i2c_dc_720p.ini sensor configuration file
20:33:57  <       majestic> [     sdk] start_sdk@385                 App was built with MPP version: Hi3518_MPP_V1.0.B.0.
20:33:57  <       majestic> [     sdk] start_sdk@390                 Current MPP version: HI_VERSION=Hi3518_MPP_V1.0.B.0
20:33:57  <       majestic> [     sdk] start_sdk@418                 sensor ov9712
20:33:57  <       majestic> [     sdk] start_sdk@427                   dev [1280x720]@0x0 25fps, BGGR
20:33:57  <       majestic> [  sensor] try_to_load@19                trying to load /usr/lib/sensors/libsns_ov9712_i2c_dc.so
20:33:57  <       majestic> [     hal] hisi_free_mem_ex@753          Free MMZ mem before allocation: 19580KB
20:33:57  <       majestic> [     sdk] dump_vb_configuration@2174    VB configuration:
20:33:57  <       majestic> [     sdk] dump_vb_configuration@2177      [0]: 1413120 x 6
20:33:57  <       majestic> [     hal] hisi_free_mem_ex@753          Free MMZ mem after allocation: 11288KB
20:33:57  <       majestic> [     sdk] init_sensor@2369              Sensor driver loaded
20:33:57  <       majestic> [     sdk] log_venc_chn@1253             h264 1280x720@15fps 1024kbit 15gop
20:33:57  <     iq_monitor> [     sdk] monitor_thread@307            Loading IQ profile /etc/sensors/iq/default.ini
20:33:58  <       majestic> [     sdk] create_vpss_chn@993             venc_chn: 0, vpss_grp: 0, vpss_chn: 0
20:33:58  <       majestic> [     sdk] start_video@1917                venc took 3444KB of MMZ [720]
20:33:58  <       majestic> [     sdk] log_venc_chn@1253             MJPEG 1280x720@5fps 25q
20:33:58  <       majestic> [     sdk] create_vpss_chn@993             venc_chn: 1, vpss_grp: 0, vpss_chn: 1
20:33:58  <       majestic> [     sdk] start_mjpeg@1378                mpeg_enc took 916KB of MMZ [900]
20:33:58  <       majestic> [     osd] init_osd@103                  OSD initialized
20:33:58  <       majestic> [     sdk] start_sdk@515                   OSD took 176KB of MMZ
20:33:58  <       majestic> [     sdk] start_sdk@542                 HiSilicon SDK started
20:33:58  <       majestic> [     hal] hisi_free_mem_ex@753          Free MMZ mem finally: 2588KB
20:33:58  <       majestic> [    rtsp] rtsp_init@28                  RTSP server started on port 554
20:33:58  <       majestic> [    gpio] set_gpio@26                   set_gpio(76, 0)
20:33:58  <       majestic> [    gpio] set_gpio@26                   set_gpio(79, 1)
20:33:58  <       majestic> [    gpio] set_gpio@26                   set_gpio(73, 0)
20:33:58  <       majestic> [   night] setup_gpio_listener@189       Starting monitor for hardware sensor
20:33:58  <       majestic> [    gpio] set_gpio@26                   set_gpio(76, 0)
20:33:58  <       majestic> [    gpio] set_gpio@26                   set_gpio(79, 0)
20:33:58  <       majestic> [  client] RtspClient_before@403         RTSP OPTIONS rtsp://10.0.3.78/stream=0 CSeq=1 from 10.0.0.4:55164
20:33:58  <       majestic> [  client] RtspClient_before@403         RTSP OPTIONS rtsp://10.0.3.78/stream=0 CSeq=2 from 10.0.0.4:55164
20:33:58  <       majestic> [  client] RtspClient_before@403         RTSP DESCRIBE rtsp://10.0.3.78/stream=0 CSeq=3 from 10.0.0.4:55164
20:33:58  <       majestic> [  client] RtspClient_before@403         RTSP SETUP rtsp://10.0.3.78/stream=0/video CSeq=4 from 10.0.0.4:55164
20:33:58  <       majestic> [  client] RtspClient_setup@314          TCP RTSP SETUP done, stream_id=0 from 10.0.0.4:55164
20:33:58  <       majestic> [  client] RtspClient_before@403         RTSP PLAY rtsp://10.0.3.78/stream=0 CSeq=5 from 10.0.0.4:55164
Segmentation fault (core dumped)
root@openipc-hi3518ev100:~#

majestic.2440.11.1723254891.core.gz

also updated majestic.yaml (as I was disabling second video1 stream)

system:
  webPort: 80
  httpsPort: 443
  logLevel: debug
  buffer: 2048
  plugins: false
isp:
  antiFlicker: disabled
  blkCnt: 6
  dis: false
image:
  mirror: false
  flip: false
  rotate: 0
  contrast: 50
  hue: 50
  saturation: 50
  luminance: 50
video0:
  enabled: true
  codec: h264
  fps: 15
  bitrate: 1024
  rcMode: vbr
  gopSize: 1.0
  gopMode: normal
  size: 1280x720
  sliceUnits: 0
video1:
  enabled: false
  codec: h264
  size: 640x360
  fps: 10
  bitrate: 512
  gopMode: normal
  rcMode: vbr
  sliceUnits: 0
  gopSize: 1.0
jpeg:
  enabled: true
  qfactor: 25
  fps: 5
osd:
  enabled: true
  font: "/usr/share/fonts/truetype/UbuntuMono-Regular.ttf"
  template: "%F %T"
  posX: 16
  posY: 16
audio:
  enabled: false
  volume: 30
  srate: 8000
  codec: opus
  outputEnabled: false
  outputVolume: 30
rtsp:
  enabled: true
  port: 554
nightMode:
  colorToGray: true
  irCutSingleInvert: false
  lightMonitor: true
  lightSensorInvert: false
  backlightPin: 73
  irCutPin1: 76
  irCutPin2: 79
  lightSensorPin: 77
  monitorDelay: 0
motionDetect:
  enabled: true
  visualize: false
  debug: false
records:
  enabled: false
  path: "/mnt/mmcblk0p1/%F/%H.mp4"
  maxUsage: 95
outgoing:
  enabled: false
watchdog:
  enabled: true
  timeout: 60
hls:
  enabled: false
onvif:
  enabled: true
Flole998 commented 2 weeks ago

For me it hasn't crashed yet since 32 hours, however I've seen some majestic versions work better than others in the past, so this could be coincidence. I'll give you an update in a few days how it's going.

Flole998 commented 2 weeks ago

It looks like it's fixed on hi3516ev300 but my hi3518ev100 keeps crashing.

skilurius commented 2 weeks ago

The buffer size has now been increased to width * height * 1.5

Flole998 commented 1 week ago

Unfortunately I'm still seeing crashes, now also on hi3516ev300 again. I'm running master+11648f1, 2024-08-14.

zerog2k commented 1 week ago

@Flole998 same here w/ 8-13 build of majestic if i do not connect any rtsp streamer, it will stay up for many days (tested 5) but if I connect rtsp streamer, it will segfault after a few hours at most

Curious: I'm using Blue Iris 5. What rtsp streamer are you using? also Im using osmem=40M, (mmz_size: 24M) what are you using?

zerog2k commented 1 week ago

also @skilurius thanks for looking into this. Not sure if it's helpful but here is some data from OEM firmware on my device, which has been up for over 19 days streaming rtsp without issue.

# cat /proc/umap/h264e

[H264E] Version: [Hi3518_MPP_V1.0.A.1 ], Build Time[Aug 24 2015, 22:43:00]

-----MODULE PARAM--------------------------------------------------------------
    ContiBuffer  H264eVBSource  H264eRcnEqualRef  H264eMiniBufMode
              0              1                 0                 0
-----CHN ATTR------------------------------------------------------------------
     ID  MaxWidth MaxHeight  PicWidth PicHeight   profile   MainStr   RefMode   BufSize   ByFrame   MaxStrCnt  PoolId
      0      1280       720      1280       720        mp       Yes        1X   1382400       Yes         200       4
      1       352       288       352       288        mp       Yes        1X    152064       Yes         200       5

-----PICTURE INFO--------------------------------------------------------------
  ID     EncdStart   EncdSucceed    Lost    Disc    Skip  BufLeak   PSkip  Recode      RlsStr  UnrdStr   GetVBFail
   0        428460        428459  285640       0  285640        0       0       1      428459        0           0
   1        428459        428459  285640       0  285640        0       0       0      428459        0           0

-----STREAM BUFFER-------------------------------------------------------------
     ID     Base        RdTail      RdHead      WrTail      WrHead      DataLen     BufFree
     0      0xc5e00000  0x1df40     0x1df40     0x1df40     0x1df40     0           1384384
     1      0xc4580000  0x11340     0x11340     0x11340     0x11340     0           155584

-----RefParam INFO--------------------------------------------------------------
     ID    EnPred      Base   Enhance     EnIDR
      0       Yes         1         0       Yes
      1       Yes         1         0       Yes

-----ROI INFO------------------------------------------------------------------
     ID     index    bAbsQp    Qp     width    height    startx    starty

-----Syntax INFO---------------------------------------------------------------
     ID   slcsplt   slcmode   slcsize
      0       No        N/A       N/A
      1       No        N/A       N/A

-----Inter & Intra prediction INFO---------------------------------------------
     ID   profile  HWsize  VWsize  P16x16   P16x8   P8x16    P8x8   MvExt  I16x16    Inxn    Ipcm
      0        mp       5       2     Yes     Yes     Yes     Yes     Yes     Yes     Yes     Yes
      1        mp       5       2     Yes     Yes     Yes     Yes     Yes     Yes     Yes     Yes

-----Syntax INFO--------------------------------------------------------------
     ID   profile   EntrpyI   EntrpyP  Itrans  ptrans QMatrix   POC   DblkIdc   alpha    beta
      0        mp     cabac     cabac     4x4     4x4     No      2         0       0       0
      1        mp     cabac     cabac     4x4     4x4     No      2         0       0       0

on my majestic test device (same hardware):

root@openipc-hi3518ev100:~# cat /proc/umap/h264e

[H264E] Version: [Hi3518_MPP_V1.0.B.0 ], Build Time[Nov 19 2015, 16:37:03]

-----MODULE PARAM--------------------------------------------------------------
    ContiBuffer  H264eVBSource  H264eRcnEqualRef  H264eMiniBufMode
              0              1                 0                 0
-----CHN ATTR------------------------------------------------------------------
     ID  MaxWidth MaxHeight  PicWidth PicHeight   profile   MainStr   RefMode   BufSize   ByFrame   MaxStrCnt  PoolId
      0      1280       720      1280       720        mp       Yes        1X   1382400       Yes         200       1

-----PICTURE INFO--------------------------------------------------------------
  ID     EncdStart   EncdSucceed    Lost    Disc    Skip  BufLeak   PSkip  Recode      RlsStr  UnrdStr   GetVBFail
   0        840604        840600       0       0       0        0       0       4      840600        0           0

-----STREAM BUFFER-------------------------------------------------------------
     ID     Base        RdTail      RdHead      WrTail      WrHead      DataLen     BufFree
     0      0xc3a00000  0xdb3c0     0xdb3c0     0xdb3c0     0xdb3c0     0           1384384

-----RefParam INFO--------------------------------------------------------------
     ID    EnPred      Base   Enhance     EnIDR
      0       Yes         1         0       Yes

-----ROI INFO------------------------------------------------------------------
     ID     index    bAbsQp    Qp     width    height    startx    starty

-----Syntax INFO---------------------------------------------------------------
     ID   slcsplt   slcmode   slcsize
      0       No        N/A       N/A

-----Inter & Intra prediction INFO---------------------------------------------
     ID   profile  HWsize  VWsize  P16x16   P16x8   P8x16    P8x8   MvExt  I16x16    Inxn    Ipcm
      0        mp       5       2     Yes     Yes     Yes     Yes     Yes     Yes     Yes     Yes

-----Syntax INFO--------------------------------------------------------------
     ID   profile   EntrpyI   EntrpyP  Itrans  ptrans QMatrix   POC   DblkIdc   alpha    beta
      0        mp     cabac     cabac     4x4     4x4     No      2         0       0       0
zerog2k commented 1 week ago

ok and here is one after majestic crashed after just about 20 minutes of running stream:

root@openipc-hi3518ev100:~# cat /proc/umap/h264e

[H264E] Version: [Hi3518_MPP_V1.0.B.0 ], Build Time[Nov 19 2015, 16:37:03]

-----MODULE PARAM--------------------------------------------------------------
    ContiBuffer  H264eVBSource  H264eRcnEqualRef  H264eMiniBufMode
              0              1                 0                 0
-----CHN ATTR------------------------------------------------------------------
     ID  MaxWidth MaxHeight  PicWidth PicHeight   profile   MainStr   RefMode   BufSize   ByFrame   MaxStrCnt  PoolId
      0      1280       720      1280       720        mp       Yes        1X   1382400       Yes         200       1

-----PICTURE INFO--------------------------------------------------------------
  ID     EncdStart   EncdSucceed    Lost    Disc    Skip  BufLeak   PSkip  Recode      RlsStr  UnrdStr   GetVBFail
   0          2269          2269   15064       0       0    15064       0       0        2069      200           0

-----STREAM BUFFER-------------------------------------------------------------
     ID     Base        RdTail      RdHead      WrTail      WrHead      DataLen     BufFree
     0      0xc4a00000  0x151f40    0xa1b40     0xa1b40     0xa1b40     0           721856

-----RefParam INFO--------------------------------------------------------------
     ID    EnPred      Base   Enhance     EnIDR
      0       Yes         1         0       Yes

-----ROI INFO------------------------------------------------------------------
     ID     index    bAbsQp    Qp     width    height    startx    starty

-----Syntax INFO---------------------------------------------------------------
     ID   slcsplt   slcmode   slcsize
      0       No        N/A       N/A

-----Inter & Intra prediction INFO---------------------------------------------
     ID   profile  HWsize  VWsize  P16x16   P16x8   P8x16    P8x8   MvExt  I16x16    Inxn    Ipcm
      0        mp       5       2     Yes     Yes     Yes     Yes     Yes     Yes     Yes     Yes

-----Syntax INFO--------------------------------------------------------------
     ID   profile   EntrpyI   EntrpyP  Itrans  ptrans QMatrix   POC   DblkIdc   alpha    beta
      0        mp     cabac     cabac     4x4     4x4     No      2         0       0       0

logread:

Aug 21 19:56:32 openipc-hi3518ev100 user.info majestic[2614]: Majestic Lite for HiSilicon, version master+6f4d2d7, built on 2024-08-20
Aug 21 19:56:32 openipc-hi3518ev100 user.debug majestic[2614]: Detected Hisilicon Watchdog
Aug 21 19:56:32 openipc-hi3518ev100 user.debug majestic[2614]: Watchdog timeout set to 300 seconds
Aug 21 19:56:32 openipc-hi3518ev100 user.debug majestic[2614]: Set server path: /var/www
Aug 21 19:56:32 openipc-hi3518ev100 user.info majestic[2614]: HTTP server started on :::80
Aug 21 19:56:33 openipc-hi3518ev100 user.info majestic[2614]: Loading main configuration: /etc/majestic.yaml
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: SENSOR=ov9712
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: matched sensor config: ov9712_i2c_dc_720p.ini
Aug 21 19:56:33 openipc-hi3518ev100 user.info majestic[2614]: Using /etc/sensors/ov9712_i2c_dc_720p.ini sensor configuration file
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: App was built with MPP version: Hi3518_MPP_V1.0.B.0.
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: Current MPP version: HI_VERSION=Hi3518_MPP_V1.0.B.0
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: sensor ov9712
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]:   dev [1280x720]@0x0 15fps, BGGR
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: trying to load /usr/lib/sensors/libsns_ov9712_i2c_dc.so
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: Free MMZ mem before allocation: 23676KB
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: VB configuration:
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]:   [0]: 1413120 x 6
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: Free MMZ mem after allocation: 15384KB
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: Sensor driver loaded
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: h264 1280x720@15fps 1024kbit 15gop
Aug 21 19:56:33 openipc-hi3518ev100 user.info majestic[2614]: Loading IQ profile /etc/sensors/iq/default.ini
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]:   venc_chn: 0, vpss_grp: 0, vpss_chn: 0
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]:   venc took 4076KB of MMZ [1350]
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: MJPEG 1280x720@5fps 50q
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]:   venc_chn: 1, vpss_grp: 0, vpss_chn: 1
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]:   mpeg_enc took 916KB of MMZ [900]
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: HiSilicon SDK started
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: Free MMZ mem finally: 6228KB
Aug 21 19:56:33 openipc-hi3518ev100 user.info majestic[2614]: RTSP server started on port 554
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 1)
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 0)
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: Starting monitor for hardware sensor
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: TCP RTSP SETUP done, stream_id=0 from 10.0.0.4:64576
Aug 21 19:56:45 openipc-hi3518ev100 user.info majestic[2614]: Change mode to NIGHT
Aug 21 19:56:45 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 1)
Aug 21 19:56:45 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:56:45 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 1)
Aug 21 19:56:46 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:56:46 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:56:51 openipc-hi3518ev100 user.info majestic[2614]: Change mode to DAY
Aug 21 19:56:51 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:56:51 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 1)
Aug 21 19:56:51 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 0)
Aug 21 19:56:51 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:56:51 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:57:10 openipc-hi3518ev100 user.info majestic[2614]: Change mode to NIGHT
Aug 21 19:57:10 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 1)
Aug 21 19:57:10 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:57:10 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 1)
Aug 21 19:57:10 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:57:10 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:57:19 openipc-hi3518ev100 user.info majestic[2614]: Change mode to DAY
Aug 21 19:57:19 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:57:19 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 1)
Aug 21 19:57:19 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 0)
Aug 21 19:57:19 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:57:19 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:57:43 openipc-hi3518ev100 user.info majestic[2614]: Signal HUP received, reloading config
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 0)
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: Stopped 1 channel
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: Stopped 0 channel
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: Stop sdk Ok!
Aug 21 19:57:43 openipc-hi3518ev100 user.info majestic[2614]: Loading main configuration: /etc/majestic.yaml
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: SENSOR=ov9712
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: matched sensor config: ov9712_i2c_dc_720p.ini
Aug 21 19:57:43 openipc-hi3518ev100 user.info majestic[2614]: Using /etc/sensors/ov9712_i2c_dc_720p.ini sensor configuration file
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: App was built with MPP version: Hi3518_MPP_V1.0.B.0.
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: Current MPP version: HI_VERSION=Hi3518_MPP_V1.0.B.0
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: sensor ov9712
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]:   dev [1280x720]@0x0 15fps, BGGR
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: trying to load /usr/lib/sensors/libsns_ov9712_i2c_dc.so
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: Free MMZ mem before allocation: 23676KB
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: VB configuration:
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]:   [0]: 1413120 x 6
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: Free MMZ mem after allocation: 15384KB
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: Sensor driver loaded
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: h264 1280x720@15fps 1024kbit 15gop
Aug 21 19:57:44 openipc-hi3518ev100 user.info majestic[2614]: Loading IQ profile /etc/sensors/iq/default.ini
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]:   venc_chn: 0, vpss_grp: 0, vpss_chn: 0
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]:   venc took 4076KB of MMZ [1350]
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: MJPEG 1280x720@5fps 50q
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]:   venc_chn: 1, vpss_grp: 0, vpss_chn: 1
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]:   mpeg_enc took 916KB of MMZ [900]
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: OSD initialized
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]:   OSD took 176KB of MMZ
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: HiSilicon SDK started
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: Free MMZ mem finally: 6052KB
Aug 21 19:57:44 openipc-hi3518ev100 user.info majestic[2614]: RTSP server started on port 554
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 1)
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 0)
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: Starting monitor for hardware sensor
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:57:48 openipc-hi3518ev100 user.debug majestic[2614]: RTSP chn 0 sps_len 20
Aug 21 19:57:48 openipc-hi3518ev100 user.debug majestic[2614]: RTSP chn 0 pps_len 3
Aug 21 19:57:48 openipc-hi3518ev100 user.debug majestic[2614]: RTSP chn 0 profile_level_id 001f95
Aug 21 19:57:48 openipc-hi3518ev100 user.debug majestic[2614]: TCP RTSP SETUP done, stream_id=0 from 10.0.0.4:64728
root@openipc-hi3518ev100:~#
zerog2k commented 1 week ago

core.majestic.gz

skilurius commented 1 week ago

It is a strange issue and difficult to repeatedly reproduce (it takes around 40-80min on my test device). One very likely scenario is a regression with the modern compiler and the very old gen1 libraries, also a mismatch between uclibc and musl functions is possible.

zerog2k commented 1 week ago

If we can't really find the source of the issue, we may just need to live with frequently crashing majestic. In which case we could either try to just have a supervisor process restart majestic, or maybe something more simple, like letting /etc/init.d/S95majestic retry restarting majestic in some loop after exit with non-zero status (i.e. after crash). But this would need some sane retry/backoff limit, like 5 or 10 times - after which, we need the hardware watchdog to kick in and reboot the device. The theory is if majestic got stuck in a bad state, i.e. due to some hardware/registers/etc corruption - we would not want to try to infinitely restart majestic, but at some point backoff and just let wdt timer expire and reset the cpu.

But I think this brings us back to a secondary concern, which is that currently, the wdt does not work for some reason, but as @Flole998 pointed out, we need to uncomment nodaemon param of insmod wdt.ko #nodeamon=1 in /usr/bin/load_hisilicon script https://github.com/OpenIPC/firmware/blob/master/general/package/hisilicon-osdrv-hi3516cv100/files/script/load_hisilicon#L348

I'm hoping this is a common problem (wdt not working) with all hi3516cv100, such that submitting a PR to fix this will work for everyone with this cpu family.