Open zerog2k opened 1 month ago
I am seeing the same thing. The watchdog seems to be broken on other Chipsets aswell, looks like it gets disabled on crash, dmesg
suggests that.
The watchdog has been effectively disabled on all HiSilicon platforms. If you want to enable it again, re-add the nodeamon=1
parameter for the insmod wdt
command in load_hisilicon. For me that at least causes it to automatically reboot.
I got access to a device for testing (HI3518CV100+IMX138), but unfortunately could not reproduce the rtsp crash.
I tried three sessions with 1hour runtime each: mpv rtsp://root:12345@192.168.1.10/stream=0
root@openipc-hi3518cv100:~# cat /proc/umap/h264e
[H264E] Version: [Hi3518_MPP_V1.0.B.0 ], Build Time[Nov 19 2015, 16:37:03]
-----MODULE PARAM--------------------------------------------------------------
ContiBuffer H264eVBSource H264eRcnEqualRef H264eMiniBufMode
0 1 0 0
-----CHN ATTR------------------------------------------------------------------
ID MaxWidth MaxHeight PicWidth PicHeight profile MainStr RefMode BufSize ByFrame MaxStrCnt PoolId
0 1280 720 1280 720 mp Yes 1X 691200 Yes 200 1
-----PICTURE INFO--------------------------------------------------------------
ID EncdStart EncdSucceed Lost Disc Skip BufLeak PSkip Recode RlsStr UnrdStr GetVBFail
0 61869 61869 30934 0 30934 0 0 0 61869 0 0
-----STREAM BUFFER-------------------------------------------------------------
ID Base RdTail RdHead WrTail WrHead DataLen BufFree
0 0xc2e00000 0x6c2c0 0x6c2c0 0x6c2c0 0x6c2c0 0 692160
-----RefParam INFO--------------------------------------------------------------
ID EnPred Base Enhance EnIDR
0 Yes 1 0 Yes
-----ROI INFO------------------------------------------------------------------
ID index bAbsQp Qp width height startx starty
-----Syntax INFO---------------------------------------------------------------
ID slcsplt slcmode slcsize
0 No N/A N/A
-----Inter & Intra prediction INFO---------------------------------------------
ID profile HWsize VWsize P16x16 P16x8 P8x16 P8x8 MvExt I16x16 Inxn Ipcm
0 mp 5 2 Yes Yes Yes Yes Yes Yes Yes Yes
-----Syntax INFO--------------------------------------------------------------
ID profile EntrpyI EntrpyP Itrans ptrans QMatrix POC DblkIdc alpha beta
0 mp cabac cabac 4x4 4x4 No 2 0 0 0
Try leaving it running for longer. I am seeing this on
# ipcinfo -cs
hi3516ev300
imx335
and the camera has an uptime of about 20 hours now. So it does happen eventually, but it is not really plannable.
anyone from majestic dev team get a chance to analyze the coredump? with closed source and no symbols, it would be exceedingly hard for us to guess what is going on, and possibly theorize ways to reproduce it more reliably
The core dump is unfortunately missing some function pointer. However, the problem seems to be the presumed buffer leak, it moves the datastream after a while which leads to the segmentation fault. The H264 buffer size was slightly increased on the current release, please check it and report back.
root@openipc-hi3518ev100:~# majestic
20:33:56 < majestic> [ main] main@237 Majestic Lite for HiSilicon, version master+d3f2c7a, built on 2024-08-08
20:33:57 < majestic> [watchdog] watchdog_start@94 Detected Hisilicon Watchdog
20:33:57 < majestic> [watchdog] watchdog_start@115 Watchdog timeout set to 60 seconds
20:33:57 < majestic> [ httpd] start_web_server@383 Set server path: /var/www
20:33:57 < majestic> [ httpd] new_http_server@371 HTTP server started on :::80
20:33:57 < majestic> [ main] init_sdk@66 Loading main configuration: /etc/majestic.yaml
20:33:57 < majestic> [ sdk] try_sensor_config@3276 SENSOR=ov9712
20:33:57 < majestic> [ sdk] find_sensor_config@3244 matched sensor config: ov9712_i2c_dc_720p.ini
20:33:57 < majestic> [ sdk] start_sdk@380 Using /etc/sensors/ov9712_i2c_dc_720p.ini sensor configuration file
20:33:57 < majestic> [ sdk] start_sdk@385 App was built with MPP version: Hi3518_MPP_V1.0.B.0.
20:33:57 < majestic> [ sdk] start_sdk@390 Current MPP version: HI_VERSION=Hi3518_MPP_V1.0.B.0
20:33:57 < majestic> [ sdk] start_sdk@418 sensor ov9712
20:33:57 < majestic> [ sdk] start_sdk@427 dev [1280x720]@0x0 25fps, BGGR
20:33:57 < majestic> [ sensor] try_to_load@19 trying to load /usr/lib/sensors/libsns_ov9712_i2c_dc.so
20:33:57 < majestic> [ hal] hisi_free_mem_ex@753 Free MMZ mem before allocation: 19580KB
20:33:57 < majestic> [ sdk] dump_vb_configuration@2174 VB configuration:
20:33:57 < majestic> [ sdk] dump_vb_configuration@2177 [0]: 1413120 x 6
20:33:57 < majestic> [ hal] hisi_free_mem_ex@753 Free MMZ mem after allocation: 11288KB
20:33:57 < majestic> [ sdk] init_sensor@2369 Sensor driver loaded
20:33:57 < majestic> [ sdk] log_venc_chn@1253 h264 1280x720@15fps 1024kbit 15gop
20:33:57 < iq_monitor> [ sdk] monitor_thread@307 Loading IQ profile /etc/sensors/iq/default.ini
20:33:58 < majestic> [ sdk] create_vpss_chn@993 venc_chn: 0, vpss_grp: 0, vpss_chn: 0
20:33:58 < majestic> [ sdk] start_video@1917 venc took 3444KB of MMZ [720]
20:33:58 < majestic> [ sdk] log_venc_chn@1253 MJPEG 1280x720@5fps 25q
20:33:58 < majestic> [ sdk] create_vpss_chn@993 venc_chn: 1, vpss_grp: 0, vpss_chn: 1
20:33:58 < majestic> [ sdk] start_mjpeg@1378 mpeg_enc took 916KB of MMZ [900]
20:33:58 < majestic> [ osd] init_osd@103 OSD initialized
20:33:58 < majestic> [ sdk] start_sdk@515 OSD took 176KB of MMZ
20:33:58 < majestic> [ sdk] start_sdk@542 HiSilicon SDK started
20:33:58 < majestic> [ hal] hisi_free_mem_ex@753 Free MMZ mem finally: 2588KB
20:33:58 < majestic> [ rtsp] rtsp_init@28 RTSP server started on port 554
20:33:58 < majestic> [ gpio] set_gpio@26 set_gpio(76, 0)
20:33:58 < majestic> [ gpio] set_gpio@26 set_gpio(79, 1)
20:33:58 < majestic> [ gpio] set_gpio@26 set_gpio(73, 0)
20:33:58 < majestic> [ night] setup_gpio_listener@189 Starting monitor for hardware sensor
20:33:58 < majestic> [ gpio] set_gpio@26 set_gpio(76, 0)
20:33:58 < majestic> [ gpio] set_gpio@26 set_gpio(79, 0)
20:33:58 < majestic> [ client] RtspClient_before@403 RTSP OPTIONS rtsp://10.0.3.78/stream=0 CSeq=1 from 10.0.0.4:55164
20:33:58 < majestic> [ client] RtspClient_before@403 RTSP OPTIONS rtsp://10.0.3.78/stream=0 CSeq=2 from 10.0.0.4:55164
20:33:58 < majestic> [ client] RtspClient_before@403 RTSP DESCRIBE rtsp://10.0.3.78/stream=0 CSeq=3 from 10.0.0.4:55164
20:33:58 < majestic> [ client] RtspClient_before@403 RTSP SETUP rtsp://10.0.3.78/stream=0/video CSeq=4 from 10.0.0.4:55164
20:33:58 < majestic> [ client] RtspClient_setup@314 TCP RTSP SETUP done, stream_id=0 from 10.0.0.4:55164
20:33:58 < majestic> [ client] RtspClient_before@403 RTSP PLAY rtsp://10.0.3.78/stream=0 CSeq=5 from 10.0.0.4:55164
Segmentation fault (core dumped)
root@openipc-hi3518ev100:~#
majestic.2440.11.1723254891.core.gz
also updated majestic.yaml (as I was disabling second video1 stream)
system:
webPort: 80
httpsPort: 443
logLevel: debug
buffer: 2048
plugins: false
isp:
antiFlicker: disabled
blkCnt: 6
dis: false
image:
mirror: false
flip: false
rotate: 0
contrast: 50
hue: 50
saturation: 50
luminance: 50
video0:
enabled: true
codec: h264
fps: 15
bitrate: 1024
rcMode: vbr
gopSize: 1.0
gopMode: normal
size: 1280x720
sliceUnits: 0
video1:
enabled: false
codec: h264
size: 640x360
fps: 10
bitrate: 512
gopMode: normal
rcMode: vbr
sliceUnits: 0
gopSize: 1.0
jpeg:
enabled: true
qfactor: 25
fps: 5
osd:
enabled: true
font: "/usr/share/fonts/truetype/UbuntuMono-Regular.ttf"
template: "%F %T"
posX: 16
posY: 16
audio:
enabled: false
volume: 30
srate: 8000
codec: opus
outputEnabled: false
outputVolume: 30
rtsp:
enabled: true
port: 554
nightMode:
colorToGray: true
irCutSingleInvert: false
lightMonitor: true
lightSensorInvert: false
backlightPin: 73
irCutPin1: 76
irCutPin2: 79
lightSensorPin: 77
monitorDelay: 0
motionDetect:
enabled: true
visualize: false
debug: false
records:
enabled: false
path: "/mnt/mmcblk0p1/%F/%H.mp4"
maxUsage: 95
outgoing:
enabled: false
watchdog:
enabled: true
timeout: 60
hls:
enabled: false
onvif:
enabled: true
For me it hasn't crashed yet since 32 hours, however I've seen some majestic versions work better than others in the past, so this could be coincidence. I'll give you an update in a few days how it's going.
It looks like it's fixed on hi3516ev300 but my hi3518ev100 keeps crashing.
The buffer size has now been increased to width * height * 1.5
Unfortunately I'm still seeing crashes, now also on hi3516ev300 again. I'm running master+11648f1, 2024-08-14.
@Flole998 same here w/ 8-13 build of majestic if i do not connect any rtsp streamer, it will stay up for many days (tested 5) but if I connect rtsp streamer, it will segfault after a few hours at most
Curious: I'm using Blue Iris 5. What rtsp streamer are you using? also Im using osmem=40M, (mmz_size: 24M) what are you using?
also @skilurius thanks for looking into this. Not sure if it's helpful but here is some data from OEM firmware on my device, which has been up for over 19 days streaming rtsp without issue.
# cat /proc/umap/h264e
[H264E] Version: [Hi3518_MPP_V1.0.A.1 ], Build Time[Aug 24 2015, 22:43:00]
-----MODULE PARAM--------------------------------------------------------------
ContiBuffer H264eVBSource H264eRcnEqualRef H264eMiniBufMode
0 1 0 0
-----CHN ATTR------------------------------------------------------------------
ID MaxWidth MaxHeight PicWidth PicHeight profile MainStr RefMode BufSize ByFrame MaxStrCnt PoolId
0 1280 720 1280 720 mp Yes 1X 1382400 Yes 200 4
1 352 288 352 288 mp Yes 1X 152064 Yes 200 5
-----PICTURE INFO--------------------------------------------------------------
ID EncdStart EncdSucceed Lost Disc Skip BufLeak PSkip Recode RlsStr UnrdStr GetVBFail
0 428460 428459 285640 0 285640 0 0 1 428459 0 0
1 428459 428459 285640 0 285640 0 0 0 428459 0 0
-----STREAM BUFFER-------------------------------------------------------------
ID Base RdTail RdHead WrTail WrHead DataLen BufFree
0 0xc5e00000 0x1df40 0x1df40 0x1df40 0x1df40 0 1384384
1 0xc4580000 0x11340 0x11340 0x11340 0x11340 0 155584
-----RefParam INFO--------------------------------------------------------------
ID EnPred Base Enhance EnIDR
0 Yes 1 0 Yes
1 Yes 1 0 Yes
-----ROI INFO------------------------------------------------------------------
ID index bAbsQp Qp width height startx starty
-----Syntax INFO---------------------------------------------------------------
ID slcsplt slcmode slcsize
0 No N/A N/A
1 No N/A N/A
-----Inter & Intra prediction INFO---------------------------------------------
ID profile HWsize VWsize P16x16 P16x8 P8x16 P8x8 MvExt I16x16 Inxn Ipcm
0 mp 5 2 Yes Yes Yes Yes Yes Yes Yes Yes
1 mp 5 2 Yes Yes Yes Yes Yes Yes Yes Yes
-----Syntax INFO--------------------------------------------------------------
ID profile EntrpyI EntrpyP Itrans ptrans QMatrix POC DblkIdc alpha beta
0 mp cabac cabac 4x4 4x4 No 2 0 0 0
1 mp cabac cabac 4x4 4x4 No 2 0 0 0
on my majestic test device (same hardware):
root@openipc-hi3518ev100:~# cat /proc/umap/h264e
[H264E] Version: [Hi3518_MPP_V1.0.B.0 ], Build Time[Nov 19 2015, 16:37:03]
-----MODULE PARAM--------------------------------------------------------------
ContiBuffer H264eVBSource H264eRcnEqualRef H264eMiniBufMode
0 1 0 0
-----CHN ATTR------------------------------------------------------------------
ID MaxWidth MaxHeight PicWidth PicHeight profile MainStr RefMode BufSize ByFrame MaxStrCnt PoolId
0 1280 720 1280 720 mp Yes 1X 1382400 Yes 200 1
-----PICTURE INFO--------------------------------------------------------------
ID EncdStart EncdSucceed Lost Disc Skip BufLeak PSkip Recode RlsStr UnrdStr GetVBFail
0 840604 840600 0 0 0 0 0 4 840600 0 0
-----STREAM BUFFER-------------------------------------------------------------
ID Base RdTail RdHead WrTail WrHead DataLen BufFree
0 0xc3a00000 0xdb3c0 0xdb3c0 0xdb3c0 0xdb3c0 0 1384384
-----RefParam INFO--------------------------------------------------------------
ID EnPred Base Enhance EnIDR
0 Yes 1 0 Yes
-----ROI INFO------------------------------------------------------------------
ID index bAbsQp Qp width height startx starty
-----Syntax INFO---------------------------------------------------------------
ID slcsplt slcmode slcsize
0 No N/A N/A
-----Inter & Intra prediction INFO---------------------------------------------
ID profile HWsize VWsize P16x16 P16x8 P8x16 P8x8 MvExt I16x16 Inxn Ipcm
0 mp 5 2 Yes Yes Yes Yes Yes Yes Yes Yes
-----Syntax INFO--------------------------------------------------------------
ID profile EntrpyI EntrpyP Itrans ptrans QMatrix POC DblkIdc alpha beta
0 mp cabac cabac 4x4 4x4 No 2 0 0 0
ok and here is one after majestic crashed after just about 20 minutes of running stream:
root@openipc-hi3518ev100:~# cat /proc/umap/h264e
[H264E] Version: [Hi3518_MPP_V1.0.B.0 ], Build Time[Nov 19 2015, 16:37:03]
-----MODULE PARAM--------------------------------------------------------------
ContiBuffer H264eVBSource H264eRcnEqualRef H264eMiniBufMode
0 1 0 0
-----CHN ATTR------------------------------------------------------------------
ID MaxWidth MaxHeight PicWidth PicHeight profile MainStr RefMode BufSize ByFrame MaxStrCnt PoolId
0 1280 720 1280 720 mp Yes 1X 1382400 Yes 200 1
-----PICTURE INFO--------------------------------------------------------------
ID EncdStart EncdSucceed Lost Disc Skip BufLeak PSkip Recode RlsStr UnrdStr GetVBFail
0 2269 2269 15064 0 0 15064 0 0 2069 200 0
-----STREAM BUFFER-------------------------------------------------------------
ID Base RdTail RdHead WrTail WrHead DataLen BufFree
0 0xc4a00000 0x151f40 0xa1b40 0xa1b40 0xa1b40 0 721856
-----RefParam INFO--------------------------------------------------------------
ID EnPred Base Enhance EnIDR
0 Yes 1 0 Yes
-----ROI INFO------------------------------------------------------------------
ID index bAbsQp Qp width height startx starty
-----Syntax INFO---------------------------------------------------------------
ID slcsplt slcmode slcsize
0 No N/A N/A
-----Inter & Intra prediction INFO---------------------------------------------
ID profile HWsize VWsize P16x16 P16x8 P8x16 P8x8 MvExt I16x16 Inxn Ipcm
0 mp 5 2 Yes Yes Yes Yes Yes Yes Yes Yes
-----Syntax INFO--------------------------------------------------------------
ID profile EntrpyI EntrpyP Itrans ptrans QMatrix POC DblkIdc alpha beta
0 mp cabac cabac 4x4 4x4 No 2 0 0 0
logread:
Aug 21 19:56:32 openipc-hi3518ev100 user.info majestic[2614]: Majestic Lite for HiSilicon, version master+6f4d2d7, built on 2024-08-20
Aug 21 19:56:32 openipc-hi3518ev100 user.debug majestic[2614]: Detected Hisilicon Watchdog
Aug 21 19:56:32 openipc-hi3518ev100 user.debug majestic[2614]: Watchdog timeout set to 300 seconds
Aug 21 19:56:32 openipc-hi3518ev100 user.debug majestic[2614]: Set server path: /var/www
Aug 21 19:56:32 openipc-hi3518ev100 user.info majestic[2614]: HTTP server started on :::80
Aug 21 19:56:33 openipc-hi3518ev100 user.info majestic[2614]: Loading main configuration: /etc/majestic.yaml
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: SENSOR=ov9712
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: matched sensor config: ov9712_i2c_dc_720p.ini
Aug 21 19:56:33 openipc-hi3518ev100 user.info majestic[2614]: Using /etc/sensors/ov9712_i2c_dc_720p.ini sensor configuration file
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: App was built with MPP version: Hi3518_MPP_V1.0.B.0.
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: Current MPP version: HI_VERSION=Hi3518_MPP_V1.0.B.0
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: sensor ov9712
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: dev [1280x720]@0x0 15fps, BGGR
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: trying to load /usr/lib/sensors/libsns_ov9712_i2c_dc.so
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: Free MMZ mem before allocation: 23676KB
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: VB configuration:
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: [0]: 1413120 x 6
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: Free MMZ mem after allocation: 15384KB
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: Sensor driver loaded
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: h264 1280x720@15fps 1024kbit 15gop
Aug 21 19:56:33 openipc-hi3518ev100 user.info majestic[2614]: Loading IQ profile /etc/sensors/iq/default.ini
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: venc_chn: 0, vpss_grp: 0, vpss_chn: 0
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: venc took 4076KB of MMZ [1350]
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: MJPEG 1280x720@5fps 50q
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: venc_chn: 1, vpss_grp: 0, vpss_chn: 1
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: mpeg_enc took 916KB of MMZ [900]
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: HiSilicon SDK started
Aug 21 19:56:33 openipc-hi3518ev100 user.debug majestic[2614]: Free MMZ mem finally: 6228KB
Aug 21 19:56:33 openipc-hi3518ev100 user.info majestic[2614]: RTSP server started on port 554
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 1)
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 0)
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: Starting monitor for hardware sensor
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:56:34 openipc-hi3518ev100 user.debug majestic[2614]: TCP RTSP SETUP done, stream_id=0 from 10.0.0.4:64576
Aug 21 19:56:45 openipc-hi3518ev100 user.info majestic[2614]: Change mode to NIGHT
Aug 21 19:56:45 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 1)
Aug 21 19:56:45 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:56:45 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 1)
Aug 21 19:56:46 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:56:46 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:56:51 openipc-hi3518ev100 user.info majestic[2614]: Change mode to DAY
Aug 21 19:56:51 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:56:51 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 1)
Aug 21 19:56:51 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 0)
Aug 21 19:56:51 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:56:51 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:57:10 openipc-hi3518ev100 user.info majestic[2614]: Change mode to NIGHT
Aug 21 19:57:10 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 1)
Aug 21 19:57:10 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:57:10 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 1)
Aug 21 19:57:10 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:57:10 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:57:19 openipc-hi3518ev100 user.info majestic[2614]: Change mode to DAY
Aug 21 19:57:19 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:57:19 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 1)
Aug 21 19:57:19 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 0)
Aug 21 19:57:19 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:57:19 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:57:43 openipc-hi3518ev100 user.info majestic[2614]: Signal HUP received, reloading config
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 0)
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: Stopped 1 channel
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: Stopped 0 channel
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: Stop sdk Ok!
Aug 21 19:57:43 openipc-hi3518ev100 user.info majestic[2614]: Loading main configuration: /etc/majestic.yaml
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: SENSOR=ov9712
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: matched sensor config: ov9712_i2c_dc_720p.ini
Aug 21 19:57:43 openipc-hi3518ev100 user.info majestic[2614]: Using /etc/sensors/ov9712_i2c_dc_720p.ini sensor configuration file
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: App was built with MPP version: Hi3518_MPP_V1.0.B.0.
Aug 21 19:57:43 openipc-hi3518ev100 user.debug majestic[2614]: Current MPP version: HI_VERSION=Hi3518_MPP_V1.0.B.0
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: sensor ov9712
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: dev [1280x720]@0x0 15fps, BGGR
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: trying to load /usr/lib/sensors/libsns_ov9712_i2c_dc.so
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: Free MMZ mem before allocation: 23676KB
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: VB configuration:
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: [0]: 1413120 x 6
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: Free MMZ mem after allocation: 15384KB
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: Sensor driver loaded
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: h264 1280x720@15fps 1024kbit 15gop
Aug 21 19:57:44 openipc-hi3518ev100 user.info majestic[2614]: Loading IQ profile /etc/sensors/iq/default.ini
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: venc_chn: 0, vpss_grp: 0, vpss_chn: 0
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: venc took 4076KB of MMZ [1350]
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: MJPEG 1280x720@5fps 50q
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: venc_chn: 1, vpss_grp: 0, vpss_chn: 1
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: mpeg_enc took 916KB of MMZ [900]
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: OSD initialized
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: OSD took 176KB of MMZ
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: HiSilicon SDK started
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: Free MMZ mem finally: 6052KB
Aug 21 19:57:44 openipc-hi3518ev100 user.info majestic[2614]: RTSP server started on port 554
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 1)
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(73, 0)
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: Starting monitor for hardware sensor
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(76, 0)
Aug 21 19:57:44 openipc-hi3518ev100 user.debug majestic[2614]: set_gpio(79, 0)
Aug 21 19:57:48 openipc-hi3518ev100 user.debug majestic[2614]: RTSP chn 0 sps_len 20
Aug 21 19:57:48 openipc-hi3518ev100 user.debug majestic[2614]: RTSP chn 0 pps_len 3
Aug 21 19:57:48 openipc-hi3518ev100 user.debug majestic[2614]: RTSP chn 0 profile_level_id 001f95
Aug 21 19:57:48 openipc-hi3518ev100 user.debug majestic[2614]: TCP RTSP SETUP done, stream_id=0 from 10.0.0.4:64728
root@openipc-hi3518ev100:~#
It is a strange issue and difficult to repeatedly reproduce (it takes around 40-80min on my test device). One very likely scenario is a regression with the modern compiler and the very old gen1 libraries, also a mismatch between uclibc and musl functions is possible.
If we can't really find the source of the issue, we may just need to live with frequently crashing majestic. In which case we could either try to just have a supervisor process restart majestic, or maybe something more simple, like letting /etc/init.d/S95majestic retry restarting majestic in some loop after exit with non-zero status (i.e. after crash). But this would need some sane retry/backoff limit, like 5 or 10 times - after which, we need the hardware watchdog to kick in and reboot the device. The theory is if majestic got stuck in a bad state, i.e. due to some hardware/registers/etc corruption - we would not want to try to infinitely restart majestic, but at some point backoff and just let wdt timer expire and reset the cpu.
But I think this brings us back to a secondary concern, which is that currently, the wdt does not work for some reason, but as @Flole998 pointed out, we need to uncomment nodaemon
param of insmod wdt.ko #nodeamon=1
in /usr/bin/load_hisilicon script
https://github.com/OpenIPC/firmware/blob/master/general/package/hisilicon-osdrv-hi3516cv100/files/script/load_hisilicon#L348
I'm hoping this is a common problem (wdt not working) with all hi3516cv100, such that submitting a PR to fix this will work for everyone with this cpu family.
make BOARD=hi3518ev100_lite
) + sysupgradeIssue description
Majestic crashes (segfault) after some time (minutes to hours) streaming rtsp h264. (No crashes if streaming only mjpeg.)
Another observation is that when majestic dies, even though watchdog is enabled, the camera never resets. OS stays up and available but majestic is gone. (I'm certain watchdog was working properly on majestic/openipc versions from at least as recent as February. Majestic did crash, but I believe watchdog was keeping this from being a terminal issue.)
might be similar to #159
Expectations
no crash
Steps to reproduce
rtsp://CAMERAIP/?stream=0
Information to attach
majestic.4731.11.1720275013.core.zip majestic.yaml.zip