segmentation fault while trying to use hls on RPI Zero

[MittaAxt]

Hi again,

I've build picam on my RPI ZeroW (took some time) and also the node-rtsp. Starting Picam with rtspout while the node server is running works fine. Running picam with hls output produces an error "Segmentation fault" and picam closes. As the node-rtsp is causing a drop of ~5fps (while having only ~22-25fps on full hd and ~30 at 1280x720) I wanted to try the nginx HLS variant, but it is failing directly at the startup. Same if I try to run with --tcpout.

./picam --verbose -w 1280 -h 720 -o /run/shm/hls video_width=1280 video_height=720 video_fps=30.000000 video_pts_step=3000 video_gop_size=30 video_hflip=0 video_vflip=0 video_bitrate=3000000 video_avc_profile=high video_avc_level=4.1 video_qp_min=-1 video_qp_max=-1 video_qp_initial=-1 video_slice_dquant=-1 video_brightness=0 video_contrast=0 video_saturation=0 video_sharpness=0 video_autofocus_mode=continuous video_lens_position=-1.000000 alsa_dev=hw:0,0 audio_channels=1 audio_sample_rate=48000 audio_bitrate=40000 audio_volume_multiply=1.000000 ng_thresh_volume=1.000000 ng_attack_time=0.200000 ng_hold_time=1.000000 ng_release_time=0.500000 is_hlsout_enabled=1 is_hls_encryption_enabled=0 hls_keyframes_per_segment=1 hls_number_of_segments=3 hls_encryption_key_uri=stream.key hls_encryption_key=0x75b0a81de17487c88a47507a7e1fdf73 hls_encryption_iv=0x000102030405060708090a0b0c0d0e0f hls_output_dir=/run/shm/hls rtsp_enabled=0 rtsp_video_control_path=/tmp/node_rtsp_rtmp_videoControl rtsp_audio_control_path=/tmp/node_rtsp_rtmp_audioControl rtsp_video_data_path=/tmp/node_rtsp_rtmp_videoData rtsp_audio_data_path=/tmp/node_rtsp_rtmp_audioData tcp_enabled=0 tcp_output_dest= auto_exposure_enabled=0 auto_exposure_threshold=5.000000 is_vfr_enabled=0 white_balance=auto exposure_control=auto awb_red_gain=0.000000 awb_blue_gain=0.000000 metering=average manual_exposure_compensation=0 exposure_compensation=0.000000 manual_exposure_aperture=0 exposure_aperture=0 manual_exposure_shutter_speed=0 exposure_shutter_speed=0 manual_exposure_sensitivity=0 exposure_sensitivity=0 roi_left=0.000000 roi_top=0.000000 roi_width=1.000000 roi_height=1.000000 min_fps=-1.000000 max_fps=-1.000000 is_timestamp_enabled=0 timestamp_format=%a %b %d %l:%M:%S %p timestamp_layout=15 timestamp_horizontal_margin=10 timestamp_vertical_margin=10 is_timestamp_abs_pos_enabled=0 timestamp_pos_x=0 timestamp_pos_y=0 timestamp_text_align=1 timestamp_font_name=FreeMono:style=Bold timestamp_font_file= timestamp_font_face_index=(null) timestamp_font_points=14.000000 timestamp_font_dpi=96 timestamp_color=ffffff timestamp_stroke_color=000000 timestamp_stroke_width=1 timestamp_letter_spacing=0 is_preview_enabled=0 is_previewrect_enabled=0 preview_x=0 preview_y=0 preview_width=0 preview_height=0 preview_hdmi=0 is_audio_preview_enabled=0 audio_preview_dev=plughw:0,0 record_buffer_keyframes=5 state_dir=state hooks_dir=hooks failed to enable HDR mode opening ALSA device for capture: hw:0,0 error: cannot open audio capture device 'hw:0,0': No such file or directory hint: specify correct ALSA device with '--alsadev ' warning: audio capturing is disabled disable_audio_capturing: 1 configuring devices Running without preview window Opening camera... [0:48:39.884609961] [778] INFO Camera camera_manager.cpp:299 libcamera v0.0.4+22-923f5d70 [0:48:40.054999667] [780] INFO RPI raspberrypi.cpp:1476 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media3 and ISP device /dev/media0 Acquired camera -1096228376 Configuring video... [0:48:40.061973614] [778] INFO Camera camera.cpp:1028 configuring streams: (0) 1280x720-YUV420 [0:48:40.065551587] [780] INFO RPI raspberrypi.cpp:851 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 1920x1080-SGBRG10_1X10 - Selected unicam format: 1920x1080-pGAA Camera streams configured Available controls: AeMeteringMode : [0..3] ExposureTime : [118..969249] Brightness : [-1.000000..1.000000] AnalogueGain : [1.000000..63.937500] AeExposureMode : [0..3] FrameDurationLimits : [32660..969367] ColourCorrectionMatrix : [-16.000000..16.000000] Saturation : [0.000000..32.000000] ColourGains : [0.000000..32.000000] NoiseReductionMode : [0..4] ExposureValue : [-8.000000..8.000000] AeConstraintMode : [0..3] AwbEnable : [false..true] AwbMode : [0..7] AeEnable : [false..true] Contrast : [0.000000..32.000000] ScalerCrop : [(348, 434)/64x64..(348, 434)/1928x1080] Sharpness : [0.000000..16.000000] Buffers allocated and mapped preview: startPreview begin preview: startPreview end Video setup complete Opened VideoEncoder on /dev/video11 as fd 25 Got 6 output buffers Got 12 capture buffers Codec streaming started Requests created Using crop h/H cfr frame_time=33333 camera_set_metering_mode: average camera_set_exposure_control: auto camera_set_white_balance: auto enable AWB: 0 camera_set_custom_awb_gains: red=0.0, blue=0.0 camera_set_brightness: 0.0 camera_set_contrast: 1.0 camera_set_saturation: 1.0 camera_set_sharpness: 0.0 skipping autofocus because it is not supported by the camera Camera started! audio setup sample_rate: 48000 nb_samples: 1024 format: 1 audio_codec_ctx->ch_layout: 4294967297 av_frame->channel_layout: 4294967297 buffer_size=2048 allocated 2048 bytes for audio samples audio_pts_step: 1920 audio device configured Segmentation fault

any idea what might cause this?

[iizukanao]

Unfortunately, I am unable to reproduce the segmentation fault on my Raspberry Pi 4, 3, and 2. Can you use the gdb command to see which line is causing the segmentation fault?

[MittaAxt]

haven't used gdb before. Hope this helps, if not please let me know what exactly I should do in gdb

Starting program: /home/myuser/picam/picam --verbose -w 1280 -h 720 -o /run/shm/hls [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction. 0xb678e5c8 in ?? () from /lib/arm-linux-gnueabihf/libcrypto.so.1.1

apologies for the late reply I was on vacation

[iizukanao]

Thank you for your cooperation in the investigation.

It appears that SIGILL is a normal operation of OpenSSL (libcrypto). (ref. https://forums.raspberrypi.com/viewtopic.php?t=15132, https://www.openssl.org/docs/faq.html#PROG17)

In the gdb session, please enter handle SIGILL nostop noprint before run.

[MittaAxt]

here you go

gdb picam GNU gdb (Raspbian 10.1-1.7) 10.1.90.20210103-git Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-linux-gnueabihf". Type "show configuration" for configuration details. For bug reporting instructions, please see: https://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from picam... (No debugging symbols found in picam) (gdb) handle SIGILL nostop noprint Signal Stop Print Pass to program Description SIGILL No No Yes Illegal instruction (gdb) run --verbose -w 1280 -h 720 -o /run/shm/hls Starting program: /home/camadmin/picam/picam --verbose -w 1280 -h 720 -o /run/shm/hls [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1". video_width=1280 video_height=720 video_fps=30.000000 video_pts_step=3000 video_gop_size=30 video_hflip=0 video_vflip=0 video_bitrate=3000000 video_avc_profile=high video_avc_level=4.1 video_qp_min=-1 video_qp_max=-1 video_qp_initial=-1 video_slice_dquant=-1 video_brightness=0 video_contrast=0 video_saturation=0 video_sharpness=0 video_autofocus_mode=continuous video_lens_position=-1.000000 alsa_dev=hw:0,0 audio_channels=1 audio_sample_rate=48000 audio_bitrate=40000 audio_volume_multiply=1.000000 ng_thresh_volume=1.000000 ng_attack_time=0.200000 ng_hold_time=1.000000 ng_release_time=0.500000 is_hlsout_enabled=1 is_hls_encryption_enabled=0 hls_keyframes_per_segment=1 hls_number_of_segments=3 hls_encryption_key_uri=stream.key hls_encryption_key=0x75b0a81de17487c88a47507a7e1fdf73 hls_encryption_iv=0x000102030405060708090a0b0c0d0e0f hls_output_dir=/run/shm/hls rtsp_enabled=0 rtsp_video_control_path=/tmp/node_rtsp_rtmp_videoControl rtsp_audio_control_path=/tmp/node_rtsp_rtmp_audioControl rtsp_video_data_path=/tmp/node_rtsp_rtmp_videoData rtsp_audio_data_path=/tmp/node_rtsp_rtmp_audioData tcp_enabled=0 tcp_output_dest= auto_exposure_enabled=0 auto_exposure_threshold=5.000000 is_vfr_enabled=0 white_balance=auto exposure_control=auto awb_red_gain=0.000000 awb_blue_gain=0.000000 metering=average manual_exposure_compensation=0 exposure_compensation=0.000000 manual_exposure_aperture=0 exposure_aperture=0 manual_exposure_shutter_speed=0 exposure_shutter_speed=0 manual_exposure_sensitivity=0 exposure_sensitivity=0 roi_left=0.000000 roi_top=0.000000 roi_width=1.000000 roi_height=1.000000 min_fps=-1.000000 max_fps=-1.000000 is_timestamp_enabled=0 timestamp_format=%a %b %d %l:%M:%S %p timestamp_layout=15 timestamp_horizontal_margin=10 timestamp_vertical_margin=10 is_timestamp_abs_pos_enabled=0 timestamp_pos_x=0 timestamp_pos_y=0 timestamp_text_align=1 timestamp_font_name=FreeMono:style=Bold timestamp_font_file= timestamp_font_face_index=(null) timestamp_font_points=14.000000 timestamp_font_dpi=96 timestamp_color=ffffff timestamp_stroke_color=000000 timestamp_stroke_width=1 timestamp_letter_spacing=0 is_preview_enabled=0 is_previewrect_enabled=0 preview_x=0 preview_y=0 preview_width=0 preview_height=0 preview_hdmi=0 is_audio_preview_enabled=0 audio_preview_dev=plughw:0,0 record_buffer_keyframes=5 state_dir=state hooks_dir=hooks failed to enable HDR mode created directory: ./rec/tmp created HLS directory: /run/shm/hls [New Thread 0xb539e1c0 (LWP 732)] opening ALSA device for capture: hw:0,0 error: cannot open audio capture device 'hw:0,0': No such file or directory hint: specify correct ALSA device with '--alsadev ' warning: audio capturing is disabled disable_audio_capturing: 1 configuring devices Running without preview window Opening camera... [0:33:55.228690053] [728] INFO Camera camera_manager.cpp:299 libcamera v0.0.4+22-923f5d70 [New Thread 0xb49ff1c0 (LWP 733)] [New Thread 0xb3fff1c0 (LWP 734)] [New Thread 0xb37fe1c0 (LWP 735)] [0:33:55.674744136] [733] INFO RPI raspberrypi.cpp:1476 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media3 and ISP device /dev/media0 Acquired camera -1090526808 Configuring video... [0:33:55.682773922] [728] INFO Camera camera.cpp:1028 configuring streams: (0) 1280x720-YUV420 [0:33:55.685565847] [733] INFO RPI raspberrypi.cpp:851 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 1920x1080-SGBRG10_1X10 - Selected unicam format: 1920x1080-pGAA Camera streams configured Available controls: AeEnable : [false..true] AnalogueGain : [1.000000..63.937500] ExposureTime : [118..969249] Brightness : [-1.000000..1.000000] FrameDurationLimits : [32660..969367] AeMeteringMode : [0..3] Saturation : [0.000000..32.000000] ColourCorrectionMatrix : [-16.000000..16.000000] ColourGains : [0.000000..32.000000] Sharpness : [0.000000..16.000000] ScalerCrop : [(348, 434)/64x64..(348, 434)/1928x1080] NoiseReductionMode : [0..4] ExposureValue : [-8.000000..8.000000] AwbEnable : [false..true] AwbMode : [0..7] Contrast : [0.000000..32.000000] AeExposureMode : [0..3] AeConstraintMode : [0..3] Buffers allocated and mapped preview: startPreview begin [New Thread 0xb28111c0 (LWP 736)] preview: startPreview end Video setup complete Opened VideoEncoder on /dev/video11 as fd 25 Got 6 output buffers Got 12 capture buffers Codec streaming started [New Thread 0xb1a901c0 (LWP 737)] [New Thread 0xb128f1c0 (LWP 738)] Requests created Using crop h▒a cfr frame_time=33333 camera_set_metering_mode: average camera_set_exposure_control: auto camera_set_white_balance: auto enable AWB: 0 camera_set_custom_awb_gains: red=0.0, blue=0.0 camera_set_brightness: 0.0 camera_set_contrast: 1.0 camera_set_saturation: 1.0 camera_set_sharpness: 0.0 skipping autofocus because it is not supported by the camera [New Thread 0xb0a8e1c0 (LWP 739)] Camera started! audio setup sample_rate: 48000 nb_samples: 1024 format: 1 audio_codec_ctx->ch_layout: 4294967297 av_frame->channel_layout: 4294967297 buffer_size=2048 allocated 2048 bytes for audio samples audio_pts_step: 1920 audio device configured [New Thread 0xb028d1c0 (LWP 741)]

Thread 10 "picam" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb028d1c0 (LWP 741)] 0x015d5bf8 in ?? ()

[iizukanao]

Good. Now enter bt when SIGSEGV is caught.

[MittaAxt]

had to restart gdb, hope this does not mess with some of the hex values.

allocated 2048 bytes for audio samples audio_pts_step: 1920 audio device configured [New Thread 0xb028d1c0 (LWP 843)]

Thread 10 "picam" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb028d1c0 (LWP 843)] 0x015d5bf8 in ?? () (gdb) bt

0 0x015d5bf8 in ?? ()

1 0x000a20ac in ?? ()

Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[iizukanao]

Oops, I can't debug it further because there is no useful information in the backtrace. Did you compile picam with debug symbols? If picam has debug symbols, file picam outputs with debug_info, not stripped as follows.

$ file picam
picam: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=dc5ee53c2f456b9ebae18434e4df0aa6a3e0c06f, for GNU/Linux 3.2.0, with debug_info, not stripped

[MittaAxt]

nope unfortunately not. I just followed the build instructions from BUILDING.md

file picam
picam: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=becb183b8e185759c1200c77c0734eaf352e4f07, for GNU/Linux 3.2.0, stripped

how and what exactly needs to be changed/added during the build of picam

$ git clone https://github.com/iizukanao/picam.git
$ cd picam
$ mkdir build
$ cd build
$ cmake ..
$ make -j4

[iizukanao]

Do not execute strip picam. It strips the debug symbols.

[MittaAxt]

ok, I did a rebuild without the strip

allocated 2048 bytes for audio samples
audio_pts_step: 1920
audio device configured
[New Thread 0xb028d1c0 (LWP 1468)]

Thread 10 "picam" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb028d1c0 (LWP 1468)]
0x015d5bf8 in ?? ()
(gdb) bt
#0  0x015d5bf8 in ?? ()
#1  0x000a20ac in avformat_write_header (s=0x15cbc90, options=<optimized out>) at libavformat/mux.h:138
#2  0x00b9af44 in mpegts_open_stream ()
#3  0x000986e4 in create_new_ts ()
#4  0x00098978 in hls_write_packet ()
#5  0x00099684 in Muxer::onFrameArrive(EncodedPacket*) ()
#6  0x0007d450 in std::_Function_handler<void (long long, unsigned char*, int, int, int), Picam::event_loop()::{lambda(long long, unsigned char*, int, int, int)#1}>::_M_invoke(std::_Any_data const&, long long&&, unsigned char*&&, int&&, int&&, int&&) ()
#7  0x00097344 in Audio::encode_and_send_audio() ()
#8  0x00097a90 in Audio::loop() ()
#9  0xb6544150 in ?? () from /lib/arm-linux-gnueabihf/libstdc++.so.6
#10 0xb6b39310 in start_thread (arg=0xb028d1c0) at pthread_create.c:477
#11 0xb63f5da8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[iizukanao]

Great, thank you! I will look into it.

[iizukanao]

@MittaAxt It seems that if picam is built with the latest ffmpeg source, the segmentation fault will occur. I have pushed a fix (commit ac1260d0b405ec4ecd8fa5383da0023710f0ecbd), so could you please pull the source code from the master branch and see if it works?

[MittaAxt]

will do so today thank you

[MittaAxt]

issue is fixed no more segmentation fault :) thank you so much :)

[iizukanao]

@MittaAxt Thank you for confirming!