[BUG] rpicam-vid Encoding Halts After Several Hours Without Exceptions

jwainwright87 commented 1 month ago

Describe the bug I'm encountering an issue using rpicam-vid to encode a stream with a Raspberry Pi V3 camera on a Compute Module CM4 with my custom IO board. Here's the command I'm using: sudo rpicam-vid --timeout 0 --signal --nopreview --mode 2304:1296 --roi 0.037,0.037,0.926,0.926 --framerate 30 --level 4.2 --bitrate 6000000 --hflip --vflip --av-sync 60000 --width 1920 --height 1080 --tuning-file /home/pi/Documents/Scripts/Installers/imx708_wide_noir_nest_box.json --codec libav --libav-audio --audio-device alsa_input.platform-soc_sound.stereo-fallback --audio-bitrate 192000 --audio-channels 2 --libav-format flv --output rtmp://xx.xx.xxx.xx/live/live

After several hours of encoding, the process halts, but it doesn't exit the rpicam-vid program or raise any exceptions or error messages. The program simply stops without any notification or error messages, except for two lines that appear in dmesg: [18785.372960] vc_sm_cma_vchi_rx_ack: received response 2166729, throw away... [19071.950562] vc_sm_cma_import_dmabuf: imported vc_sm_cma_get_buffer failed -512

The message refers to a "buffer failed" issue, but I’ve discovered that this error occurs after the stream is restarted. I’ve written a small Python program to automatically restart the stream when the encoding halts, and the DMESG error only appears after the rpicam-vid stream has been restarted. Therefore, I suspect this message is related to the stream restart and not the initial encoding failure.

Important Detail on the Failure:

I am encoding both video from the camera and audio from a microphone. When the failure occurs, only the video stops encoding, while the audio stream continues uninterrupted. This suggests that the problem is isolated to the video encoding process, while the audio encoding remains unaffected.

htop Outputs: I’ve also monitored the system using htop to capture the state both when encoding is functioning properly and when the process halts.

htop output showing encoding working without issues: htop-rpicam-vid-running (1)

htop output showing the system after the encoding process halts: htop-rpicam-vid-halt (1)

My confix.txt dtparam=i2c_arm=on dtparam=i2s=on dtparam=spi=on dtparam=audio=on camera_auto_detect=1 display_auto_detect=1 auto_initramfs=1 dtoverlay=vc4-kms-v3d max_framebuffers=2 disable_fw_kms_setup=1 arm_64bit=1 disable_overscan=1 arm_boost=1 otg_mode=1 dtoverlay=gpio-fan,gpiopin=12,temp=75000 dtoverlay=tpm-slb9670 dtoverlay=spi0-2cs,cs0_pin=13 disable_splash=1 dtoverlay=rpi-i2s-mic

My Raspberry Pi Linux version: Linux pinestbox10127 6.6.31+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux

I’ve already tried the following steps to troubleshoot this problem:

Ran a full upgrade on the software
I'm using rpicam-apps version 1.5.0
Piped the output to /dev/null to rule out any network issues
Running the stream with video only and no audio
Checked the temperature, which stays below 45°C using a heatsink with a fan
Ran on a Raspberry Pi 4
Checking to see if vcgencmd get_throttled reports any throttling, which does not
Upgrading to rpicam-apps 1.5.1

It is almost identical to this issue with PyCamera2: https://github.com/raspberrypi/picamera2/issues/1090

Currently testing on Raspberry Pi Linux version below, but I don't see any updates relating to the camera: Currently testing on Raspberry Pi Linux version below, but I don't see any updates relating to the camera:

Bug report bug.txt

Additional context Add any other context about the problem here.

naushir commented 1 month ago

I've previously attempted to reproduce this (when it was posted on the forums), but didn't have any luck. Admittedly I was only recording video and no audio. Are you able to get the same lockup if you record video only?

EDIT: I see you did disable audio and got the failure.

naushir commented 1 month ago

Also, do you know if only the "streaming" case gets locked up? So if you use -o /dev/null does it lockup?

EDIT: looks like you did this and reproduced this as well.

naushir commented 1 month ago

I've managed to reproduce this problem and we are investigating. However, to keep thing simple, I'll continue the discussion on https://github.com/raspberrypi/picamera2/issues/1090.

jwainwright87 commented 1 month ago

No problem, I'm following

naushir commented 2 weeks ago

Changes have been merged and we will update kernel shortly. Closing this down for now.

raspberrypi / rpicam-apps

[BUG] rpicam-vid Encoding Halts After Several Hours Without Exceptions #733