[BUG] libcamera-vid piped into cVLC runs then crashes

[jlpoolen]

On a Pi Zero 2 W with 64-bit Bullseye and Raspberry Pi HQ camera.

Running "libcamera-vid" and piping into command line VLC. for RTSP server.

Here are the libcamera packages installed:

pi@raspberrypi:~ $ apt list --installed |grep camer

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libcamera-apps/stable,now 0~git20220105+b9a6923-1 arm64 [installed]
libcamera-tools/stable,now 0~git20220106+44d59841-2 arm64 [installed]
libcamera0/stable,now 0~git20220106+44d59841-2 arm64 [installed,automatic]

pi@raspberrypi:~ $ apt-cache show libcamera-apps/stable
Package: libcamera-apps
Version: 0~git20220105+b9a6923-1
Architecture: arm64
Maintainer: Serge Schneider <serge@raspberrypi.com>
Installed-Size: 1457
Depends: libboost-program-options1.74.0 (>= 1.74.0), libc6 (>= 2.29), libcamera0 (>= 0~git20220106+44d59841), libdrm2 (>= 2.4.99), libepoxy0 (>= 1.0), libexif12 (>= 0.6.21-1~), libgcc-s1 (>= 3.0), libjpeg62-turbo (>= 1.3.1), libpng16-16 (>= 1.6.2-1), libqt5core5a (>= 5.0.2), libqt5gui5 (>= 5.0.2) | libqt5gui5-gles (>= 5.0.2), libqt5widgets5 (>= 5.0.2), libstdc++6 (>= 7), libtiff5 (>= 4.0.3), libx11-6
Conflicts: libcamera-apps-lite
Replaces: libcamera-apps-lite
Homepage: https://github.com/raspberrypi/libcamera-apps/
Priority: optional
Section: video
Filename: pool/main/libc/libcamera-apps/libcamera-apps_0~git20220105+b9a6923-1_arm64.deb
Size: 269708
SHA256: 4cc63b495efcae547b7d6031b67d5a213866f8b4c3d7db4e3592edbcee283f23
SHA1: f0e0defa8c843dda09b1a38e9be706a21ade6a9d
MD5sum: 6e8f68c35fff5e62e62d007599840a98
Description: libcamera-apps
 Small suite of libcamera-based apps that aim to copy the functionality of the
 existing "raspicam" apps
Description-md5: eaec0c6bb8e262e76ae4dd4bc00a12bd 

pi@raspberrypi:~ $ apt-cache show libcamera-tools/stable
Package: libcamera-tools
Source: libcamera
Version: 0~git20220106+44d59841-2
Architecture: arm64
Maintainer: Debian Multimedia Maintainers <debian-multimedia@lists.debian.org>
Installed-Size: 243
Depends: libc6 (>= 2.17), libcamera0 (>= 0~git20220106+44d59841), libgcc-s1 (>= 3.0), libqt5core5a (>= 5.15.1), libqt5gui5 (>= 5.2.0) | libqt5gui5-gles (>= 5.2.0), libqt5widgets5 (>= 5.4.0), libstdc++6 (>= 5.2), libtiff5 (>= 4.0.3)
Multi-Arch: foreign
Homepage: https://libcamera.org/
Priority: optional
Section: video
Filename: pool/main/libc/libcamera/libcamera-tools_0~git20220106+44d59841-2_arm64.deb
Size: 75536
SHA256: 5d6fd4463fd7030da3636e37606b1af253fd0424e791ef7ccf044ff130d125cc
SHA1: 19858303247bfcfc522d99b55451f215035508d3
MD5sum: 06ac9561b1b95fbe7e218f8b167eb40e
Description: complex camera support library (tools)
 libcamera is a complex camera support library which handles low-level
 control of the camera devices, providing a unified higher-level
 .
 This package provides cam and qcam, libcamera "Swiss army knife" tools.
Description-md5: b37d754ba600317dd4ef2b93d937c362

pi@raspberrypi:~ $ apt-cache show libcamera0/stable
Package: libcamera0
Source: libcamera
Version: 0~git20220106+44d59841-2
Architecture: arm64
Maintainer: Debian Multimedia Maintainers <debian-multimedia@lists.debian.org>
Installed-Size: 3371
Depends: libc6 (>= 2.30), libdw1 (>= 0.158), libgcc-s1 (>= 4.2), libglib2.0-0 (>= 2.37.3), libgnutls30 (>= 3.7.0), libgstreamer-plugins-base1.0-0 (>= 1.14.0), libgstreamer1.0-0 (>= 1.14.0), libstdc++6 (>= 6), libudev1 (>= 183), libunwind8, raspberrypi-kernel (>= 1:1.20211201~)
Multi-Arch: same
Homepage: https://libcamera.org/
Priority: optional
Section: libs
Filename: pool/main/libc/libcamera/libcamera0_0~git20220106+44d59841-2_arm64.deb
Size: 752736
SHA256: 3a19aebb3cac0f85eaef0aa966b034a52a1cf6b3f0c7b0909ec84fd3a6d532c0
SHA1: 0a52cd5176c63be4f5ed9186971e7c595c2e853e
MD5sum: 7fbba97a7a5d975090508cb83b230028
Description: complex camera support library
 libcamera is a complex camera support library which handles low-level
 control of the camera devices, providing a unified higher-level
 programming interface to the applications.
Description-md5: 63804dcb07379b2f103d0ab8e8e0c2c5

pi@raspberrypi:~ $

Following instructions at https://www.raspberrypi.com/documentation/accessories/camera.html#libcamera-and-libcamera-apps which states to run a server, use this command line:

libcamera-vid -t 0 --inline -o - | cvlc stream:///dev/stdin --sout '#rtp{sdp=rtsp://:8554/stream1}' :demux=h264

Instead, I used this modified version which includes the IP address of the Pi server running this command:

libcamera-vid -t 0 --inline -o - | cvlc stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.12:8554/stream1}' :demux=h264

The server runs for about 3 minutes and then crashes, sometimes it crashes sooner

The last lines in the log are:

 [0000007f70004b70] main stream error: unknown query 0x30e in demux_vaControlHelper
 [14:45:30.434716951] [110483] ERROR RPISTREAM rpi_stream.cpp:242 Failed to queue buffer for ISP Input

Also, there are these messages:

  [14:38:37.364941891] [110483]  INFO RPISTREAM rpi_stream.cpp:133 No buffers available for ISP Output0

Attached is a script session (test_script1_Jan_29_2022_log.txt) which contains a prefix consisting of "[time stamp] [numberical sequence number]" test_script1_Jan_29_2022_log.txt

Attached as (bug-report_20220129_104241.zip) is an attempt using: bug-report_20220129_104241.zip

 camera-bug-report -c "libcamera-vid -t 0 --inline -o - | cvlc stream:///dev/stdin --sout '#rtp{sdp=rtsp://:8554/stream1}' :demux=h264"

which failed and logged as Issue #237.

[jlpoolen]

Issue #236 shares the same warning/error:

 Failed to queue buffer for ISP Input

[jlpoolen]

I started searching the code and found the comment at line 66 in qt_preview.cpp

 This preview window is expensive, so make it small by default.

And then I modified my command line to include the option of "-n " for no preview window. That's seems to have made a significant impact: the server has been running for over 30 minutes now and is on-going with 4 connections.

Works-for-me = omit the preview window

Issue #236 ought to try the same and report.

[naushir]

Issue #236 shares the same warning/error:

 Failed to queue buffer for ISP Input

Yes, I have reproduced this on my build as well. Will investigate now and see what could be the cause.

[naushir]

We have identified the cause of this problem, and a fix has been submitted for review/merge here. Hopefully this should be merged very soon.

[naushir]

The fixes for this issue have now been merged. You can get the latest libraries by doing:

sudo apt update
sudo apt upgrade

I'll close this issue down now, please re-open if you still have problems.

[jlpoolen]

Thank you, the command line quoted in the opening of this issue and from the documentation works now.

However, I have not performed any serious testing, but will share with something in a new bug after I have marshaled all the information and run some debug capture logs for you. The problems are the "WARN" lines. This was the 3rd run and they seem to come on after 3-10 minutes of running without displaying WARN messages. In the session below, I had an emacs on the desktop and an emacs through an ssh connection running; the CPU started in the 45% slowly inching its way up to 52%. I was not active elsewhere on the unit, so the slow increment climb should be attributable only to this process. Definitely need a more careful test and investigation. Also, I was noticing a lot of dropped frames. The RPiZero2W is connected to a switch which my Windows workstation is connected to, all within several feet of one another. I could be having network issues, next time I'll try to run a separate VLC on the RPi2W desktop to capture statistics which would represent no-network conditions since both processes are on the same unit.

pi@raspberrypi:~ $ date;libcamera-vid -t 0 --inline -o - | cvlc stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.12:8554/stream1}' :demux=h264
Wed  2 Feb 20:32:31 PST 2022
VLC media player 3.0.16 Vetinari (revision 3.0.13-8-g41878ff4f2)
Made X/EGL preview window
[1:28:09.219207330] [13577]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3406-e96d0201
[1:28:09.332079016] [13594]  WARN CameraSensorProperties camera_sensor_properties.cpp:141 No static properties available for 'imx477'
[1:28:09.332219499] [13594]  WARN CameraSensorProperties camera_sensor_properties.cpp:143 Please consider updating the camera sensor properties database
[1:28:09.332292996] [13594] ERROR CameraSensor camera_sensor.cpp:551 'imx477 10-001a': Camera sensor does not support test pattern modes.
[1:28:09.898558243] [13594]  INFO RPI raspberrypi.cpp:1326 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media3 and ISP device /dev/media0
[1:28:09.907316427] [13577]  INFO Camera camera.cpp:1028 configuring streams: (0) 640x480-YUV420
[1:28:09.908748917] [13594]  INFO RPI raspberrypi.cpp:747 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 2028x1520-SBGGR12_1X12 - Selected unicam format: 2028x1520-pBCC
[1:28:10.082972240] [13604]  INFO IPARPI raspberrypi.cpp:626 Request ctrl: Saturation = 1.000000
[1:28:10.086772106] [13604]  INFO IPARPI raspberrypi.cpp:626 Request ctrl: Contrast = 1.000000
[1:28:10.086877690] [13604]  INFO IPARPI raspberrypi.cpp:626 Request ctrl: Brightness = 0.000000
[1:28:10.086959990] [13604]  INFO IPARPI raspberrypi.cpp:626 Request ctrl: AwbMode = 0
[1:28:10.087020673] [13604]  INFO IPARPI raspberrypi.cpp:626 Request ctrl: Sharpness = 1.000000
[1:28:10.087111047] [13604]  INFO IPARPI raspberrypi.cpp:626 Request ctrl: ExposureValue = 0.000000
[1:28:10.087194493] [13604]  INFO IPARPI raspberrypi.cpp:626 Request ctrl: AeExposureMode = 0
[1:28:10.087251947] [13604]  INFO IPARPI raspberrypi.cpp:626 Request ctrl: FrameDurationLimits = [ 33333, 33333 ]
[1:28:10.087319662] [13604]  INFO IPARPI raspberrypi.cpp:626 Request ctrl: AeMeteringMode = 0
[1:28:10.087376751] [13604]  INFO IPARPI raspberrypi.cpp:626 Request ctrl: NoiseReductionMode = 1
[1:37:17.619861191] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:37:17.806753923] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:37:33.398333805] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:37:50.763584424] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:38:28.560442866] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:38:28.778714214] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded

jlpoole: 1:38:28 =  8:42 PM 2/2/2022

[1:38:46.522788952] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:38:59.296815287] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:19.592781848] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:23.658760551] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:32.397391063] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:48.988522591] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:49.619969200] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:49.621304750] [13594]  WARN RPI raspberrypi.cpp:2155 Dropping unmatched input frame in stream Unicam Image
[1:39:49.621550859] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:50.186740873] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:50.258247557] [13594]  WARN RPI raspberrypi.cpp:2155 Dropping unmatched input frame in stream Unicam Image
[1:39:50.676493766] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:50.958383380] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:52.289730103] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:52.359714983] [13594]  WARN RPI raspberrypi.cpp:2155 Dropping unmatched input frame in stream Unicam Image
[1:39:54.062549709] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:56.741574128] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:58.053205556] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:58.558076439] [13594]  WARN RPI raspberrypi.cpp:2118 Dropping unmatched input frame in stream Unicam Embedded
[1:39:58.620310395] [13594]  WARN RPI raspberrypi.cpp:2155 Dropping unmatched input frame in stream Unicam Image
^C
pi@raspberrypi:~ $ date
Wed  2 Feb 20:44:33 PST 2022
pi@raspberrypi:~ $ 

Here's one of the VLC statistics from an instance run on my Windows workstation:

[naushir]

Those warnings indicate probable frame drops. There could be many causes of this, but there is one in particular that we have been looking at very recently. It turns out that the info text updates on the preview window seem to be very expensive. It might be worth trying to disable those updates with the --info-text "" argument added to libcamera-vid.