search

raspberrypi / libcamera

Other
223 stars 95 forks source link

libcamerasrc doesn't work with CM4 RPI Cam V2 and V3 #115

Closed linus-gates closed 7 months ago

linus-gates commented 7 months ago

Hey,

I am currently trying to get the libcamera gstreamer element to work with the Compute Module 4 IO board. I tried both IMX708 (Raspberry Camera V3) and IMX219 (NoIR Camera V2).

Both cameras work with libcamera-vid / libcamera-raw / libcamera-hello while both v4l2src and libcamerasrc failed. I tried using the latest Raspberry Pi OS (Linux raspberrypi 6.1.0-rpi7-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.63-1+rpt1 (2023-11-24) aarch64 GNU/Linux) and built my own image with Yocto kirkstone. Same results.

I tried multiple examples with different caps that I was able to find online and it all boils down to the same log + no buffers being passed from libcamerasrc:

gst-launch-1.0 libcamerasrc ! capsfilter caps=video/x-raw,width=1920,height=1080,format=NV12,colorimetry=bt601,framerate=30/1,interlace-mode=progressive ! fakesink silent=false -v
Setting pipeline to PAUSED ...
[0:16:10.604592434] [2642]  WARN IPAManager ipa_manager.cpp:114 Public key not valid
[0:16:10.605518475] [2642]  INFO IPAManager ipa_manager.cpp:143 libcamera is not installed. Adding '/home/user/libcamera/build/src/ipa' to the IPA search path
[0:16:10.617520172] [2642]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0+46-075b54d5
[0:16:10.667653269] [2643]  INFO IPAProxy ipa_proxy.cpp:198 libcamera is not installed. Loading proxy workers from '/home/user/libcamera/build/src/libcamera/proxy/worker'
[0:16:10.670464683] [2643]  INFO IPAProxy ipa_proxy.cpp:130 libcamera is not installed. Loading IPA configuration from '/home/user/libcamera/src/ipa/rpi/vc4/data'
[0:16:10.733188927] [2643]  WARN RPI vc4.cpp:393 Mismatch between Unicam and CamHelper for embedded data usage!
[0:16:10.735353971] [2643]  INFO RPI vc4.cpp:447 Registered camera /base/soc/i2c0mux/i2c@0/imx219@10 to Unicam device /dev/media4 and ISP device /dev/media1
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = event   ******* (fakesink0:sink) E (type: stream-start (10254), GstEventStreamStart, stream-id=(string)454eae2e5435794ed123783c17e0687b/10, flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, group-id=(uint)1;) 0x559690c1c0
New clock: GstSystemClock
[0:16:10.742189232] [2647]  INFO Camera camera.cpp:1183 configuring streams: (0) 1920x1080-NV12
[0:16:10.743677022] [2643]  INFO RPI vc4.cpp:611 Sensor: /base/soc/i2c0mux/i2c@0/imx219@10 - Selected sensor format: 1920x1080-SBGGR10_1X10 - Selected unicam format: 1920x1080-pBAA
/GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0.GstLibcameraPad:src: caps = video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080, colorimetry=(string)2:4:5:4, framerate=(fraction)30/1
[0:16:12.764724982] [2643] ERROR IPCPipe ipc_pipe_unixsocket.cpp:134 Call timeout!
[0:16:12.764998932] [2643] ERROR IPCPipe ipc_pipe_unixsocket.cpp:66 Failed to call sync
[0:16:12.765084311] [2643] ERROR IPAProxy raspberrypi_ipa_proxy.cpp:261 Failed to call start
[0:16:12.800821171] [2643] ERROR IPCUnixSocket ipc_unixsocket.cpp:191 Failed to send: Connection refused
[0:16:12.801123878] [2643] ERROR IPCPipe ipc_pipe_unixsocket.cpp:66 Failed to call sync
[0:16:12.801209220] [2643] ERROR IPAProxy raspberrypi_ipa_proxy.cpp:432 Failed to call mapBuffers
[0:16:12.897791695] [2643] ERROR IPCUnixSocket ipc_unixsocket.cpp:191 Failed to send: Transport endpoint is not connected
[0:16:12.897893924] [2643] ERROR IPCPipe ipc_pipe_unixsocket.cpp:80 Failed to call async
[0:16:12.897926217] [2643] ERROR IPAProxy raspberrypi_ipa_proxy.cpp:508 Failed to call prepareIsp

Also brute-forced different caps that should be supported based on gst-device-monitor-1.0 and v4l2-ctl without any luck.

My config.txt for the IMX708 setup is:

#dtparam=i2c_arm=on
#camera_auto_detect=1
dtoverlay=imx708,cam0
dtoverlay=imx708,cam1

and for the IMX219 setup:

#dtparam=i2c_arm=on
#camera_auto_detect=1
dtoverlay=imx219,cam0

(Based on https://www.raspberrypi.com/documentation/computers/compute-module.html#attach-a-raspberry-pi-camera-module) I have also tried toggling auto detect etc.

I found documentation regarding setting up the cameras for v4l2src with v4l2-ctl and media-ctl in https://wiki.veye.cc/index.php/V4L2_mode_for_Raspberry_Pi (https://github.com/veyeimaging/raspberrypi_v4l2/blob/main/rpi5_scripts/media_setting_rpi5.sh) but couldn't figure out how to get this to work with the RPI V2 and V3 cameras.

I would really want to avoid using gstreamer fdsrc with unix pipes since this is a part of a more complex pipeline and in-house configuration systems. Since the CLI tools work I could also attempt to solve this myself but I would need some pointers in the codebase. Thanks in advance! Any advice would be appreciated 😅

naushir commented 7 months ago 
[0:16:10.604592434] [2642]  WARN IPAManager ipa_manager.cpp:114 Public key not valid
[0:16:10.605518475] [2642]  INFO IPAManager ipa_manager.cpp:143 libcamera is not installed. Adding '/home/user/libcamera/build/src/ipa' to the IPA search path
[0:16:10.617520172] [2642]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0+46-075b54d5
[0:16:10.667653269] [2643]  INFO IPAProxy ipa_proxy.cpp:198 libcamera is not installed. Loading proxy workers from '/home/user/libcamera/build/src/libcamera/proxy/worker'
[0:16:10.670464683] [2643]  INFO IPAProxy ipa_proxy.cpp:130 libcamera is not installed. Loading IPA configuration from '/home/user/libcamera/src/ipa/rpi/vc4/data'

This seems to suggest you are running a custom version of libcamera from the build directory? Have you tried with our libcamera apt packages?

linus-gates commented 7 months ago

Thanks! You helped me find a working setup. v0.1.0 with -Dv4l2=true got it to work well!

For reference, I built main based on the instructions in the README. Tried v0.2.0 now, same thing:

gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=1280,height=720' ! \
     queue ! fakesink silent=false -v

Setting pipeline to PAUSED ...
[2:19:42.329027909] [15057]  WARN IPAManager ipa_manager.cpp:114 Public key not valid
[2:19:42.329906556] [15057]  INFO IPAManager ipa_manager.cpp:143 libcamera is not installed. Adding '/home/user/libcamera/build/src/ipa' to the IPA search path
[2:19:42.339510212] [15057]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0
[2:19:42.387862881] [15058]  INFO IPAProxy ipa_proxy.cpp:198 libcamera is not installed. Loading proxy workers from '/home/user/libcamera/build/src/libcamera/proxy/worker'
[2:19:42.390854674] [15058]  INFO IPAProxy ipa_proxy.cpp:130 libcamera is not installed. Loading IPA configuration from '/home/user/libcamera/src/ipa/rpi/vc4/data'
[2:19:42.453017864] [15058]  WARN RPI vc4.cpp:347 Mismatch between Unicam and CamHelper for embedded data usage!
[2:19:42.455539343] [15058]  INFO RPI vc4.cpp:401 Registered camera /base/soc/i2c0mux/i2c@0/imx219@10 to Unicam device /dev/media4 and ISP device /dev/media1
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = event   ******* (fakesink0:sink) E (type: stream-start (10254), GstEventStreamStart, stream-id=(string)cbb3ec3cd634ecc9da0f580f9d91d40b/10, flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, group-id=(uint)1;) 0x558bbccdc0
New clock: GstSystemClock
[2:19:42.464579759] [15063]  INFO Camera camera.cpp:1183 configuring streams: (0) 1280x720-NV21
[2:19:42.466500960] [15058]  INFO RPI vc4.cpp:559 Sensor: /base/soc/i2c0mux/i2c@0/imx219@10 - Selected sensor format: 1920x1080-SBGGR10_1X10 - Selected unicam format: 1920x1080-pBAA
/GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0.GstLibcameraPad:src: caps = video/x-raw, format=(string)NV21, width=(int)1280, height=(int)720, colorimetry=(string)bt709, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)NV21, width=(int)1280, height=(int)720, colorimetry=(string)bt709, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw, format=(string)NV21, width=(int)1280, height=(int)720, colorimetry=(string)bt709, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)NV21, width=(int)1280, height=(int)720, colorimetry=(string)bt709, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = video/x-raw, format=(string)NV21, width=(int)1280, height=(int)720, colorimetry=(string)bt709, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = event   ******* (fakesink0:sink) E (type: caps (12814), GstEventCaps, caps=(GstCaps)"video/x-raw\,\ format\=\(string\)NV21\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ colorimetry\=\(string\)bt709\,\ framerate\=\(fraction\)30/1";) 0x558bbccea0
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw, format=(string)NV21, width=(int)1280, height=(int)720, colorimetry=(string)bt709, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = event   ******* (fakesink0:sink) E (type: segment (17934), GstEventSegment, segment=(GstSegment)"segment, flags=(GstSegmentFlags)GST_SEGMENT_FLAG_NONE, rate=(double)1, applied-rate=(double)1, format=(GstFormat)time, base=(guint64)0, offset=(guint64)0, start=(guint64)0, stop=(guint64)18446744073709551615, time=(guint64)0, position=(guint64)0, duration=(guint64)18446744073709551615;";) 0x558bbccf10
[2:19:44.486253824] [15058] ERROR IPCPipe ipc_pipe_unixsocket.cpp:134 Call timeout!
[2:19:44.486447824] [15058] ERROR IPCPipe ipc_pipe_unixsocket.cpp:66 Failed to call sync
[2:19:44.486497694] [15058] ERROR IPAProxy raspberrypi_ipa_proxy.cpp:261 Failed to call start
[2:19:44.503846990] [15058] ERROR IPCUnixSocket ipc_unixsocket.cpp:191 Failed to send: Connection refused
[2:19:44.504138916] [15058] ERROR IPCPipe ipc_pipe_unixsocket.cpp:66 Failed to call sync
[2:19:44.504227860] [15058] ERROR IPAProxy raspberrypi_ipa_proxy.cpp:432 Failed to call mapBuffers

v0.2.0 with -Dv4l2=true

user@raspberrypi ~/libcamera/build (tags/v0.1.0?) $ GST_PLUGIN_PATH=. gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=1280,height=720' ! \
     queue ! autovideosink
Setting pipeline to PAUSED ...
[2:52:49.955904027] [21362]  INFO IPAManager ipa_manager.cpp:143 libcamera is not installed. Adding '/home/user/libcamera/build/src/ipa' to the IPA search path
[2:52:49.968334431] [21362]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0
[2:52:50.124255114] [21370]  INFO IPAProxy ipa_proxy.cpp:130 libcamera is not installed. Loading IPA configuration from '/home/user/libcamera/src/ipa/rpi/vc4/data'
[2:52:50.142666241] [21370]  WARN RPiSdn sdn.cpp:39 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[2:52:50.151744863] [21370]  WARN RPI vc4.cpp:347 Mismatch between Unicam and CamHelper for embedded data usage!
[2:52:50.154892497] [21370]  INFO RPI vc4.cpp:401 Registered camera /base/soc/i2c0mux/i2c@0/imx219@10 to Unicam device /dev/media4 and ISP device /dev/media1
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[2:52:50.167898075] [21374]  INFO Camera camera.cpp:1183 configuring streams: (0) 1280x720-YUV420
[2:52:50.169745693] [21370]  INFO RPI vc4.cpp:559 Sensor: /base/soc/i2c0mux/i2c@0/imx219@10 - Selected sensor format: 1920x1080-SBGGR10_1X10 - Selected unicam format: 1920x1080-pBAA
[2:52:50.309585860] [21376] FATAL default ipa_base.cpp:396 assertion "it != buffers_.end()" failed in prepareIsp()
Backtrace:/ 99:99:99.
libcamera::ipa::RPi::IpaBase::prepareIsp(libcamera::ipa::RPi::PrepareParams const&)+0x188 (../src/ipa/rpi/common/ipa_base.cpp:397)
libcamera::ipa::RPi::IPAProxyRPi::ThreadProxy::prepareIsp(libcamera::ipa::RPi::PrepareParams const&)+0x34 (include/libcamera/ipa/raspberrypi_ipa_proxy.h:214)
libcamera::BoundMethodMember<libcamera::ipa::RPi::IPAProxyRPi::ThreadProxy, void, libcamera::ipa::RPi::PrepareParams const&>::invoke(libcamera::ipa::RPi::PrepareParams const&)+0x88 (../include/libcamera/base/bound_method.h:191)
std::enable_if<std::is_void<void>::value, void>::type libcamera::BoundMethodArgs<void, libcamera::ipa::RPi::PrepareParams const&>::invokePack<0ul, void>(libcamera::BoundMethodPackBase*, std::integer_sequence<unsigned long, 0ul>)+0x48 (../include/libcamera/base/bound_method.h:116)
libcamera::BoundMethodArgs<void, libcamera::ipa::RPi::PrepareParams const&>::invokePack(libcamera::BoundMethodPackBase*)+0x1c (../include/libcamera/base/bound_method.h:125)
libcamera::InvokeMessage::invoke()+0x44 (../src/libcamera/base/message.cpp:154)
libcamera::Object::message(libcamera::Message*)+0x70 (../src/libcamera/base/object.cpp:200)
libcamera::Thread::dispatchMessages(libcamera::Message::Type)+0x260 (../src/libcamera/base/thread.cpp:632)
libcamera::EventDispatcherPoll::processEvents()+0x20 (../src/libcamera/base/event_dispatcher_poll.cpp:150)
libcamera::Thread::exec()+0x54 (../src/libcamera/base/thread.cpp:340)
libcamera::Thread::run()+0x14 (../src/libcamera/base/thread.cpp:369)
libcamera::Thread::startThread()+0x100 (../src/libcamera/base/thread.cpp:320)
void std::__invoke_impl<void, void (libcamera::Thread::*)(), libcamera::Thread*>(std::__invoke_memfun_deref, void (libcamera::Thread::*&&)(), libcamera::Thread*&&)+0x78 (/usr/include/c++/12/bits/invoke.h:74)
std::__invoke_result<void (libcamera::Thread::*)(), libcamera::Thread*>::type std::__invoke<void (libcamera::Thread::*)(), libcamera::Thread*>(void (libcamera::Thread::*&&)(), libcamera::Thread*&&)+0x38 (/usr/include/c++/12/bits/invoke.h:97)
void std::thread::_Invoker<std::tuple<void (libcamera::Thread::*)(), libcamera::Thread*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>)+0x3c (/usr/include/c++/12/bits/std_thread.h:252)
std::thread::_Invoker<std::tuple<void (libcamera::Thread::*)(), libcamera::Thread*> >::operator()()+0x14 (/usr/include/c++/12/bits/std_thread.h:259)
std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (libcamera::Thread::*)(), libcamera::Thread*> > >::_M_run()+0x18 (/usr/include/c++/12/bits/std_thread.h:210)
??? [0x0000007f9f49e9dc] (/usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.30 [0x0000007f9f49e9dc])
start_thread+0x2b8 (./nptl/pthread_create.c:442)
thread_start+0xc (../sysdeps/unix/sysv/linux/aarch64/clone.S:82)
[1]    21362 abort      GST_PLUGIN_PATH=. gst-launch-1.0 libcamerasrc !  ! queue ! autovideosink

It doesn't seem like any package provides the gstreamer element? I tried installing sudo apt-get install libcamera[0.2/-dev/-tools/-apps/-v4l2] and none of them provided the element. Possibly related side note, the element would still assume libcamera isn't installed even though all of those packages were installed and ninja install was successfully executed. Couldn't find libgstlibcamera in apt-cache search and Ubuntu's package search either.

Is there a different source I should be getting the package from?

linus-gates commented 7 months ago

This is the working configuration:

libcamera 0.1.0

  Versions
    Sources                  : 0.1.0

  Paths
    LIBCAMERA_DATA_DIR       : "/usr/local/share/libcamera"
    LIBCAMERA_SYSCONF_DIR    : "/usr/local/etc/libcamera"
    IPA_PROXY_DIR            : "/usr/local/libexec/libcamera"
    IPA_CONFIG_DIR           : "/usr/local/etc/libcamera/ipa:/usr/local/share/libcamera/ipa"
    IPA_MODULE_DIR           : "/usr/local/lib/aarch64-linux-gnu/libcamera"

  Configuration
    IPA modules signed with  : gnutls
    Enabled pipelines        : imx8-isi
                               rkisp1
                               rpi/vc4
                               simple
                               uvcvideo
    Enabled IPA modules      : rkisp1
                               rpi/vc4
    Hotplug support          : YES
    Tracing support          : YES
    Android support          : NO
    GStreamer support        : YES
    Python bindings          : NO
    V4L2 emulation support   : YES
    cam application          : YES
    qcam application         : NO
    lc-compliance application: YES
    Unit tests               : NO

  Subprojects
    libyaml                  : YES

  User defined options
    v4l2                     : true

Found ninja-1.11.1 at /usr/bin/ninja

Maybe it's worth mentioning that v4l2=true is required for gstreamer usage in the README.

naushir commented 7 months ago 
[2:19:42.329027909] [15057]  WARN IPAManager ipa_manager.cpp:114 Public key not valid
[2:19:42.329906556] [15057]  INFO IPAManager ipa_manager.cpp:143 libcamera is not installed. Adding '/home/user/libcamera/build/src/ipa' to the IPA search path
[2:19:42.339510212] [15057]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0
[2:19:42.387862881] [15058]  INFO IPAProxy ipa_proxy.cpp:198 libcamera is not installed. Loading proxy workers from '/home/user/libcamera/build/src/libcamera/proxy/worker'
[2:19:42.390854674] [15058]  INFO IPAProxy ipa_proxy.cpp:130 libcamera is not installed. Loading IPA configuration from '/home/user/libcamera/src/ipa/rpi/vc4/data'

You seem to have a problem here with your libcamera build. The IPA is attempting to run in isolation mode (and not working correctly) because the module signing does not work. You probably need to fix this before looking at why your gstreamer pipeline is not working. Have you installed the libgnutls28-dev and openssl libraries via apt before doing the meson setup on libcamera?

naushir commented 7 months ago

Any update on this? If not, I'll close this down shortly.