[BUG] Raspberry HQ Camera V2 / IMX477 suport for more resolution/framerate combinations

[olokos]

Describe the bug

libcamera-vid supports only 4 resolution / framerate combinations, which is a very tiny subset of what the hardware is capable of.

Bug report Bug report for image Bug report for 50 frames video

Additional context Add any other context about the problem here.

I believe this is the first implementation: https://github.com/raspberrypi/linux/commit/fe2352942576f52afd6c5a5a14e7b49c2ef8b497

On top of being limited by only those 4 selections, they don't seem to work as expected, despite I've built the latest commit of both libcamera and libcamera-apps, unless I'm doing something wrong?

I start RTSP using this command: libcamera-vid --width 2028 --height 1080 --framerate 50 -o - -t 0 -n | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.222:7447/}' :demux=h264

But then the VLC stream is only 1920x1080@30 FPS for some reason.

Could the resolution/fps combos be improved for IMX477? ? Am I doing something wrong or should the VLC in fact stream the same resolution as provided by libcamera-vid?

The only error that comes to mind is this: main stream error: unknown query 0x30e in demux_vaControlHelper

This is the full log from the above command, where I spotted the error above: https://pastebin.com/etX4UsB1

Here is the mediainfo for the test video: https://pastebin.com/Nxg2MksD

10MB/s for 640x480 while --framerate 50 was requested :(

[naushir]

The sensor modes supported by the imx477 driver have been qualified by Sony for us to use. It is not possible to setup arbitrary resolutions and framerate combinations in the sensor, they must be chosen carefully to be workable.

As described here, the --width and --height command line arguments specify the resolution of the output image frame, not the sensor mode used. In these cases, the libcamera pipeline handler will try and find a "best match" to a sensor mode based on some heuristics. If you want to choose a particular sensor mode, you can use the --mode command line argument, e.g:

libcamera-vid --width 1280 --height 720 --framerate 120 --mode 1332:990:10

will use the 1332x990 120fps mode in the sensor and rescale to 1280x720 for the final output image.

[olokos]

Thank you for a quick reply!

As you might know, raspistill supports all resolution/framerate modes, sometimes even those not supported officially, making libcamera look like alpha version, compared to raspistill feature set, despite the fact that raspistill is considered "legacy" and libcamera is already pushed as go-to solution.

Moreover, it doesn't even support the res/fps modes as officially described in the documentation: https://www.raspberrypi.com/documentation/accessories/camera.html

I think that official raspberry documentation should be the minimum of what the libcamera-vid/still supports, but it doesn't support even those.

On top of that, raspistill supports plethora of options, most importantly -ex night More of the details can be found here: https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781119415572.app2

libcamera-vid/still at the same time supports laughably small exposition modes - normal and sport, with almost no difference between the two.

I am aware that --width --height --framerate will choose the closest possible res/fps combination available, instead of setting it firmly, the problem is that it doesn't work properly, please check my logs above, the logic meant to work, doesn't seem to work at all, almost as if it would randomly select res/fps mode instead of based on parameters provided.

I have used libcamera-vid --width 2028 --height 1080 --framerate 50, so it should select 50fps mode, but instead it goes 1280x720@30fps, which makes absolutely no sense at all, you can see that in the logs in the first message here, that mode isn't even listed in --list-cameras and it never goes above 30 fps.

I am writing this to improve HQ Camera support with libcamera-apps, because right now the state is really really bad.

I was expecting full support for all features with modern software, considering it's a 2018 product, yet even the bare minimum possible doesn't work as expected, which is quite a letdown, knowing this is an official product, released 4 years ago.

Please let me know if I can help in any possible way.

[naushir]

Thank you for a quick reply!

As you might know, raspistill supports all resolution/framerate modes, sometimes even those not supported officially, making libcamera look like alpha version, compared to raspistill feature set.

Our libcamera applications support all the resolutions and framerates that the old raspicam applications supported. I'm a bit confused by what you mean with the phrase "sometimes even those not supported officially". The old raspicam software was closed source and users could not add new sensor modes at all. With libcamera, you are able to add new sensor modes to the kernel driver, or indeed add new sensors to work with libcamera, everything is open source.

Moreover, it doesn't even support the res/fps modes as officially described in the documentation: https://www.raspberrypi.com/documentation/accessories/camera.html

I think that official raspberry documentation should be the minimum of what the libcamera-vid/still supports, but it doesn't support even those.

Could you point to a specific example of a resolution that is not supported? On recent releases, the following modes are supported in libcamera-apps with the imx477:

Available cameras
-----------------
0 : imx477 [4056x3040] (/base/soc/i2c0mux/i2c@1/imx477@1a)
    Modes: 'SRGGB10_CSI2P' : 1332x990 [120.05 fps - (696, 528)/2664x1980 crop]
           'SRGGB12_CSI2P' : 2028x1080 [50.03 fps - (0, 440)/4056x2160 crop]
                             2028x1520 [40.01 fps - (0, 0)/4056x3040 crop]
                             4056x3040 [10.00 fps - (0, 0)/4056x3040 crop]

This is the same as the list in the documentation you linked to.

On top of that, raspistill supports plethora of options, most importantly -ex night More of the details can be found here: https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781119415572.app2

libcamera-vid/still at the same time supports laughably small exposition modes - normal and sport, with almost no difference between the two.

libcamera-apps also support long exposure modes, which is equivalent to the raspicam night mode. Again, the great thing about libcamera being open source is that you can add any number of exposure modes you like.

I am aware that --width --height --framerate will choose the closest possible res/fps combination available, instead of setting it firmly, the problem is that it doesn't work properly, please check my logs above.

I have used only --framerate 50, so it should select 50fps mode, but instead it goes 640x480@30fps, which makes absolutely no sense at all.

The libcamera API is designed such that framerate cannot be used to choose the sensor mode. This is out of our hands. So we provided the --mode option to select fast framerate mode if that is what you want.

Your logs above do not show the libcamera output, only the ffmpeg output which would not help here. To get libcamera logs out, I suggest you remote the ffmpeg pipe, and append LIBCAMERA_LOG_LEVELS=RPI:0 to your command line.

I am writing this to improve HQ Camera support with libcamera-apps, because right now the state is really really bad.

I was expecting full support for all features with modern software, considering it's a 2018 product, yet even the bare minimum possible doesn't work as expected.

If you feel libcamera does not do what you want, you do have the option of switching back to the raspicam legacy stack. Instructions can be found here.

[olokos]

Thank you very much for Your valuable and blazing fast reply, it definitely helps me understand the situation a lot more!

I THINK that raspi-still and raspi-vid used to support resolutions and framerates outside of those 4, but I might be mistaken, I'll look into it once this is more or less resolved.

Specific not supported resolution is for example 480p90, I have just tested it with mode libcamera-vid --width 640 --height 480 --framerate 90 --mode 640:480:90 -o - -t 0 -n | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.6:7447/}' :demux=h264

resulting stream: 480p30fps, despite I have just set mode manually!

I thought, hey, maybe I won't set resolution and framerate manually, only use the mode parameter like this: libcamera-vid --mode 640:480:90 -o - -t 0 -n | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.6:7447/}' :demux=h264 But the result is the same, 640x480@30FPS

Okay, in that case, I'll try to use one of the listed modes, that will surely work, right? libcamera-vid --mode 1332:990:120 -o - -t 0 -n | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.6:7447/}' :demux=h264

But sadly, the stream is still 640x480@30

Then I went with approach of defining all of the parameters and also using built-in res/fps combo: libcamera-vid --width 1332 --height 990 --framerate 120 --mode 1332:990:120 -o - -t 0 -n | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.6:7447/}' :demux=h264

This results in vlc: Image resolution - 1332x990 Buffer size 1334x992 Framerate: 30

Then I thought that maybe I need to explicitly also provide fps to cvlc (console vlc), so the stream would also be 120 fps like so: libcamera-vid --width 1332 --height 990 --framerate 120 --mode 1332:990:120 -o - -t 0 -n | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.6:7447/}' :demux=h264 --h264-fps=120

But I get the same results as above, same resolution and also only 30FPS

I have also set my commandline to 256mb gpu memory, so there would be no bottleneck there.

libcamera-apps also support long exposure modes, which is equivalent to the raspicam night mode. Again, the great thing about libcamera being open source is that you can add any number of exposure modes you like.

Please do let me know how could I get similar results with libcamera-vid as previosuly with raspistill -ex night The only available exposition modes I could find doing libcamera --help were --exposure arg (=normal) Set the exposure mode (normal, sport) and that's pretty much it, I couldn't find any similar option myself.

The closest I could find to the long you mentioned was: Note that the config file must only contain the long form options. - there's no other occurence of "long" when using libcamera-hello.

Your logs above do not show the libcamera output, only the ffmpeg output which would not help here. To get libcamera logs out, I suggest you remote the ffmpeg pipe, and append LIBCAMERA_LOG_LEVELS=RPI:0 to your command line.

Please do kindly let me know the exact command I should use, I don't really know how to do it as you explained above, and I'd love to do it right. I can only think of inserting LIBCAMERA_LOG_LEVELS=RPI:0 into my /boot/config.txt but no idea about the ffmpeg pipe.

My main connection to the RPi is over SSH, so I usually turn off preview and rely on RTSP for preview, I also need RTSP for my other project, just wanted to mention that, but if it's crucial to use HDMI, I will do it.

Thank you once again for all of the valuable info, I adore the fact that libcamera-apps are finally open source, instead of being closed source as raspistill, but I'm still not convinced if I'm doing something wrong or is the state of libcamera-apps not very good. I went as far as building both libcamera and libcamera-apps, so I'll try my hardest to get it to work on the new stack, but if all fails then I'll go back to legacy, thank You so much for the link of reverting to legacy stack and all of the others aswell!

[olokos]

I do have to add that when invoking: libcamera-vid --width 640 --height 480 --framerate 90 --mode 640:480:90 -o - -t 0 -n | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.6:7447/}' :demux=h264 --h264-fps=90

the main input debug: Buffering 83% spam seems to be about 3 times larger than before, without --h264-fps=90

But vlc still reports 30 fps and it does in fact seem like it's actually 30fps.

Sorry for double message, but this seems a separate thing to me.

[6by9]

A raw H264 elementary stream (which is what you are piping from libcamera-vid to cvlc) has no timestamp information. The headers may include a framerate, but that is optional, and relies on the codec being correctly passed that information.

Sort out your libcamera-vid parameters first - saving to an mp4 file that can be analysed is probably easiest by using libcamera-vid --width 640 --height 480 --framerate 90 --codec libav -o file.mp4. Once that is correct, then worry about getting it into cvlc.

[olokos]

Thank you, here's the result:

General
Complete name                  : C:\Users\olokos\Downloads\file.mp4
Format                         : AVC
Format/Info                    : Advanced Video Codec
File size                      : 739 KiB
Duration                       : 3 s 200 ms
Overall bit rate mode          : Constant
Overall bit rate               : 1 892 kb/s
FileExtension_Invalid          : avc h264 264

Video
Format                         : AVC
Format/Info                    : Advanced Video Codec
Format profile                 : High@L4
Format settings                : CABAC / 1 Ref Frames
Format settings, CABAC         : Yes
Format settings, Reference fra : 1 frame
Duration                       : 3 s 200 ms
Bit rate mode                  : Constant
Bit rate                       : 10 000 kb/s
Width                          : 640 pixels
Height                         : 480 pixels
Display aspect ratio           : 4:3
Frame rate                     : 30.000 FPS
Color space                    : YUV
Chroma subsampling             : 4:2:0
Bit depth                      : 8 bits
Scan type                      : Progressive
Bits/(Pixel*Frame)             : 1.085
Stream size                    : 3.81 MiB

I had to change codec from libav to h264 as libav was not found, also since I'm using ssh, I had to append -n so there's no preview, but above is the mediainfo from the resulting file.

The exact command I used to generate the file is: libcamera-vid --width 640 --height 480 --framerate 90 --codec h264 -o file.mp4 -n

[6by9]

I had to change codec from libav to h264 as libav was not found, also since I'm using ssh, I had to append -n so there's no preview, but above is the mediainfo from the resulting file.

You've done it again - you've created an H264 elementary stream that has no timestamp information, just stuck a .mp4 extension on it. --codec libav was added as an option in #298, so would need a recent version of libcamera-apps.

Go for libcamera-vid --width 640 --height 480 --framerate 90 --codec h264 -o file.h264 -n --save-pts timestamps.txt. It'll create a file with the millisecond timestamp of each frame. 90fps would be 11.1ms between frames.

[naushir]

The latest version of libcamera-apps in apt does have libav integration. You might want to apt-update your libraries to run with the latest version.

[olokos]

Well, I have actually compiled both libcamera and libcamera-apps myself, just to be sure I am 1:1 with repo:

libcamera-apps build: 164da3ab69e1-intree 04-08-2022 (19:39:34)
libcamera build: v0.0.0+3805-961a6cf7

build date is exactly the same as the moment when I've built both. I also updated the entire OS, installed all possible dependencies (apart from debugging) and I'm also running latest kernel release, after that I built both.

I have hightlighted the exact specific commit of libcamera that I used for compiling libcamera-apps.

Also ffmpeg is already the newest version (7:4.3.4-0+deb11u1+rpt2).

[olokos]

file_and_timestamp.zip Above there's also the file and timestamp generated - 24ms betwen frames, 640x480 @ 30 fps in mediainfo

[naushir]

Well, I have actually compiled both libcamera and libcamera-apps myself, just to be sure I am 1:1 with repo:

The build system will only compile in libav integration if it successfully finds all the libav libraries (listed here). In you case, you don't have them installed, so the libav codec integration is unavailable.

It's really best that you use our libraries from apt when debugging these sorts of issues in case there is anything different about the build environments.

[naushir]

file_and_timestamp.zip Above there's also the file and timestamp generated - 24ms betwen frames, 640x480 @ 30 fps in mediainfo

Can you now try using the --mode command line argument to choose a particular sensor mode you want. You need to ensure the sensor mode matches one of those that are listed from the --list-cameras option.

[olokos]

Above is the config before trying to install the libraries you've mentioned, if it helps anyhow.

I was missing sudo apt install libavdevice-dev in my system, I'll rebuild now again.

I have just created a very nice bash script, which automatically pulls all changes and builds both libcamera and libcamera-apps (in this order).

I will post the results with --mode, after installing libavcodec-device and pulling all latest changes so the source will be 1:1 with both of the repositories.

If that also fails, I don't mind going for apt versions, just to be sure that's not the problem, but just a moment, it's building now, will provide results with --mode afterwards, I'll also try with libav afterwards.

[olokos]

libavcodec not found still :(

libcamera-vid --width 1332 --height 990 --framerate 90 --codec h264 -o file.h264 -n --mode 1332:990:90 --save-pts timestamps.txt

Mediainfo:

General
Complete name                  : C:\Users\olokos\Downloads\file.h264
Format                         : AVC
Format/Info                    : Advanced Video Codec
File size                      : 344 KiB
Duration                       : 3 s 200 ms
Overall bit rate mode          : Constant
Overall bit rate               : 881 kb/s

Video
Format                         : AVC
Format/Info                    : Advanced Video Codec
Format profile                 : High@L4
Format settings                : CABAC / 1 Ref Frames
Format settings, CABAC         : Yes
Format settings, Reference fra : 1 frame
Duration                       : 3 s 200 ms
Bit rate mode                  : Constant
Bit rate                       : 10 000 kb/s
Width                          : 1 332 pixels
Height                         : 990 pixels
Display aspect ratio           : 4:3
Frame rate                     : 30.000 FPS
Color space                    : YUV
Chroma subsampling             : 4:2:0
Bit depth                      : 8 bits
Scan type                      : Progressive
Bits/(Pixel*Frame)             : 0.253
Stream size                    : 3.81 MiB

24.994 is in the timestamps :(

[olokos]

From what I calculated, that gives 41,(6)FPS, on RPi 4B 4GB, with barely anything else running, also with a heatsink, to keep it cool even during load.

The first link in this issue contains my /boot/config.txt contents, maybe something is wrong there?

Here are the output files: new_file_timestamp.zip

[olokos]

This might be related to #315 I am also using X11 forwarding, but with -n parameter the preview is disabled, so this isn't a problem in itself.

olokos@RPi-Kerberos:~ $ libcamera-vid --width 1332 --height 990 --framerate 90 --codec libav -o file.h264 -n --mode 1332:990:90 --save-pts timestamps.txt
[0:46:21.702303651] [3238]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3808-26c82ce1
[0:46:21.739591560] [3239]  WARN V4L2 v4l2_pixelformat.cpp:303 Unsupported V4L2 pixel format SENS
[0:46:21.739722299] [3239]  WARN Formats formats.cpp:838 Unsupported pixel format 0x00000000
[0:46:21.740497919] [3239]  WARN V4L2 v4l2_pixelformat.cpp:303 Unsupported V4L2 pixel format BSTA
[0:46:21.740570936] [3239]  WARN Formats formats.cpp:838 Unsupported pixel format 0x00000000
[0:46:21.740820915] [3239]  INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media4 and ISP device /dev/media1
Stream configuration adjusted
[0:46:21.742081713] [3238]  INFO Camera camera.cpp:1029 configuring streams: (0) 1332x990-YUV420 (1) 2028x1520-SBGGR12_CSI2P
[0:46:21.742570985] [3239]  INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 2028x1520-SBGGR12_1X12 - Selected unicam format: 2028x1520-pBCC
[0:46:21.743353215] [3239]  WARN V4L2 v4l2_pixelformat.cpp:303 Unsupported V4L2 pixel format BSTA
[0:46:21.743416844] [3239]  WARN Formats formats.cpp:838 Unsupported pixel format 0x00000000
[0:46:21.744610644] [3239]  WARN V4L2 v4l2_pixelformat.cpp:303 Unsupported V4L2 pixel format SENS
[0:46:21.744706661] [3239]  WARN Formats formats.cpp:838 Unsupported pixel format 0x00000000
ERROR: *** Unrecognised codec libav ***
olokos@RPi-Kerberos:~ $ libcamera-vid --version
libcamera-apps build: 164da3ab69e1-intree 04-08-2022 (19:39:34)
libcamera build: v0.0.0+3808-26c82ce1

Having read the issue and PR I've noticed libavcodec-extra58, so I installed it:

olokos@RPi-Kerberos:~ $ sudo apt install libavcodec-extra58
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  libavcodec58
The following NEW packages will be installed:
  libavcodec-extra58
0 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.

but same result

[olokos]

On top of that, photos are VERY distorted, when there's even slight movement.

Command to generate images: libcamera-vid --width 1280 --height 720 --framerate 60 -n -t 0 --inline -o - | ffmpeg -i pipe: -c copy -f rtsp rtsp://127.0.0.1:8554/mystream

Sample image below:

It's just a person walking, so the speed isn't very fast, yet it's completely impossible to recognize the person... PS. The above command also produces 30fps stream.

I wouldn't be so suprised, if I would use some network to get the date, while the ip is literally localhost, what should I do?

Issues:

• Can't go above 30fps
• --framerate option changes nothing
• Using --mode does not change the framerate
• The distortion is massive, even when the movement is small.

What should I do about it?

[olokos]

Here are the vlc info about the stream: Bitrate seems to be quite high, but anyway with RTSP being localhost, this shouldn't make any difference, but it does.

The above screenshot is from a PC in a local LAN network, so dropped frames might(?) be normal considering I'm on 5GHz wifi, but the application that makes use of it and saved above distorted image, is running on the same machine as the camera is connected to, so it should be not dependent on the network, since I use 127.0.0.1 for RTSP on the same device that saves images from the RTSP.

Below is the iwconfig result.

IEEE 802.11  ESSID:"Private"
          Mode:Managed  Frequency:5.5 GHz  Access Point: PRIVATE
          Bit Rate=433.3 Mb/s   Tx-Power=31 dBm
          Retry short limit:7   RTS thr:off   Fragment thr:off
          Power Management:on
          Link Quality=60/70  Signal level=-50 dBm
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:531  Invalid misc:0   Missed beacon:0

I don't mind preview from a different PC dropping frames, but it seems to be dropping frames even on the raspberry pi locally, for some reason, causing those artifacts, maybe?

[olokos]

I have now tried invoking: libcamera-vid --width 1332 --height 990 --framerate 120 --mode 1332:990:120 -n -o testtoday.h264 --codec h264 -t 5000 --tuning-file /usr/share/libcamera/ipa/raspberrypi/imx477.json

Result is the following:

While there are no artifacts visible, it's still dropping frames, in this scenario it's like the camera stops recording for those dropped frames time and then starts recording again, while the RPi 4B 4GB is basically idle, CPU max 3% usage, 225MB/4GB ram used.

Here's the output of invoking command above:

olokos@RPi-Kerberos:~ $ libcamera-vid --width 1332 --height 990 --framerate 120 --mode 1332:990:120 -n -o testtoday.h264 --codec h264 -t 5000 --tuning-file /usr/share/libcamera/ipa/raspberrypi/imx477.json  [17:47:28.912487652] [11463]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3808-26c82ce1
[17:47:28.946029576] [11464]  WARN RPiController controller.cpp:43 This format of the tuning file will be deprecated soon! Please use the convert_tuning.py utility to update to version 2.0.
[17:47:28.948933193] [11464]  WARN V4L2 v4l2_pixelformat.cpp:303 Unsupported V4L2 pixel format SENS
[17:47:28.949006840] [11464]  WARN Formats formats.cpp:838 Unsupported pixel format 0x00000000
[17:47:28.949731790] [11464]  WARN V4L2 v4l2_pixelformat.cpp:303 Unsupported V4L2 pixel format BSTA
[17:47:28.949772586] [11464]  WARN Formats formats.cpp:838 Unsupported pixel format 0x00000000
[17:47:28.950003193] [11464]  INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media4 and ISP device /dev/media0
Stream configuration adjusted
[17:47:28.951286245] [11463]  INFO Camera camera.cpp:1029 configuring streams: (0) 1332x990-YUV420 (1) 2028x1520-SBGGR12_CSI2P
[17:47:28.951830217] [11464]  INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 2028x1520-SBGGR12_1X12 - Selected unicam format: 2028x1520-pBCC
[17:47:28.952631315] [11464]  WARN V4L2 v4l2_pixelformat.cpp:303 Unsupported V4L2 pixel format BSTA
[17:47:28.952672684] [11464]  WARN Formats formats.cpp:838 Unsupported pixel format 0x00000000
[17:47:28.953590594] [11464]  WARN V4L2 v4l2_pixelformat.cpp:303 Unsupported V4L2 pixel format SENS
[17:47:28.953690815] [11464]  WARN Formats formats.cpp:838 Unsupported pixel format 0x00000000
Halting: reached timeout of 5000 milliseconds.

[olokos]

libcamera-vid --width 1332 --height 990 --framerate 60 --mode 1332:990:60 -n --codec h264 -t 0 --tuning-file /usr/share/libcamera/ipa/raspberrypi/imx477.json --inline -o - | ffmpeg -i pipe: -c copy -f rtsp rtsp://127.0.0.1:8554/mystream

With this command again the fps is 30 and the mediainfo as in previous message.

The artifacts that I have posted earlier seem to be exactly the same situation, where the camera itself is dropping frames.

The difference between RTSP stream and pure libcamera-vid, is that: libcamera-vid frame droped = stream is freezing for a moment and unfreezing libcamera-vid + RTSP frame dropped = stream is NOT freezing, but instead trying to approximate the image between first and last dropped frame, resulting in artifacts.

to put it simple: RTSP will keep the stream going so artifacts are in place of frozen image libcamera-vid will freeze and then continue from the point of the next frame that arrived.

At this point I'm almost certain this isn't at all network related, but most likely related to the freezing problem of libcamera-vid itself, maybe connected to the always 30fps bug?

It might be pretty important to note, that it's not constantly freezing/artifacting on just movement alone.

Artifacting/freezing also does happen on still frames, every second or every few seconds, hard to tell precisely, it's just the most visible, when an object is moving, since most of the screen is covered in artifacts like in image before.

I can tell that as I have just watched the RTSP stream and people or cars moving are shown clearly, until the point of the freeze happening.

EDIT: I have just tried 1280x720@30FPS, with command as above with RTSP and it's not dropping so many frames. For 2min 45 seconds it lost 3 frames and discontinued 1.

I'm really not sure what's going on here...

I have 2 ideas: Despite using localhost and being run on local machine it's somehow still network dependent. Because of the FPS stuck at 30, there's some compression issues going on, causing the frame dropping and artifacts.

[olokos]

I have just tested the legacy camera stack again, building it myself from the latest repo commit. raspivid --width 1280 --height 720 --framerate 60 --mode 1280:720:60 -n -t 5000 -o raspivid72060.h264 - this command will save 720p25, but vlc plays it for 10 seconds approximately.

raspivid --width 1280 --height 720 --framerate 60 --mode 1280:720:60 -n -t 5000 -o raspivid72060.h264 --spstimings and the video is saved properly 720p60fps, everything looking good there.

With raspivid -o - -t 0 -n -w 2028 -h 1080 -fps 50 --spstimings | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.6:7447/}' :demux=h264

I was able to sucessfully get an RTSP stream of 2028x1080@50FPS!

Something which currently isn't possible on new camera stack.

EDIT: On the other hand, the night mode in raspi-vid isn't as good as I remembered it. I think it might have been raspistill nightmode, which will obviously be better, as there's more time for the sensor to capture light.

[naushir]

As commented multiple times, to help debug your issues, please stop piping libcamera-vid output to ffmpeg, or vlc, or anything else! It only confuses the situation for us.

I would advise you to run the following on a freshly installed and updated sdcard image (you may also want to add force_turbo=1 to your /boot/config.txt file to ensure clocks do not get throttled):

1) Test 1332x990 120fps mode.

libcamera-vid --frames 300 --denoise cdn_off -o /dev/null --save-pts pts.txt --mode 1332:990:10 --framerate 100 --width 1280 --height 720 -n

This uses the sensor 120fps mode, and scales the image to 720p in the ISP, and then finally through the video encoder. We have a timestamp analysis script in the libcamera-apps repo that you are free to use. Here are my results with the app:

pi@pi4:~ $ libcamera-apps/utils/timestamp.py pts.txt
Minimum: 9.936 ms at frame 75
Maximum: 10.054 ms at frame 74
Average: 9.996 ms
Total: 99 samples
Outliers: 0 (100%) 0 (10.0%) 0 (1.0%) 8 (0.1%)

I see no frame drops at 100fps. Note that I have set the framerate to 100 and not 120 for this test - we seem to have a small regression and at 120 fps and drop around 1% of frames. I need to investigate this.

2) Test 2028x1080 50fps mode

libcamera-vid --frames 100 --denoise cdn_off -o /dev/null --save-pts pts.txt --mode 2028:1080:12 --framerate 50 --width 1280 --height 720 -n

pi@pi4:~ $ libcamera-apps/utils/timestamp.py pts.txt
Minimum: 19.982 ms at frame 91
Maximum: 19.998 ms at frame 90
Average: 19.990 ms
Total: 99 samples
Outliers: 0 (100%) 0 (10.0%) 0 (1.0%) 0 (0.1%)

I see no frame drops at 50fps in this mode.

3) Test 2028x1520 40fps mode

libcamera-vid --frames 100 --denoise cdn_off -o /dev/null --save-pts pts.txt --mode 2028:1520:12 --framerate 40 --width 1280 --height 720 -n

pi@pi4:~ $ libcamera-apps/utils/timestamp.py pts.txt
Minimum: 24.991 ms at frame 36
Maximum: 24.999 ms at frame 35
Average: 24.995 ms
Total: 99 samples
Outliers: 0 (100%) 0 (10.0%) 0 (1.0%) 0 (0.1%)

As you can see, all the sensor modes work as expected. Even the 120fps mode gave the right framerate, just userland could not keep up for some reason and we dropped 1% of the frames.

You may have noticed the output goes to /dev/null. This is to remove any sdcard speed variability from the tests. Please report back what you numbers look like from these exact command.

[olokos]

Some of the results I've posted previously were without piping, but most of them could've been with piping indeed, sorry.

Here are the results, in the same order as You requested: https://pastebin.com/MQjYmsTw

I wanted to make also pts-3 but pressed enter too fast, anyway, it's overwritten and the numbers seem to be correct.

Maybe I need to specify some different command, so the resulting stream/recording would actually save proper information about framerate?

Anything similar to --spstimings for libcamera?

[naushir]

Some of the results I've posted previously were without piping, but most of them could've been with piping indeed, sorry.

Here are the results, in the same order as You requested: https://pastebin.com/MQjYmsTw

So looking at these results, all is working as I expected, and you are getting the expected framerate in all the sensor modes.

I wanted to make also pts-3 but pressed enter too fast, anyway, it's overwritten and the numbers are right.

What grinds my gears are the parameters used: Test 1332x990 120fps mode: --frames 300 (!) mode :12(bit depth?) and --framerate 100(that you already mentioned about the 120fps bug, I understood that) Test 2028x1080 50fps mode --frames 100(?) --mode 12(depth again?) --framerate 50 (but why declare it twice?) Test 2028x1520 40fps mode --frames 100(again 100?) --mode 12(depth?) --framerate 40 (again doesn't line up with first parameter)

It's just so confusing that the first command has parameter of 300 frames, while the other two have 100 frames. At the same time --framerate also doesn't line up with first parameter.

--frames tells the application to capture N frames then stop. I only put 300 frames on the fast framerate mode to get a larger sample set, that's it. You can choose however long you like through this parameter.

mode :12(bit depth?) You can have sensor modes that are identical in resoultion but differ in bit depth. This is a way of choosing the mode with the required bit depth. See here for details.

--framerate is what framerate you want on the output stream from libcamera. This can be different from the fastest possible framerate the sensor can produce. It is entirely possible (and likely) that the 2028x1080 50 fps sensor mode will only be used to produce a 30fps output stream typically. This is done by running the sensor at 30fps instead of 50fps. You only ever specify one value for framerate - that is the output stream framerate.

[naushir]

Maybe I need to specify some different command, so the resulting stream/recording would actually save proper information about framerate?

What more information do you need? I'm still unclear what problems you are encountering...

Anything similar to --spstimings for libcamera?

--inline ought to do this I think.

[olokos]

I'm sorry, I've edited my post moments before Your reply, as I've realised what the other parameters do, but indeed my wrong understanding of some parameters was not helping.

I've been looking for a way to create an RTSP stream based on libcamera-vid output, but there's just no way to get any stream above 30FPS.

Combining the 1st command with the pipe to ffmpeg & RTSP and --inline, I built this command: libcamera-vid --denoise cdn_off --mode 1332:990:10 --framerate 100 --width 1280 --height 720 -n -t 0 --codec h264 --tuning-file /usr/share/libcamera/ipa/raspberrypi/imx477.json --inline -o - | ffmpeg -i pipe: -c copy -f rtsp rtsp://127.0.0.1:8554/mystream

Results in 720@30fps

When I do: libcamera-vid --denoise cdn_off --mode 1332:990:10 --framerate 100 --width 1280 --height 720 -n -t 0 --codec h264 --tuning-file /usr/share/libcamera/ipa/raspberrypi/imx477.json --inline -o - | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.6:7447/}' :demux=h264

I'm also getting 30fps.

I just want to have a stream above 30fps with libcamera, as You can see in my previous messages, it was and still is possible with --spsdata, but --inline doesn't seem to do the same thing.

As I previously mentioned executing: raspivid -o - -t 0 -n -w 2028 -h 1080 -fps 50 --spstimings | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.6:7447/}' :demux=h264 results in proper 2028x1080@50FPS being streamed, but with libcamera it's not possible for some reason, 30fps is max.

[olokos]

I also have gstreamer and all required packages for streaming, but I wasn't able to start an RTSP stream with gstreamer and libcamera/raspivid at all, as I couldn't get the parameters right to both include rpi4b and RTSP stream, but I know it's possible, I just haven't figured out how to do it with gstreamer yet.

While ffmpeg and cvlc both stream max 30fps with libcamera-vid, which isn't a problem with raspivid.

I really want to try and use libcamera-vid and RTSP stream it, but I couldn't find a single way that would do above 30 frames/s

[olokos]

From what I know, ffmpeg will take the same framerate as source, so that makes me think that rapivid did send framerate info with --spstimings while libcamera doesn't, with --inline or not.

Source: https://superuser.com/a/1514949

This might be the entire problem here, libcamera-vid not outputting framerate information or am I wrong here?

PS: Did a small change to the command that I used 2 messages above this one, making width&height the same as the camera mode, hoping it helps with anything: libcamera-vid --denoise cdn_off --mode 1332:990:10 --framerate 100 --width 1332 --height 990 -n -t 0 --codec h264 --tuning-file /usr/share/libcamera/ipa/raspberrypi/imx477.json --inline -o - | ffmpeg -i pipe: -c copy -f rtsp rtsp://127.0.0.1:8554/mystream

But it's still 30fps regardless.

[naushir]

As 6by9 mentioned above, your streaming framerate is wrong as you are piping an elementary h.264 stream that does not have any timing information. So the applications (vlc/ffmpeg) assume 30fps always.

ffmpeg has a fps parameter that you could try using the specify the input stream framerate. Perhaps there is something similar in vlc as well, you ought to take a look?

You might also have better luck streaming with --codec libav (please use the apt versions of the apps for this!) with an mpegts output format, which does contain timestamp information. See here for more details.

[olokos]

It's just so weird, that both h264 and mp4 work with raspivid, but won't work with libcamera, seems like a downgrade as it requires more parameters for same result and forces using a single very specific coded, as compared to raspivid, which allows all codecs and also saves frame info properly, but I'm not a genius in those matters.

Executing this command: libcamera-vid --denoise cdn_off --mode 1332:990:10 --framerate 100 --width 1332 --height 990 -n -t 0 --tuning-file /usr/share/libcamera/ipa/raspberrypi/imx477.json --inline --codec libav --libav-format mpegts -o - | ffmpeg -i pipe: -c copy -f rtsp rtsp://127.0.0.1:8554/mystream Results in - ERROR: unrecognised option '--libav-format'

so I removed that parameter and used: libcamera-vid --denoise cdn_off --mode 1332:990:10 --framerate 100 --width 1332 --height 990 -n -t 0 --tuning-file /usr/share/libcamera/ipa/raspberrypi/imx477.json --inline --codec libav -o - | ffmpeg -i pipe: -c copy -f rtsp rtsp://127.0.0.1:8554/mystream

so the exact same, but without --libav-format ERROR: Unrecognised codec libav

ffmpeg later mentions WARNING: library configuration mismatch.

libavformat-dev is already the newest version (7:4.3.4-0+deb11u1+rpt2). libavcodec-dev is already the newest version (7:4.3.4-0+deb11u1+rpt2). ffmpeg is already the newest version (7:4.3.4-0+deb11u1+rpt2).

It's not possible to set fps for ffmpeg, since the command is using streamcopy (so should be fast) Filtergraph 'fps=15' was defined for video output stream 0:0 but codec copy was selected. Filtering and streamcopy cannot be used together. So ffmpeg is only able to produce 30fps

vlc in the meantime has --h264-fps=50 command, but with libcamera-vid, this also is always only 30fps stream.

libcamera-vid --width 2028 --height 1080 --framerate 50 --mode 2028:1080:10 -o - -t 0 -n | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.6:7447/}' :demux=h264 --h264-fps=50

I understand that the video produced with libcamera-vid does not contain any information regarding framerate in the output stream, regardless if I use .mp4 of .h264, but raspivid does.

This makes me believe there is a defect in libcamera-vid.

How else can you explain the fact that to achieve 60fps stream with raspivid all you need is this: raspivid -o - -t 0 -n -w 2028 -h 1080 -fps 50 --spstimings | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.6:7447/}' :demux=h264

While with libcamera-vid this has to be used: libcamera-vid --width 2028 --height 1080 --framerate 50 --mode 2028:1080:10 -o - -t 0 -n | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://192.168.1.6:7447/}' :demux=h264 --h264-fps=50

But still does not provide expected results, being locked to 30 fps, contrary to raspivid?

Reconverting/reencoding the stream once again seems like twice the performance hit, to gain the same result.

I am going to rebuild ffdshow aswell, including all dependencies, if that fails, I'll try prebuilt packages once again, but I remember far more issues with prebuilts than those I compiled myself.

[naushir]

so the exact same, but without --libav-format ERROR: Unrecognised codec libav

This is because you have built your own version of libcamera-apps without linking libav!

Please just use our prebuilt libraries and applications, they have libav correctly linked into libcamera-apps. Start with a fresh sdcard image if needed.

[olokos]

olokos@RPi-Kerberos:~ $ sudo apt install --reinstall libcamera-apps
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 265 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.raspberrypi.org/debian bullseye/main armhf libcamera-apps armhf 0~git20220707+35266e8-1 [265 kB]
Fetched 265 kB in 1s (463 kB/s)
(Reading database ... 252931 files and directories currently installed.)
Preparing to unpack .../libcamera-apps_0~git20220707+35266e8-1_armhf.deb ...
Unpacking libcamera-apps (0~git20220707+35266e8-1) over (0~git20220707+35266e8-1) ...
Setting up libcamera-apps (0~git20220707+35266e8-1) ...
Processing triggers for libc-bin (2.31-13+rpt2+rpi1+deb11u3) ...
olokos@RPi-Kerberos:~ $ libcamera-vid --denoise cdn_off --mode 1332:990:10 --framerate 100 --width 1332 --height 990 -n -t 0 --tuning-file /usr/share/libcamera/ipa/raspberrypi/imx477.json --inline --codec libav --libav-format mpegts -o - | ffmpeg -i pipe: -c copy -filter:v fps=15 -f rtsp rtsp://127.0.0.1:8554/mystream
Closing Libcamera application(frames displayed 0, dropped 0)
Camera stopped!
Tearing down requests, buffers and configuration
Camera closed
ERROR: *** unrecognised option '--libav-format' ***
ffmpeg version 4.3.4-0+deb11u1+rpt2 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 10 (Raspbian 10.2.1-6+rpi1)

This did not help.

I also couldn't find any mention on how to link the libav in any of the compilation guides, apart from ffdshow guide: https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu

In order to start with a fresh image, but still be able to use both legacy stack and the new stack, I'd have to download Raspberry OS Bullseye but 32bit, so I could still enable 64 bits and use 64 bit kernel, but with the option to switch to legacy, right?

I cannot just get the 64bit bullseye and easily switch to legacy stack from there?

[naushir]

If you want to switch to the legacy stack, you can only use the 32-bit image - specifically this image. It will also have the libav integration in libcamera-apps.

If you are unable to use this image, it may be best to keep with the legacy stack as this does exactly what you want.

[olokos]

I am compiling ffdshow myself now, for aarch64 and it seems to also compile libav.

I'll see if that helps, since it does contain libav source code in itself, I was planning to do that anyway sometime ago.

If self-compiling ffdshow won't help then I will start again with the specific image you've linked, thank You very much!

Currently the following prevents me from compiling ffdshow, but I'll try to figure it out:

In file included from ./libavutil/bswap.h:38,
                 from ./libavutil/intreadwrite.h:25,
                 from libavfilter/vf_curves.c:25:
libavfilter/vf_curves.c: In function ‘parse_psfile’:
./libavutil/aarch64/bswap.h:31:5: error: invalid 'asm': invalid operand for code 'w'
   31 |     __asm__("rev16 %w0, %w0" : "+r"(x));
      |     ^~~~~~~

[naushir]

You cannot build upstream ffmpeg/libav on our platform. We maintain our own fork with modifications. What you are trying to do with simply not work.

[6by9]

The V4L2 encoder does accept being told the framerate via VIDIOC_S_PARM on V4L2_BUF_TYPE_VIDEO_OUTPUT, and sets that in the SPS headers of the H264 stream, default being 30. However I don't see h264_encoder.cpp setting that, so the headers will include the default 30fps, but the timestamps (which is what should normally be relied on) will be correct.

struct v4l2_streamparm parm;
parm.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
parm.output.timeperframe.numerator = 1000 / options->fps;
parm.output.timeperframe.denominator = 1000;
if (xioctl(fd_, VIDIOC_S_PARM, &parm< 0)
   throw std::runtime_error("failed to set level");

or similar would solve that, although I don't know if the encoder gets passed the framerate in the options.

[olokos]

The V4L2 encoder does accept being told the framerate via VIDIOC_S_PARM on V4L2_BUF_TYPE_VIDEO_OUTPUT, and sets that in the SPS headers of the H264 stream, default being 30. However I don't see h264_encoder.cpp setting that, so the headers will include the default 30fps, but the timestamps (which is what should normally be relied on) will be correct.

From what I understand, you've just wrote a possible reason for the output framerate always being 30, reffering to the specific parts of the code, with a rough sketch of a possible fix to the problem? 😃

I have managed to workaround that asm error when compiling ffdshow by compiling for arm instead of arm64 and that issue is gone and it's compiling ffmpeg on RPi 4B now, if there won't be any other errors, ffmpeg should compile with release 5.0 as source branch.

I have configured ffdshow as follows:

./configure \
    --extra-cflags="-I/usr/local/include" \
    --extra-ldflags="-L/usr/local/lib" \
    --extra-libs="-lpthread -lm -latomic" \
    --arch=arm \
    --enable-gmp \
    --enable-gpl \
    --enable-libass \
    --enable-libdav1d \
    --enable-libdrm \
    --enable-libfreetype \
    --enable-libkvazaar \
    --enable-libmp3lame \
    --enable-libopus \
    --enable-librtmp \
    --enable-libsnappy \
    --enable-libsoxr \
    --enable-libssh \
    --enable-libvorbis \
    --enable-libvpx \
    --enable-libwebp \
    --enable-libx264 \
    --enable-libx265 \
    --enable-libxml2 \
    --enable-nonfree \
    --enable-version3 \
    --target-os=linux \
    --enable-pthreads \
    --enable-openssl \
    --enable-hardcoded-tables \
  && make -j$(nproc) \
  && sudo make install

I haven't managed to compile libaom, that's why I have not enabled it, but with the other options such as kvazzar and h265, I'm not sure if there's anything to gain by fighting with libaom compilation, if hardware acceleration can be used with other options above.

[olokos]

ffmpeg --version
ffmpeg version 54e0971 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 10 (Raspbian 10.2.1-6+rpi1)
  configuration: --extra-cflags=-I/usr/local/include --extra-ldflags=-L/usr/local/lib --extra-libs='-lpthread -lm -latomic' --arch=arm --enable-gmp --enable-gpl --enable-libass --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libkvazaar --enable-libmp3lame --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libssh --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-nonfree --enable-version3 --target-os=linux --enable-pthreads --enable-openssl --enable-hardcoded-tables
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
  libpostproc    56.  3.100 / 56.  3.100

It seems in the end while self-compiling I have lost a lot of optional configuration packages, but:

I have learned that ffmpeg -c copy means codec=copy so to change the output codec for ffmpeg to libkvazaar or libx265 -c copy has to be replaced with desired codec for output stream.

Unfortunately libav doesn't work for neither ffmpeg nor libcamera-vid still.

I have constructed this command: libcamera-vid --denoise cdn_off --mode 1332:990:10 --framerate 100 --width 1280 --height 720 -n -t 0 --tuning-file /usr/share/libcamera/ipa/raspberrypi/imx477.json --inline -o - | ffmpeg -i pipe: -codec libx265 -f rtsp rtsp://127.0.0.1:8554/mystream

With the above command, the stream at least changes the codec to actually be HEVC, but it doesn't help much since it's at around 4FPS... perhaps because I did not build oma, as the CPU gets maxed out and barely any frames are sent via RTSP.

The stream is still locked at 30 fps though.

[naushir]

I'm not going to repeat myself on how to get libav working since you seem to have ignored my earlier comments.

[olokos]

I'm sorry, I just wanted to make myself realise that self-compiling isn't a good way to go, especially since I was quite deep there.

I have just plugged in the MC into the PC and I'll install a fresh OS and try again.

[olokos]

I have now installed the exact same image and setup ssh etc. olokos@pi:~ $ libcamera-hello --version libcamera-apps build: 35266e84043f 07-07-2022 (17:54:29) libcamera build: v0.0.0+3700-f30ad033

libcamera now works with libav, exactly as you've mentioned, but I'm still unable to get an RTSP stream out of this, here's full log:

https://hastebin.com/wiwedoruci.apache and the part of the above log with the error:

with the input command being: libcamera-vid --codec libav --libav-format mpegts --denoise cdn_off --mode 2028:1080:10 --framerate 50 --width 2028 --height 1080 -n -t 0 --inline -o - | ffmpeg -i pipe: -codec copy -f rtsp rtsp://127.0.0.1:8554/mystream

Seems like I'm close, but not quite there yet.

To create the RTSP stream I'm using https://hub.docker.com/r/aler9/rtsp-simple-server using docker run -d --network=host aler9/rtsp-simple-server

after which I'm testing the libcamera-vid piping to ffmpeg.

If I remove libav and mpegts from the command, so it becomes: libcamera-vid --denoise cdn_off --mode 2028:1080:10 --framerate 50 --width 2028 --height 1080 -n -t 0 --inline -o - | ffmpeg -i pipe: -codec copy -f rtsp rtsp://127.0.0.1:8554/mystream

Then the stream works, but dropping 25% of the frames.

This is the mediainfo without libav:

And the result of the commandline without libav:

Yes, I am aware that you have told me to use libav and mpegts, but then that prevents me from getting RTSP stream completely, perhaps missing some other parameter?

I've also noticed that despite I did nothing else than sudo dist-upgrade and autoremove, there's still configuration mismatch warning?... It most likely relates to the fact that the build target=/=local hardware, which is odd.

[naushir]

From the documentation I linked to earlier, if you use --codec libav, there is no need to pipe the libcamera-vid output to ffmepg to setup a RTSP stream. You can do something like:

libcamera-vid --codec libav --libav-format mpegts --denoise cdn_off --mode 2028:1080:10 --framerate 50 --width 2028 --height 1080 -n -t 0 --inline -o "rtsp://127.0.0.1:8554/mystream"

which might work - I've not tried it myself.

I've pushed the change suggested by 6by9 at https://github.com/raspberrypi/libcamera-apps/pull/339 as it does no harm, and does correctly add the framerate to the SPS. With either of these approaches, it should give you what you want.

[olokos]

Awesome news!

I tried the command you've sent and here's the result:

olokos@pi:~ $ libcamera-vid --codec libav --libav-format mpegts --denoise cdn_off --mode 2028:1080:10 --framerate 50 --width 2028 --height 1080 -n -t 0 --inline -o "rtsp://127.0.0.1:8554/mystream"
[0:15:46.408834196] [1440]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3700-f30ad033
[0:15:46.456883981] [1441]  INFO RPI raspberrypi.cpp:1368 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media3 and ISP device /dev/media0
Stream configuration adjusted
[0:15:46.458115898] [1440]  INFO Camera camera.cpp:1029 configuring streams: (0) 2028x1080-YUV420 (1) 2028x1080-SBGGR12_CSI2P
[0:15:46.458548025] [1441]  INFO RPI raspberrypi.cpp:759 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 2028x1080-SBGGR12_1X12 - Selected unicam format: 2028x1080-pBCC
[h264_v4l2m2m @ 0x8af5a0]  <<< v4l2_encode_init: fmt=181/0
[h264_v4l2m2m @ 0x8af5a0] Using device /dev/video11
[h264_v4l2m2m @ 0x8af5a0] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x8af5a0] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0x8af5a0] Failed to set number of B-frames: Invalid argument
[h264_v4l2m2m @ 0x8af5a0] Failed to set gop size: Invalid argument
Output #0, mpegts, to 'rtsp://127.0.0.1:8554/mystream':
    Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, bt709), 2028x1080, q=-1--1, 200 kb/s, 1000k tbn
terminate called after throwing an instance of 'std::runtime_error'
  what():  libav: unable to open output mux for rtsp://127.0.0.1:8554/mystream: Protocol not found
Aborted

So almost!

Thank You very much for pushing the change!

Is there some way I could try out this fix, without having to recompile libcamera-apps, but just working with packages?

With --libav-format removed, so deduce this from the filename approach is used as explained in --help the result is different, but still fails:

olokos@pi:~ $ libcamera-vid --codec libav --denoise cdn_off --mode 2028:1080:10 --framerate 50 --width 2028 --height 1080 -n -t 0 --inline -o "rtsp://127.0.0.1:8554/mystream"
[0:27:11.028154197] [1532]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3700-f30ad033
[0:27:11.075976821] [1533]  INFO RPI raspberrypi.cpp:1368 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media3 and ISP device /dev/media0
Stream configuration adjusted
[0:27:11.077076847] [1532]  INFO Camera camera.cpp:1029 configuring streams: (0) 2028x1080-YUV420 (1) 2028x1080-SBGGR12_CSI2P
[0:27:11.077477325] [1533]  INFO RPI raspberrypi.cpp:759 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 2028x1080-SBGGR12_1X12 - Selected unicam format: 2028x1080-pBCC
[NULL @ 0x83e8d0] Unable to find a suitable output format for 'rtsp://127.0.0.1:8554/mystream'
ERROR: *** libav: cannot allocate output context ***
olokos@pi:~ $

[naushir]

You will likely need to find the correct libav syntax for the rtsp protocol. I've only ever tested it with tcp:// and udp:// streaming, not rtsp.

The only way to try out the fix right now is to recompile. It will eventually land in apt, but there is no timeline for that just yet.

[olokos]

Is there some easy way (or not easy, but very clean way) to uninstall the self-compiled libcamera-apps, so I could switch to packages again on a new release?

I know that ffmpeg has make uninstall but I wasn't able to find anything similar for libcamera-apps.

[naushir]

Unfortunately not. You will have to do this manually.

[olokos]

I'll try to compile with opencv this time around.

This will be my cmake command: cmake .. -DENABLE_DRM=1 -DENABLE_X11=1 -DENABLE_QT=1 -DENABLE_OPENCV=1 -DENABLE_TFLITE=0 -DENABLE_COMPILE_FLAGS_FOR_TARGET=armv8-neon

Currently waiting for pip install opencv-contrib-python to finish, but sadly it's compiling from source using a single thread, so it's been compiling for around 2-3 hours now :/

[olokos]

Well, I tried to install opencv twice, but not only it takes 3+ hours as it's single threaded bound compilation but it also failed twice. sudo pip install opencv-contrib-python also fails in the same way: https://pastebin.com/6d6htzhA

With that said I'll have to give up on opencv for now, I tried following this guide until step 4a: https://pyimagesearch.com/2019/09/16/install-opencv-4-on-raspberry-pi-4-and-raspbian-buster/

I tried compiling libcamera-apps with opencv regardless, but result is:

CMake Warning at post_processing_stages/CMakeLists.txt:34 (message):
  OpenCV support was enabled but no libraries found!

If you have any idea on how to get opencv with libcamera-apps working right, please let me know as I have a feeling I might have used a wrong guide for that.

So I'll give up on opencv for now, will build libcamera-apps with: cmake .. -DENABLE_DRM=1 -DENABLE_X11=1 -DENABLE_QT=1 -DENABLE_OPENCV=0 -DENABLE_TFLITE=0 -DENABLE_COMPILE_FLAGS_FOR_TARGET=armv8-neon

[olokos]

aaaaaaaaaand the libcamera-apps fail to compile aswell on a fresh new OS install...