search

raspberrypi / Raspberry-Pi-OS-64bit

Repository for containing issues on the 64 bit operating system (as distinct from the 32 bit one)
466 stars 21 forks source link

ffmpeg h264_omx doesn't work #98

Closed gogobd closed 1 year ago

gogobd commented 3 years ago

Hello, every 01!

ffmpeg lists h264_omx as a supported Decoder/Encoder, however if I try to use it it says

[h264_omx @ 0x5578fb7e10] libOMX_Core.so not found7014:32:22.77 bitrate= -0.0kbits/s speed=N/A
[h264_omx @ 0x5578fb7e10] libOmxCore.so not found

I tried to compile ffmpeg from source which worked except that it complains that it can't find the headers:

ERROR: OpenMAX IL headers from raspberrypi/firmware not found

I have installed libomxil-bellagio-dev but can't get hardware support for ffmpeg either way.

lurch commented 3 years ago

Possibly related to the MMAL changes? #10 & https://github.com/raspberrypi/Raspberry-Pi-OS-64bit/issues/86#issuecomment-674852181

gogobd commented 3 years ago

It's really hard for me to tell what's missing, however the speed impact of losing OMX ist remarkable. I noticed a header file is present:

/usr/include/bellagio/omxcore.h

But including it didn't help. When I tried to compile ffmpeg from source I followed this tutorial "Compiling FFmpeg on your Raspberry Pi" and this is my configure step:

./configure --extra-cflags="-I/usr/local/include" --extra-ldflags="-L/usr/local/lib" --extra-libs="-lpthread -lm -latomic" --enable-gmp --enable-gpl --enable-libaom --enable-libass --enable-libdav1d --enable-libdrm --enable-libfdk-aac --enable-libfreetype --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libssh --enable-libvorbis --enable-libvpx --enable-libzimg --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-nonfree --enable-omx --enable-omx-rpi --enable-version3 --target-os=linux --enable-pthreads --enable-openssl --enable-hardcoded-tables
gogobd commented 3 years ago

This post suggests that mmal doesn't affect omx, though:

https://stackoverflow.com/questions/40175644/ffmpeg-hardware-acceleration-on-raspberry-pi

--enable-mmal – Enable Broadcom Multi-Media Abstraction Layer (Raspberry Pi) via MMAL. For hardware decoding of H.264, VC-1, MPEG-2, MPEG-4. As a dependency you'll need the linux-raspberrypi-headers (Arch Linux) or linux-headers-*-raspi2 (Ubuntu) package which provides the required header file mmal.h.

--enable-omx-rpi – Enable OpenMAX IL code for Raspberry Pi. For hardware encoding of H.264 (encoder is named h264_omx) and MPEG-4 (mpeg4_omx). As a dependency you'll need the libomxil-bellagio (Arch Linux) or libomxil-bellagio-dev (Ubuntu) package which provides the required header file OMX_Core.h.
6by9 commented 3 years ago

OpenMax IL will NEVER be supported on 64bit. The hell involved in mapping 64bit pointers to pass to the 32bit VPU just isn't worth it.

The V4L2 stateful codec API is the preferred approach for codecs, and is available in both 32 and 64bit OSes. In FFmpeg use h264_v4l2m2m instead.

2bdkid commented 3 years ago

@6by9 does h264_v4l2m2m use any hardware support? I'm not familiar with it.

6by9 commented 3 years ago

does h264_v4l2m2m use any hardware support? I'm not familiar with it.

Yes, it's a V4L2 API wrapper on top of the MMAL components, but doing that allows us to add 64bit compatibility relatively easily. Userspace MMAL usage is currently not compatible with 64bit, but will be fixed up when resource allows.

2bdkid commented 3 years ago

@6by9 So I'm curious now. On my RPI4 (64bit), ffmpeg has the h264_v4l2m2m encoder available - I take it it's not using omx underneath so then what is it actually doing? Some other software encoder through MMAL?

lurch commented 3 years ago

A quick search finds https://www.kwasi-ich.de/blog/2017/11/26/omx/ which seems to suggest that OpenMax is a portable-wrapper on top of the Broadcom-specific MMAL. But https://www.raspberrypi.org/documentation/raspbian/applications/camera.md says "All the applications are driven from the command line, and written to take advantage of the MMAL API which runs over OpenMAX. The MMAL API provides an easier to use system than that presented by OpenMAX. Note that MMAL is a Broadcom-specific API used only on VideoCore 4 systems." :man_shrugging:

6by9 commented 3 years ago

h264_v4l2m2m uses the V4L2 stateful decoder API to then access the hardware codec (encode and decode) via the MMAL framework.

MMAL and OpenMAX IL both share the components at the lowest level. That was termed internally as Reduced-IL (or RIL) as it removes all the common state machine stuff from the individual components to give a reduced API.

MMAL and IL provide two different routes down to the RIL level, with MMAL being the more recent, and tries to get rid of a load of the bits of IL that caused grief (particularly all the asynchronous callback for completing state transitions and the like).

The MMAL framework code within the kernel has been adapted to handle 64bit kernels. Where the client needs to pass a 64bit pointer through a 32bit IPC field, it goes through an allocation and lookup table. The MMAL userspace library did have the same done (https://github.com/raspberrypi/userland/pull/586), but it has currently been reverted due to observed issues.

If you felt like picking IL apart to do the same, then a PR would be welcome. It's far from straightforward though as for a start you have 6 userspace pointers stored in OMX_BUFFERHEADERTYPE to start with, each of which need to be picked apart, stored, and looked back up again on buffer return. All that is open source in https://github.com/raspberrypi/userland/blob/master/interface/vmcs_host/vcilcs.c (VideoCore IL Component Service) and the other counterpart files, but it's not the easiest code to follow. It's bad enough that there are also non-natively packed structs in the default IL headers (the nTimeStamp field in OMX_BUFFERHEADERTYPE being the most annoying), so you'd need to repack that struct between 64bit and the IPC.

lurch commented 3 years ago

Thanks for the detailed info @6by9 ! :+1:

gogobd commented 3 years ago

h264_v4l2m2m produces green video output - so I would consider it broken. I also wonder where this could be reported.

6by9 commented 3 years ago

Green output but a visible image? The chroma planes are set to 0. I will double check, but I'm 99.99% certain it's an FFmpeg bug that is resolved in their master but not in the Debian version.

Following normal Debian policies of not changing major version within a release you won't find FFmpeg being significantly bumped, but we can see if that fix can be simply backported.

JooJooBee666 commented 2 years ago

I'm still waiting on this before I can even use the x64 version here. And it sucks as I'm doing live encoding of video stream from a queue of frames in RAM (doing surveillance with motion detection). The 2GB limit is killing me. But, this is a deal breaker. Is there just no hope this will ever be resolved?

6by9 commented 2 years ago

As https://github.com/raspberrypi/Raspberry-Pi-OS-64bit/issues/98#issuecomment-715321951,

OpenMax IL will NEVER be supported on 64bit.

I thought h264_v4l2m2m had been fixed, but haven't double checked.

JooJooBee666 commented 2 years ago

Hmm, I missed that. I just saw this issue was still open. I'll test out ffmpeg with h264_v4l2m2m instead. I guess if that is working, and knowing the history with OMS, maybe this should be closed with "won't fix" and a final reference to use the other hardware encoding option would be best?

gogobd commented 2 years ago

Please leave a comment how to get hardware support for decoding / encoding with ffmpeg if you find a solution! I'm still using older Pis which are much faster because they do support this.

Andrei-Iosifescu123 commented 1 year ago

h264_v4l2m2m produces green video output - so I would consider it broken. I also wonder where this could be reported.

I have the same problem.

popcornmix commented 1 year ago

Make sure you are on an updated RpiOS bullseye image. If you still have an issue then you can report it here. Include enough detail to reproduce the problem.

Andrei-Iosifescu123 commented 1 year ago

Make sure you are on an updated RpiOS bullseye image. If you still have an issue then you can report it here. Include enough detail to reproduce the problem.

I'm on Ubuntu 20.04

ghollingworth commented 1 year ago

I think you should report it to Ubuntu then

gogobd commented 1 year ago

I've seen this green output problem on other architectures, too. It could be a ffmpeg problem.

JamesH65 commented 1 year ago

IIRC, an all-green screen is what you get when a YUV image is entirely 0s.

Andrei-Iosifescu123 commented 1 year ago

I think you should report it to Ubuntu then

It works when compiling ffmpeg from source, but not when installing with apt get-install ffmpeg.

popcornmix commented 1 year ago

It works when compiling ffmpeg from source, but not when installing with apt get-install ffmpeg.

Which suggests Ubuntu are building the apt ffmpeg without support for this. Either not using the source you are building from, or not configuring with the required options.

That's not something we can help you with here (on an RPiOS issue tracker). You need to report this to Ubuntu.