OMX H.265/HEVC support

[reufer]

I have an allpication using OMX for video decoding and I'm happy to finally have H.265 hardware support with the new Videocore! While other parts of the OMX API have been extended recently, there's still no proper value for OMX_VIDEO_CODINGTYPE. Has this just been forgotten or will there be H.265 support in OMX soon?

[6by9]

H.265 will never be supported via OMX or MMAL. Those are exclusively for functions running on the VideoCore VPU. IL is also officially an inactive standard from Khronos, therefore no changes will ever be forthcoming to the main standard.

There are no IP issues with the new hardware block, and it is supported from the ARM side. At present it is presented via custom patches for FFmpeg, but hopefully we'll expose it via V4L2's M2M API when time permits.

[dividuum]

Where is the HVEC decoder running, if not on the VideoCore VPU? Will there be a way to unify playback of both H264 & HVEC? Of do we have to resort to two different methods?

Could you link to those FFmpeg patches?

[popcornmix]

https://github.com/popcornmix/FFmpeg/tree/2711

[reufer]

I'd also appreciate a lot some clarification about the concept of these different approaches. The properly documented, relatively easy to use OMX/MMAL abstraction layers have always been a big pro argument for discussions when people started to compare the Raspberry Pi with the countless Allwinner boards on the marked. When I'm now forced to dig into ffmpeg patches for my tiny OpenMAX application, I start questioning myself...

[6by9]

Where is the HVEC decoder running, if not on the VideoCore VPU?

ARM userspace and kernel.

Will there be a way to unify playback of both H264 & HVEC? Of do we have to resort to two different methods?

Use FFmpeg or similar to insulate yourself from the differences.

[ghollingworth]

The OMX standard has been dropped by Khronos and therefore it is not in our interest in the long term to add future codec support to it. The MMAL interface is specifically for interfacing with the VideoCore co-processor for video acceleration.

This codec was designed to have ARM side interfaces so the Linux drivers can use open interfaces where possible. So it has been designed to plug into the FFMPEG hardware acceleration API.

As 6by9 says above, we'll provide V4L2 interfaces (which is the 'approved' way of communicating with video decoder and encoder hardware in Linux) as and when we get there

Gordon

[dividuum]

ARM userspace and kernel.

And the kernel then interfaces with some hardware for acceleration?

Use FFmpeg or similar to insulate yourself from the differences.

Sounds good in theory, although I'm not sure yet how that would exactly work. If there then a unified and programmatic way to both decode H264 & H265 into a dispmanx layer and into a GL texture? Both was possible in the past with OMX (render/egl_render).

[6by9]

It is possible to create ARM side MMAL components, and indeed there are a bundle in https://github.com/raspberrypi/userland/tree/master/interface/mmal/components, including avcodec_video_decoder.c. These were originally written in order to be able to test the MMAL framework on non-Pi hardware.

I couldn't say when the mmal_components_extra target was last built, and it's likely to have fun and games due to the libav vs ffmpeg library name mess from Debian/Ubuntu a few years ago, but it may give you an option to stick with MMAL.

IL is definitely not going to be updated.

[6by9]

And the kernel then interfaces with some hardware for acceleration?

Currently the same as /dev/gpio-mem - the appropriate registers are exposed through /dev/argon* nodes.

[    4.261672] argon-mem feb00000.hevc-decoder: argon-hevcmem initialised: Registers at 0xfeb00000 length 0x0000ffff
[    4.262129] argon-mem feb10000.argon-local-intc: argon-intcmem initialised: Registers at 0xfeb10000 length 0x00000fff
[    4.262491] argon-mem feb20000.h264-decoder: argon-h264mem initialised: Registers at 0xfeb20000 length 0x0000ffff
[    4.262820] argon-mem feb30000.vp9-decoder: argon-vp9mem initialised: Registers at 0xfeb30000 length 0x0000ffff

https://github.com/raspberrypi/linux/blob/rpi-4.19.y/arch/arm/boot/dts/bcm2838.dtsi#L412

[dividuum]

Not using IL isn't really a loss. I never liked working with it anyway :-)

It's just a lot new to absorb and figuring out how everything works together. Back on earlier releases, having the small examples in /opt/vc/src/hello_pi was really useful in messing around with the abilities and eventually understanding how everything works. Do you think you might rework/extend those to show the new abilities in a compact way?

[reufer]

I'm fully with @dividuum , having a minimal hello_video supporting both, H.264 and H.265 would be a great starting point for many peoples.

[JamesH65]

@reufer And it would have to use the base ffmeg libraries/similar, and would therefore need to be a completely new application that works in a completely different way. And probably 2 apps, one for X, one for console. It's all a bit non-trivial, but we will get round to it. The end result should be that we start to adhere to more standard Linux libraries rather than the vendor specific MMAL.

[ghollingworth]

I think the answer will be a simple version of ffmpeg... Which should support all formats directly

[c1ngular]

any updates on this ?

[6by9]

any updates on this ?

What particular "this" are you referring to? The first line of the second comment still stands.

H.265 will never be supported via OMX or MMAL.

Reworking examples isn't a priority. AIUI @dividuum has sorted it for Infobeamer, and Kodi/LibreElec is using it, so the information is available.

[dividuum]

I think the "never supported via OMX or MMAL" part is a bit confusing. The FFmpeg patch doesn't use or provide a MMAL component for decoding HEVC, but still does produce data usable in MMAL buffers. The remainder of a video player (scheduling, presentation) can still use normal MMAL components like the scheduler and the video_render component.

[reufer]

Reworking examples isn't a priority.

I'm sure there are more urgent things, however I'm sure these examples were the base for quite a lot of projects making the Raspberry Pi so popular. Referring to Kodi/LibreElec as an example may scare a lot of people... including me, when looking at the ffmpeg patch!

[6by9]

Anything can produce data that can be consumed by OMX or MMAL - they're only memory buffers after all. I could bitbash GPIOs and feed that into a buffer that is then presented to MMAL, but I wouldn't call it MMAL support for GPIOs.

Is it clearer if I state that H265 video decoding will not be available via a MMAL vc.ril.video_decode component?

(That has reminded me there was the MMAL avcodec_video_decoder component which wraps libavcodec as a MMAL component. It's complicated by being written against the libavcodec fork rather than ffmpeg's libavcodec, so confusion reigns supreme as to who wants what and where, and the APIs diverged. Now that the libavcodec vs FFmpeg war is over, pulling it back to FFmpeg would be potentially useful. If anyone wanted to have a play then PRs are welcome!)

[dividuum]

Is it clearer if I state that H265 video decoding will not be available via a MMAL vc.ril.video_decode component?

Much better ;-)

About the avcodec component: What's the state of the HEVC/FFmpeg integration? I noticed that the latest apt package now includes a patch for the hardware HEVC decoder. Is that how the API is going to work in the future as well once the V4L2 (?) method is used?

[reufer]

Anything can produce data that can be consumed by OMX or MMAL - they're only memory buffers after all.

Sure, that's how I'm already decoding audio with alsa. However, when using ffmpeg to decode video frames, that's another story in terms of efficient buffer handling. I just remember I wanted to understand the ffmpeg patch some time ago and there were lot of tweaks to avoid buffer copying.

I really wish to have a simple example of H.265 video decoding and passing it to the render, MMAL or OMX. I don't even care about audio and scheduling, it's just to have a working code as "minimal viable player" ;-)

[6by9]

About the avcodec component: What's the state of the HEVC/FFmpeg integration? I noticed that the latest apt package now includes a patch for the hardware HEVC decoder. Is that how the API is going to work in the future as well once the V4L2 (?) method is used?

(Going off on a tangent, but never mind). FFmpeg is going to be able to decode HEVC using hardware acceleration on the Pi4 via either V4L2 stateless or the patches direct to the register mappings. AIUI it should be able to automagically switch between any available codec when it is asked to decode HEVC, therefore it shouldn't matter which one has been enabled when building FFmpeg, and it should even drop back to software or the Pi3 hardware assisted HEVC decode if available. Hopefully that means that avcodec_video_decoder component can be agnostic to exactly which codec implementation to use.

The efficient way of getting the buffer will be via AV_PIX_FMT_DRM_PRIME buffers (aka dmabufs). This is still being thrashed out within FFmpeg-land as to "the correct way" to implement this, so it'll come at some point. Those can be either presented to DRM for rendering, or imported into vcsm for use as zero copy MMAL buffers. Supporting this is optional, so no need to try and get it working with avcodec_video_decoder for now.

[imbens]

I noticed that the latest apt package now includes a patch for the hardware HEVC decoder.

Would that be the 4.1.4-1+rpt1~deb10u1 version of the package? How do you see that it has a patch for the HEVC decoder?

[XECDesign]

ffmpeg 7:4.1.4-1+rpt6~deb10u1 vlc 3.0.8-0+deb10u1+rpt7

https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=257395