Enhancing OpenWrt Integration: Publish Rockchip's MPP Module to Mainline Kernel

[joopdo]

This ticket proposes the publication of Rockchip's Media Processing Platform (MPP) module to the mainline Linux kernel, with the goal of enhancing integration with OpenWrt, an open-source operating system for routers and embedded devices.

The MPP module is a critical component for multimedia tasks and is essential for applications and devices utilizing Rockchip's SoCs. By making the MPP module available in the mainline kernel, it ensures compatibility with OpenWrt and provides a valuable resource for developers and users.

Publishing the MPP module to the mainline kernel brings several advantages. It simplifies the integration process of Rockchip's SoCs into OpenWrt, streamlines multimedia processing across devices, and improves performance, stability, and interoperability. Additionally, it allows Rockchip to benefit from community contributions and ongoing development efforts, resulting in a more robust ecosystem for their SoCs.

This collaborative effort between Rockchip and the open-source community would foster innovation, advancement, and broader reach for Rockchip's SoCs in Linux-based distributions and projects that rely on the mainline kernel.

A relevant discussion on the OpenWrt community forum sheds light on the issue, specifically regarding the Nanopc-T6 board and its compatibility with OpenWrt. The thread, which can be found at This forum post

[mo123]

@joopdo There is already v4l2 drivers(H264, H265, VP9, AV1) work in progress replacing mpp in the mainline kernel for Rockchip devices.

[HermanChen]

@joopdo The current MPP module has two parts: the userspace libmpp.so in the repo and the mpp_service in kernel drivers/video/rockchip/mpp. This framework simplifies the kernel driver and keep most complicated and flexible part in userspace libmpp.so. We use this framework in consider of our product requirement. We need to support multiple kernel version and have to update custorm's very old version firmware to fix issues. The kernel update is quite hard for already mass-produced firmware so we choose to update libmpp.so in userspace for custerms with different kernel version. This reduce our workload a lot. About the kernel v4l2 driver it is quit hard to adapt to diffferent user requirements for example fast display on start playing. And also move all video codec protocol handling code to kernel is very hard to be accepted for kernel maintainers. So base on these historical and realistic consideration we do not develop a common v4l2 like driver in mainline kernel. But for next generation MPP driver we are considering move the whole driver to kernel.

[joopdo]

@HermanChen Thank you for providing the explanation regarding the current MPP module setup and your considerations for its design and insights.

I understand that maintaining extensive kernel updates would require to support multiple versions and maintaining older environments. The solution you found would allow for quicker updates for your product requirements.

However, for Rockchip's success case, it would be best once the kernel is mainline. Therefore I'm looking forward to the next generation MPP driver in the kernel, as average users won't apply patches and recompile drivers and compatability remains theoretical until that moment.

[nyanmisaka]

@HermanChen For many years, the support of Rockchip video decoder/encoder/iep/rga in the Linux mainline has been very unsatisfactory. I agree with you that community-contributed V4L2 has been slow to progress and limits the capabilities of MPP due to inflexibility. There is even a stop-gap like libv4l-mpp, but it is not generic and requires hacking the Chromium browser.

What are your thoughts on adopting Libva/VA-API? This is a video framework widely used by Intel and AMD GPUs and is closely related to DRM. A common approach is to maintain a driver in KMD (e.g. gpu/drm/rockchip/rkvideo), which is mainly responsible for scheduling, power management, registers and CS(Command Stream, but i don't think RK hw have this). This is similar to what mpp_service currently does. In this way, the codec protocol and HAL can be placed in UMD (e.g. rockchip_drv_video.so) for easy modification without going to disturb KMD side of things once the DRM uapi is finalized.

MPP can be derived from VA-API UMD driver and expand advanced functions. Just like the relationship between Intel's MediaSDK/oneVPL and media-driver.

If we go this way, we can get full zero-copy decode->display and transcoding support in nearly all common Linux software, including but not limited to Chromium, MPV player, ffmpeg, and gstreamer. Another point is that VA-API can also be used on Android.

I read the code of Rockchip's former employee who once wanted to use V4L2 to implement a VA-API wrapper, but it seemed to have been discontinued long ago. The community has become tired of such wrapper or shim to MPP, and it is unlikely that upstream software will link to MPP directly. Or preferably, do you have a better approach to support at least one open standard and would you mind sharing it?

[prahal]

@joopdo I believe @HermanChen meant that they will include their MPP stack in their fork. I cannot see MPP ever included mainline. So your only way to use MPP will be using rockchip's fork of the kernel.