This is a set of GStreamer 1.0 plugins for NXP's i.MX platform, which make use of the i.MX multimedia capabilities.
Currently tested on: i.MX6DL, i.MX6Q, i.MX7, i.MX8m quad/mini/plus, i.MX8 QuadMax/QuadXPlus
These plugins are licensed under the LGPL v2.
There is one decoder element for each format the i.MX SoC supports. The list of supported formats depends on
the i.MX variant. For example, if h.264 is supported, there will be a imxvpudec_h264
element.
The same is true of the encoder elements.
For the full list of formats that could theoretically be supported, check out the libimxvpuapi library version 2.3.0 or later.
Video transform, video sink, and compositor elements are available that perform their operations using hardware blitters.
The following element types are available (not all are possible with all 2D blitter APIs):
use-vsync
property to TRUE to make use of this.) Aspect ratio can be preserved - excess space on the
framebuffer is then letterboxed. Just like the videotransform elements, these video sinks can
handle GstVideoOverlayCompositionMeta data. However, see the notes below.NOTE: Compositor elements are only available with GStreamer 1.16 or later. Compositor support in GStreamer 1.14 was not yet in gst-plugins-base and had serious bugs.
NOTE: 2D blitter video sink elements do not work on i.MX8 machines, since these no longer use the older MXC framebuffer driver (which these video sink elements rely on).
The following blitters are supported by these elements:
All elements use internal "uploader" code that uploads frames into DMA memory if necessary. If incoming frames are not aligned in a way that is compatible with what the blitters require, internal frame copies are automatically done. These frame copies are CPU-based, so performance may suffer, but otherwise, such frames could not be processed at all.
GstVideoOverlayCompositionMeta support in video sinks needs to be explicitly enabled by setting
the render-overlays
property to TRUE. This is both for backwards compatibility and because of
flickering artifacts when vsync is not used. With vsync is turned off, that flickering appears,
because it is actually caused by tearing.
Consider the following example: A video frame is to be shown with a small overlay at the top left corner. Without vsync, it is possible that the scanout driver scans and displays the contents of the framebuffer after the main video frame was written to it but before the overlay is painted. Then, the next frame is written to the framebuffer, and this time, the scanout driver scans and displays the contents just after the overlay is finished. Due to this asynchronicity between overlay painting and scanout driver, the overlay sometimes is shown, sometimes is not shown, and flickering is the outcome.
Tearing is fixed by enabling vsync. So is this timing problem. VSync makes sure that the contents that were written to the framebuffer are shown only after all of the rendering is done. Consequently, when a video sink shall do the overlay composition, it is strongly recommended to also turn on vsync.
The mxc_v4l2
V4L2 driver on the i.MX6 for various capture devices such as the OmniVision OV5640 is
unfortunately severely broken, and requires numerous workarounds in userspace. Furthermore, it contains
several i.MX specific extras for working with physical addresses. For that reason, GStreamer's v4l2src
and v4l2sink
elements do not work on the i.MX6 for these devices.
To fix this, gstreamer-imx contains two elements that access V4L2 and apply workarounds for the driver issues. These elements are:
imxv4l2videosrc
: v4l2src
equivalent for capturing video framesimxv4l2videosink
: v4l2sink
equivalent for displaying video framesNOTE: These elements exist only as a workaround for these driver bugs, since fixing that driver is impractical. They are not meant to be used on any newer i.MX platform, since starting with the i.MX8, a different (and better) driver is used.
The -Dv4l2-mxc-source-sink
Meson option can be passed to meson to enable / disable these two elements.
If set to true
(the default value), they will be enabled. false
will disable them.
The imx-kernels support the Amphion VPU that is present on the i.MX8 QuadMax/QuadXPlus through the
V4L2 memory-to-memory API. However, imx-kernels 5.4 and older have a VPU driver that is riddled
with severe bugs that required substantial patching of GStreamer's V4L2 stack, causing the v4l2
elements to not behave in a stable manner. To be able to use the decoder, code was added to
the gstreamer-imx v4l2 plugin that enabkes Amphion decoder elements. Their names start with
imxv4l2amphiondec_
. For example, an h264 decoder is called imxv4l2amphiondec_h264
. With this,
it becomes possible to use the Amphion Malone decoder on the i.MX8 QuadMax/QuadXPlus even with
an unpatched GStreamer.
NOTE: Newer imx-kernels (5.15 and above) come with an entirely new Amphion VPU driver that works correctly, obviating the need for these special elements.
To enable/disable the Amphion decoder elements, use the v4l2-amphion
Meson feature option.
Also, the i.MX8 QuadMax/QuadXPlus SoCs contain the ISI (Image Sensing Interface), which can be
used for colorspace conversions and downscaling (but not upscaling). This functionality is
available through the V4L2 memory-to-memory API. But, like with the Amphion VPU driver situation,
the support for it in imx-kernels <= 5.4 is plagued with bugs, thus requiring patches on top
of GStreamer - or, as it is the case here, a custom transform element that takes care of
applying bug workarounds. And, just like with the VPU driver, newer imx-kernels (>= 5.15)
have an ISI mem-2-mem driver that works correctly, so this is mostly interesting for older
systems. The transform element is called imxv4l2isivideotransform
, and can be enabled and
disabled by setting the v4l2-isi
Meson option to true
or false
, respectively.
NOTE: The ISI transform device is typically not available by default. To enable it, modify the devicetree, for example like here (on a Toradex apalis-imx8 module):
--- a/arch/arm64/boot/dts/freescale/imx8qm-apalis-v1.1-eval.dts
+++ b/arch/arm64/boot/dts/freescale/imx8qm-apalis-v1.1-eval.dts
@@ -15,3 +15,15 @@
"toradex,apalis-imx8",
"fsl,imx8qm";
};
+
+&isi_0 {
+ status = "okay";
+
+ cap_device {
+ status = "okay";
+ };
+
+ m2m_device {
+ status = "okay";
+ };
+};
imxuniaudiodec
: audio decoder plugin based on NXP's unified audio (UniAudio) architectureimxmp3audioenc
: MP3 audio encoder plugin based on NXP's MP3 encoderAll DMA buffer allocators that are used by the elements implement the GstPhysMemoryAllocator interface. This makes it possible to integrate them with other GStreamer elements, as long as said elements can make use of the GstPhysMemory functionality to extract physical addresses. In addition, if the i.MX Soc BSP supports ION based DMA buffer allocation, the elements will use an allocator based on GstDmaBufAllocator. This allocator also implemnts the GstPhysMemoryAllocator interface, so with this, it is possible to access the underlying DMA buffer by its DMA-BUF file descriptor and also by its physical address, making integration even easier.
This is also why the old imxeglvivsink
element from the earlier "v1" branch is not present anymore -
it is now unnecessary, since glimagesink
and waylandsink
both are full replacements.
Furthermore, if applications or other GStreamer elements wish to access some of the gstreamer-imx specific functionality (particularly its allocators and interfaces), the gstimxcommon library can be used for this purpose, since its API is public.
Also, the videoparsersbad
plugin from the gst-plugins-bad
package in GStreamer is needed, since
this plugin contains video parsers like h265parse
, h264parse
, mpegvideoparse
(for MPEG1 and MPEG2),
and mpeg4videoparse
(for MPEG4).
You must also use a Linux kernel with i.MX additions for i.MX specific subsystems such as the VPU, IPU, PxP etc. Such kernels are typically referred to as "imx-kernels".
gstreamer-imx uses meson as its build system. Amongst other reasons, this makes integration with existing GStreamer build setups easier, such as Cerbero.
At least version 0.53.2 is required. (Older versions can (partially) work, but build scripts may require changes/workarounds.) Versions prior to 0.54.0 require passing the -Dsysroot
option; with Meson 0.54.0 and above,
using -Dsysroot
is optional. It allows for explicitly specifying the sysroot path.
First, set up the necessary cross compilation configuration for meson.
Then, create a build directory for an out-of-tree build:
make build
cd build
Now set up the build by running meson:
meson ..
You might want to look into the --buildtype=plain
flag if the compiler flags Meson adds are a problem.
This is particularly useful for packagers. Read here for more.
Also, you might be interested in the -Dprefix
and -Dlibdir
arguments to control where to install the
resulting binaries. Again, this is particularly useful for packagers.
Finally, build and install the code by running ninja:
ninja install
The following meson configuration options are available. Use them with the -D
flag. For example, to
set the vpu
option to "true", specify -Dvpu=enabled
.
See the Meson documentation about build option types to understand what values can be passed to these options.
Since most options are of type feature
, this is the type in use unless a different
type is explicitly defined in the list below.
vpu
: VPU en/decoder elements.uniaudiodec
: The imxuniaudiodec
element.mp3encoder
: The imxmp3audioenc
element.g2d
: 2D blitter elements based on the Vivante G2D API.g2d-based-on-dpu
: Special option for use with i.MX8 QuadMax and QuadPlus SoCs. G2D
is emulated via the DPU on these machines. The DPU behaves somewhat differently.
It is recommended to set this option to true
on these SoCs for better performance.
On all other SoCs, this must be set to false
(the default value). Type: boolean
.ipu
: 2D blitter elements based on the NXP Image Processing Unit (IPU).pxp
: 2D blitter elements based on the NXP Pixel Pipeline (PxP).imx-headers-path
: Path to extra imx kernel headers. These are used for IPU and PxP
code. The build scripts attempt to autodetect this path, so specifying this typically
is not necessary. Type: string
.imx2d-videosink
: Enables/disables building 2D blitter video sink elements.
Default value is true
. Setting this to false
makes sense on i.MX8 machines,
since rendering to the framebuffer is not possible on those. Type: boolean
.imx2d-compositor
: Enables/disables building 2D blitter compositor elements.
Type: boolean
.v4l2-mxc-source-sink
: Enables/disables building the custom Video4Linux2
source / sink elements. See the Video4Linux2 section above for details. Type: boolean
.v4l2-isi
: Enables/disables building the custom Video4Linux2 video transform element
that uses the ISI mem-2-mem device. See the Video4Linux2 section above for details.
Type: boolean
.v4l2-amphion
: Enables/disables building the custom Video4Linux2 Amphion Malone VPU
decoder that uses the Amphion mem-2-mem device. See the Video4Linux2 section above for
details. Type: feature
.package-name
: GStreamer package name to use in the plugins. Type: string
.package-origin
: GStreamer package origin to use in the plugins. Type: string
.