dropping mosaic frame

[atiti]

Hi there,

Love the library, so much cleaner and understandable! I'm trying to decode an intra-refreshed h264 stream (with only P-frames), and the decoder is returning "trace: dropping mosaic frame" for each frame.

Do you have any ideas on what could be the cause?

Cheers, Attila

[dv1]

Hi,

thanks! The mosaic frames are somewhat mysterious. Would it be possible to generate a test stream that triggers this problem? When I generate a test stream with GStreamer and x264enc with intra-refresh enabled, I don't see this problem.

The pipeline I used (with GStreamer 1.2.0) is:

gst-launch-1.0 videotestsrc num-buffers=300 ! x264enc intra-refresh=true byte-stream=true ! filesink location=test-intra-refresh.h264

[atiti]

Hi dv1,

Thanks for the fast reply! Here is a test .h264 file which gets the mosaic frame issue:

http://curl.io/get/xz4cwwk2/9892e7e3f1a3c9594282973b9165dbfc6039410b

Plays back with mplayer fine.

[dv1]

I don't get a file. Just "oops".

[atiti]

Sorry, try this: http://attila.zsuatt.com/test.h264

[dv1]

This file appears to be broken. When I play this with anything libav-based, I get tons of errors about too many slices. Furthermore, in the beginning, there are dozens of SPS/PPS and SEI NAL units, and only afterwards there are slices. Also, this isn't using access unit delimiters, and also is not using the byte-stream format (Annex B in the h264 spec). Try encoding with byte-stream and access unit delimiters.

[atiti]

I think the errors come from the fact that libav/ffmpeg is by default compiled with 16 slices per frame max, but we have more slices as we would like to have NALs fitting into the MTU. As far as I can tell, it is already Annex B, at least I can see it in the NAL headers: !! Found NAL at offset 4 (0x0004), size 25 (0x0019) XX 00 00 00 01 67 4D 40 28 A6 80 78 02 27 E5 84 00

==================== NAL ==================== forbidden_zero_bit : 0 nal_ref_idc : 3 nal_unit_type : 7 ( Sequence parameter set ) ======= SPS =======

It's pretty weird with so many SPS/PPS and SEIs, but apparently it plays back fine with gstreamer-0.10 with vpudec.

[dv1]

Right, I missed the byte stream start markers. But still, it is not using the access unit delimiters. Try adding these.

[atiti]

Added it, but seems to be the same: http://attila.zsuatt.com/test3.h264

I've also disabled I-frame searching in vpu wrapper, as that was a problem in case of gstreamer as it was waiting for the never coming I-frame.

[dv1]

I cannot play this with vpudec. Can you provide me with a gst-launch-0.10 pipeline that uses vpudec and can play it?

[atiti]

Hi dv1,

Sorry for the late reply, I've been out travelling so I haven't been online. I'm using the following pipeline, which plays it back fine:

# gst-launch filesrc location=test.h264 ! h264parse ! vpudec ! mfw_v4lsink sync=false
MFW_GST_V4LSINK_PLUGIN 3.0.9 build on Aug 26 2014 17:32:46.
Setting pipeline to PAUSED ...
[INFO]  Product Info: i.MX6Q/D/S
vpudec versions :)
    plugin: 3.0.9
    wrapper: 1.0.46(VPUWRAPPER_ARM_LINUX Build on Aug  1 2014 16:50:55)
    vpulib: 5.4.23
    firmware: 3.1.1.46056
Pipeline is PREROLLING ...
[INFO]  bitstreamMode 1, chromaInterleave 1, mapType 0, tiled2LinearEnable 0
[WARN]  VPU iram is less than needed, some parts don't use iram
set v4l rotate sucessfully
>>V4L_SINK: Actually buffer status:
    hardware buffer : 5
    software buffer : 0
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
full screen size:1920x1080
[V4L Update Display]: left=0, top=0, width=1920, height=1080
set v4l display crop sucessfully
New clock: GstSystemClock
[INFO]  bitstreamMode 1, chromaInterleave 1, mapType 0, tiled2LinearEnable 0
[WARN]  VPU iram is less than needed, some parts don't use iram
[INFO]  bitstreamMode 1, chromaInterleave 1, mapType 0, tiled2LinearEnable 0
[WARN]  VPU iram is less than needed, some parts don't use iram
Got EOS from element "pipeline0".
Execution ended after 4243097667 ns.
Setting pipeline to PAUSED ...
Running time 0:00:04.245257334 render fps 38.631
Setting pipeline to READY ...
Setting pipeline to NULL ...
Total rendered:164
Freeing pipeline ...

I did have to disable I-frame searching in libfslvpuwrap, the patch you can find here in this thread: https://community.freescale.com/message/342913#342913

[dv1]

No, it still doesn't work. I suspect it is because I have the 3.0.11 fsl plugins for GStreamer 1.0 , while you have the 3.0.9 ones.

However, I am working on an alternate backend which doesn't use the VPU wrapper, but works with imx-vpu directly. It is able to display the video (the example program expects access units delimiters though; not sure yet if the VPU itself requires them). It will take a while to be ready, though. I'll push it soon, but keep in mind, its still in an early stage.

Either way, your bitstream appears broken. libav gives me tons of slice number related errors, the huge number of SPS/PPS and SEI NALUs at the beginning is very weird, and the first few frames of the video are broken. I'll try to get it working properly, but you should investigate why the bitstream is so strange.

[atiti]

Awesome, looking forward to using the one without the wrapper. Seems like there are quite a few hacks in the wrapper...

On another note, do you have some examples on using the v4l output / IPU output? I've been studying the gstreamer code, but seems pretty complex too..

[dv1]

This project unfortunately got delayed a lot. I can finally continue, and just pushed the first working version of the vpulib backend. Note that the encoder is not implemented yet there.

I never did anything with the V4L output, but recommend to use G2D instead of IPU for displaying frames. IPU is very picky with regards to the input parameters, and requires tiling sometimes. G2D is much more generic. It supports less formats, but this is mostly irrelevant for video playback, since the VPU outputs I420 data for all bitstream formats (except in some cases with MJPEG data).

[atiti]

Looks awesome! Will play around with it. :-)

Since the last post we've implemented GLES for display, so we can also use the GPU for post processing.