search

Freescale / libimxvpuapi

i.MX VPU API Library
GNU Lesser General Public License v2.1
88 stars 54 forks source link

Delayed decoding of IDR NAL (slice_type 2) for baseline profile h264 bitstream #21

Closed elisescu closed 3 years ago

elisescu commented 7 years ago

I'm trying to decode a h264 bitstream that is baseline profile on an iMX6 VPU decoder, but it seems the decoder is always with a frame behind.

I'm giving to the decoder a SPS and a PPS NAL (both can be found in frame_0.h264 attached here) and then I'm giving it a IDR NAL (which is in frame_1.h264 attached here). The slice_type of this IDR frame is 2 (see the h264_analyze_output.txt). I expect that right after processing the IDR NAL to get a decoded RAW frame. However the decoder doesn't produce anything, giving back a "not enough input data" return code, which seems to come because of the "decodingSuccess" being set to "no data decoded or ready for display" (you can see the attached logs in decoder_logs.txt). I've added some //elisescu: tags in the log to point out the key lines there.

The decoder should be able and decode the IDR NAL to a RAW frame, since it has all the needed information to do so. I've tried doing that with ffmpeg, and ffmpeg can decode it (just concatenated frame_0.h264 and frame_1.h264 and decoded them with ffmpeg).

On top of this, what is more curious is that, if I give again to the decoder the SPS and PPS NALs (the same ones that were fed before the IDR NAL), right after the IDR NAL, then it outputs the decoded picture. So: SPS, PPS, IDR => no decoded raw frame, but SPS, PPS, IDR, SPS, PPS => decoded raw frame This doesn't make much sense to me, since the two NALs don't bring new info for the decoder to be able to decode the previously sent IDR NAL.

Because the iframe is only decoded when the next nal is available, we get a pointless delay in the stream. We use the decoder for live streaming. If there is no motion on the screen, there are no input frames, so the delay to the next frame can be very large. Also, when the decoder finally outputs the decoded I-frame, the pts is incorrect. It is taken from the following frame instead.

I've attached more info about the NALs as dumped with h264_analyzer to the file h264_analyze_output.txt.

I should also mention that this is not the same behaviour I get when I feed the decoder with a different bitstream (encoded with a different h264 encoder, x264 in this case, with main profile - this makes the IDR to have the slice_type 7). In this case, the decoder always outputs the decoded frame right after it was fed with the IDR NAL, as expected.

decoder_logs.txt h264_analyze_output.txt frames_0_1_h264.zip

antarespilot commented 5 years ago

Hi Elis, did you figure out any way to work around this issue? We're seeing the same thing here, same kind of h264 stream. We've tested the same stream with Openh264 which has no issues decoding the IDRs.

elisescu commented 5 years ago

@antarespilot, we ended up not using libimxvpuapi anymore. We had too many issues with it, so switched to use directly vpulib.

dv1 commented 5 years ago

@elisescu Sorry about the silence. I am 100% busy with the rewrite for imx8. Can you send me a summary of the issues you had with it? Since I have redesigned the API, it would be good to get that feedback.