Failure to decode video when first frame doesn't result in a decoded frame

[dougnazar]

Found some files in our library that are failing with the 2.x branch. It fails trying to push the second frame, since even though IMX_VPU_API_DEC_OUTPUT_CODE_MORE_INPUT_DATA_NEEDED was returned, staged_encoded_frame_set is still set.

I added a decoder->staged_encoded_frame_set = FALSE; to imx_vpu_api_dec_decode() right after setting *output_code = IMX_VPU_API_DEC_OUTPUT_CODE_MORE_INPUT_DATA_NEEDED; and that seems to work but am not that confident if that's the correct fix. It doesn't seem to be leaking anything, but I'm wondering if it should be more like the VP8 workaround with it skipping an internal frame.

Input #0, avi, from '/mnt/Data/Songs/311900.avi':
  Duration: 00:04:55.24, start: 0.000000, bitrate: 337 kb/s
  Stream #0:0: Video: mpeg4 (Simple Profile) (XVID / 0x44495658), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 136 kb/s, 25 fps, 25 tbr, 25 tbn, 25 tbc
  Stream #0:1: Audio: mp3 (U[0][0][0] / 0x0055), 44100 Hz, stereo, fltp, 192 kb/s

gst.log

[dv1]

Do you have an example video that replicates this?

[dougnazar]

I was going to say it was so long ago I'd forgotten which file but lucky it was in the above comment. I've trimmed it down to 10 frames and verified that it fails to play if I remove that line.

https://nazar.ca/311900.avi

0:00:05.905330449 27003 0xb055d4c0 LOG                imxvpudec gstimxvpudec.c:904:gst_imx_vpu_dec_decode_queued_frames:<imxvpudecmpeg4-0> acquired gstbuffer 0xb1e44f38 with DMA buffer 0xb045ee78
0:00:05.908747040 27003 0xb055d4c0 LOG                imxvpudec gstimxvpudec.c:1462:gst_imx_vpu_dec_decode_queued_frames:<imxvpudecmpeg4-0> VPU has no more data to decode
0:00:05.909604772 27003 0xb055d4c0 LOG                imxvpudec gstimxvpudec.c:554:gst_imx_vpu_dec_handle_frame:<imxvpudecmpeg4-0> pushing frame with 16504 bytes at virtual address 0xb1e229d8 into VPU (system frame number: 1)
0:00:05.909811788 27003 0xb055d4c0 ERROR              imxvpuapi imxvpuapi2_imx6_coda.c:1952:imx_vpu_api_dec_push_encoded_frame: tried to push an encoded frame before a previous one was decoded
0:00:05.910132478 27003 0xb055d4c0 ERROR              imxvpudec gstimxvpudec.c:562:gst_imx_vpu_dec_handle_frame:<imxvpudecmpeg4-0> could not push input frame data into decoder: invalid call
0:00:05.913750752 27003   0x71b048 WARN                avidemux gstavidemux.c:5785:gst_avi_demux_loop:<avidemux0> error: Internal data stream error.
0:00:05.913996437 27003   0x71b048 WARN                avidemux gstavidemux.c:5785:gst_avi_demux_loop:<avidemux0> error: streaming stopped, reason error (-5)

[dv1]

The very first frame is incomplete even when mpeg4videoparse is placed before imxvpudec_mpeg4. I suspect this worked by coincidence with v1 because it does not have something like the staged_encoded_frame_set checks in v2. I think v1 implicitly discarded the first staged encoded frame information, overwriting it with that of the second encoded frame later on.

This change "fixes" it (as in, prevents the errors), but results in the data loss / overwrite mentioned above:

diff --git a/imxvpuapi2/imxvpuapi2_imx6_coda.c b/imxvpuapi2/imxvpuapi2_imx6_coda.c
index 5586121..b71fd60 100644
--- a/imxvpuapi2/imxvpuapi2_imx6_coda.c
+++ b/imxvpuapi2/imxvpuapi2_imx6_coda.c
@@ -2336,6 +2336,7 @@ ImxVpuApiDecReturnCodes imx_vpu_api_dec_decode(ImxVpuApiDecoder *decoder, ImxVpu
                {
                        IMX_VPU_API_DEBUG("not enough input data was available");
                        *output_code = IMX_VPU_API_DEC_OUTPUT_CODE_MORE_INPUT_DATA_NEEDED;
+                       decoder->staged_encoded_frame_set = FALSE;
                }

                /* Report dropped frames. */

A proper fix would require implementing an internal queue of staged encoded frames. That would be a substantial change though. It would perhaps be better instead to check if this is a bug in mpeg4videoparse.

[dougnazar]

The reason I was assuming it was an IMX issue was the video played fine with v1, avdec_mpeg4, & various media players. It wasn't a single file, if I remember right it was at least several hundred so I needed to find a quick solution.

I'll take a look at mpeg4videoparse but that'll be a low priority item since I'm at least currently stable with the work around.

Thanks

[dougnazar]

Fixed by f5130e0c802562fa986626caedf7c9de5e6b912b