Top portion of frame distorted

[mikeDonahoe]

Working on Accelerated playback in Chromium on the iMX6 and I'm noticing a strange issue (see video). Looks to only appear on videos that are in Portrait mode. Initially, I had been using v1 of the API and never observed this issue.

https://youtu.be/DF8BmeSCNGw

One thing I noticed is the actual_frame_width/height always seems to match aligned_frame_width/height.

[6031:6048:0814/200253.282748:VERBOSE1:imxvpu_video_decode_accelerator.cc(480)] Associating picture buffer 0/17 ID 0 with framebuffer #0 virtual address 0x52d09000 physical address 0x76480000 actual width 640 actual height 368 aligned width 640 aligned height 368 texture id 23 [6031:6048:0814/200253.282843:VERBOSE1:imxvpu_video_decode_accelerator.cc(1021)] Adding output buffer 0 to queue

Video that glitches [6314:6332:0814/200550.319460:VERBOSE1:imxvpu_video_decode_accelerator.cc(480)] Associating picture buffer 2/17 ID 2 with framebuffer #2 virtual address 0x57003000 physical address 0x76300000 actual width 368 actual height 368 aligned width 368 aligned height 368 texture id 25

I ended up adding code that dumps the raw frame right after a IMX_VPU_API_DEC_OUTPUT_CODE_DECODED_FRAME_AVAILABLE and I see that the image is corrupted.

-Thanks

[mikeDonahoe]

For clarity, the actual frames are NV12. I inadvertently selected NV21 on rawpixels.

[mikeDonahoe]

Looking at this again. I added code to dump the raw H264 stream from Chromium, I then decoded the stream using the provided decode-example and the frames suffer the same issue. To verify that the encoded stream was not corrupted I then played the raw H264 dump via mplayer and the frames are displayed correctly.

Video shows VPU decode on the left and mplayer on the right. https://youtu.be/858mbNwGud4

At the very least I can now use the decode-example to debug so that helps. Here is the raw file. https://www.dropbox.com/s/0ysalh4t03cxps6/rawH264?dl=0

sample of the output from decode-example global decoder information: semi planar frames supported: 1 fully planar frames supported: 1 decoded frames are from buffer pool: 0 min required stream buffer size: 5236736 required stream buffer physaddr alignment: 512 required stream buffer size alignment: 1024 num supported compression formats: 8 MPEG-2 part 2 MPEG-4 part 2 h.263 h.264 / AVC WMV3 / Windows Media Video 9 VC-1 / Windows Media Video 9 Advanced Profile JPEG VP8 ../git/imxvpuapi2/imxvpuapi2_imx6_coda_ipu.c:50 (imx_vpu_api_imx6_coda_open_ipu_voda_fd) trace: opened IPU VDOA file descriptor 6 ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:1631 (imx_vpu_api_dec_open) debug: opening decoder, frame size: 0 x 0 pixel ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:230 (imx_coda_vpu_load) log: VPU init instance counter: 0 ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:239 (imx_coda_vpu_load) info: libimxvpuapi version 2.0.1 vpulib backend ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:243 (imx_coda_vpu_load) debug: loaded VPU ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:1645 (imx_vpu_api_dec_open) debug: successfully opened decoder pushing encoded frame with context 0x1000 and 1605632 byte into encoder ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:985 (imx_vpu_api_dec_push_input_data) log: bitstream buffer status: read ptr 0x75800000 write ptr 0x75800000 num free bytes 3143679 ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:1975 (imx_vpu_api_dec_push_encoded_frame) debug: staged encoded frame draining decoder ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:1878 (imx_vpu_api_dec_enable_drain_mode) debug: enabled decoder drain mode ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:2220 (imx_vpu_api_dec_decode) log: waiting for decoding completion ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:2291 (imx_vpu_api_dec_decode) log: output info: indexFrameDisplay -3 indexFrameDecoded 0 NumDecFrameBuf 0 picTyp e 0 idrFlg 3 numOfErrMBs 0 hScaleFlag 0 vScaleFlag 0 notSufficientPsBuffer 0 notSufficientSliceBuffer 0 decodingSuccess 1 interlacedFrame 0 mp4PackedP Bframe 0 h264Npf 0 pictureStructure 0 topFieldFirst 0 repeatFirstField 0 fieldSequence 0 decPicWidth 368 decPicHeight 368 ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:2403 (imx_vpu_api_dec_decode) debug: staged frame reported as decoded; unstaging ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:2445 (imx_vpu_api_dec_decode) log: nothing yet to display ; indexFrameDisplay: -3 ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:2220 (imx_vpu_api_dec_decode) log: waiting for decoding completion ../git/imxvpuapi2/imxvpuapi2_imx6_coda.c:2291 (imx_vpu_api_dec_decode) log: output info: indexFrameDisplay -3 indexFrameDecoded 1 NumDecFrameBuf 0 picType 1 idrFlg 2 numOfErrMBs 0 hScaleFlag 0 vScaleFlag 0 notSufficientPsBuffer 0 notSufficientSliceBuffer 0 decodingSuccess 1 interlacedFrame 0 mp4PackedPBframe 0 h264Npf 0 pictureStructure 0 topFieldFirst 0 repeatFirstField 0 fieldSequence 0 decPicWidth 368 decPicHeight 368

Playing rawH264 via mplayer. libavformat version 56.40.101 (external) libavformat file format detected. [lavf] stream 0: video (h264), -vid 0 VIDEO: [H264] 360x360 0bpp 30.000 fps 0.0 kbps ( 0.0 kbyte/s) Load subtitles in ./ Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family libavcodec version 56.60.100 (external) Selected video codec: [ffh264] vfm: ffmpeg (FFmpeg H.264) Audio: no sound Starting playback... Movie-Aspect is 1.00:1 - prescaling to correct movie aspect. VO: [vdpau] 360x360 => 360x360 Planar YV12

[dv1]

For a long time I had no clue what could cause this. But lately I've seen something similar show up on another project that patched Chromium. It turns out to be a fencing issue: The VPU decodes into contiguous memory blocks that are still being read by the GPU because the media stack returned that memory block to the VPU too early. This is effectively a race condition, just in the GPU-VPU pair. The solution was to activate GPU fences to make sure the memory block is only returned when the GPU is really truly done drawing the frame. This is accomplished by enabling the read_lock_fences_enabled flag in the video frame metadata: https://chromium.googlesource.com/chromium/src/media/+/refs/heads/main/base/video_frame_metadata.h#110

If you use Picture instances, these have that flag as well: https://chromium.googlesource.com/chromium/src/media/+/refs/heads/main/video/picture.h#139

[mikeDonahoe]

Testing without Chromium produced the same results. I simply was taking the raw h264 from Chromium to a file. See the previous post where I explain even using the provided decode-example resulted in corrupt frames.

https://github.com/Freescale/libimxvpuapi/issues/41#issuecomment-678488588