AV1 videos on bilibili not displayed well

[amazingfate]

I'm working on adding av1 hardware decode to chromium browser, but I encountered bad performance when playing videos on bilibili. Here is my screen capture: https://www.bilibili.com/video/BV13Y4y1f7R7/. I thought that was a chromium related issue JeffyCN/libv4l-rkmpp#6, but after I downloaded the av1 video from bilibili and played it with gst-play-1.0, I also got bad display. I'm using kernel from tag linux-5.10-gen-rkr3.6 and the latest mpp from this repo. I think this issue should also be easily reproduced from your internal repo. Here is the video sample:

https://user-images.githubusercontent.com/5022901/215238716-4b4a64f5-257a-4f56-b424-fb7eb5793e8d.mp4

AV1 videos on youtube are played well. Maybe bilibili has done something different on video encoding.

[amazingfate]

I uploaded this video to an online video converter to re-encode it. Now it is played well:

https://user-images.githubusercontent.com/5022901/215414530-e804f197-0fd1-4f97-ab45-8d1430902244.mp4

[FumasterLin]

0002-av1d_parser-fix-parse-err-with-unknown-metadata.patch 0001-av1d_parser-fix-split-frame-issue.patch

Please apply the patches and test the issue video again.

[amazingfate]

@FumasterLin video is still not played well. Here is the mpp log when playing: bad.log

[FumasterLin]

@amazingfate Please input cmd：export av1d_debug=0xf, and record the log when playing again.

[amazingfate]

@FumasterLin here is the new log: av1_debug_patched.log

[FumasterLin]

The video seems to be different from the one you uploaded before. Can you upload the test source?

[amazingfate]

I'm testing with the first video uploaded in this issue bilibili-av1-test.mp4. I redownloaded that video and checked the md5 values of it, which has the same value with my local video.

jfliu@rock-5b:~/test$ md5sum ../bilibili-av1-test.mp4
4ba6dc58e3b09f0b96c96ba3470da0c2  ../bilibili-av1-test.mp4
jfliu@rock-5b:~/test$ md5sum 215238716-4b4a64f5-257a-4f56-b424-fb7eb5793e8d.mp4
4ba6dc58e3b09f0b96c96ba3470da0c2  215238716-4b4a64f5-257a-4f56-b424-fb7eb5793e8d.mp4
jfliu@rock-5b:~/test$ ls -lh ../bilibili-av1-test.mp4
-rw-rw-r-- 1 jfliu jfliu 990K  1月 19 18:55 ../bilibili-av1-test.mp4
jfliu@rock-5b:~/test$ ls -lh 215238716-4b4a64f5-257a-4f56-b424-fb7eb5793e8d.mp4
-rw-rw-r-- 1 jfliu jfliu 990K  1月 28 10:59 215238716-4b4a64f5-257a-4f56-b424-fb7eb5793e8d.mp4

[amazingfate]

@FumasterLin Although these two files are the same by md5, I did get different log compared with the one downloaded from github, so I compress this local video and wish github would not make magic change to it. bilibili-av1-test.zip

[FumasterLin]

Can you record a video of the issue video as it plays and upload?

[amazingfate]

I found that when the video is first played, logs are much longer than the second time. av1_github_debug_orig.log is the first time play log of the issue video. av1_github_debug_orig2.log is the second time log.

[amazingfate]

@FumasterLin here is the screen record:

https://user-images.githubusercontent.com/5022901/215920858-b04d391a-532e-420c-8886-ea5b11378b7d.mp4

[amazingfate]

@FumasterLin one guess: all the videos with this issue are variable frame rate encoded, and all of them are from bilibili.

[FumasterLin]

The video you provided is playing fine on my device. check the version of the mpp so with cmd: strings librockchip_mpp.so | grep author.

[amazingfate]

@FumasterLin

strings /usr/lib/aarch64-linux-gnu/librockchip_mpp.so.0| grep author
a1cdb03 author: Yandong Lin    2023-01-30 fix parse err with unknown metadata  (HEAD)
3ee748d author: Yandong Lin    2023-01-30 fix split frame issue
d63affb author: Grey Li        2022-10-20 [jpegd]: fix overflow in copy mode  (grafted, tag: linux-5.10-gen-rkr3.5, tag: linux-5.10-gen-rkr3.4)
a1cdb03 author: Yandong Lin    2023-01-30 fix parse err with unknown metadata

[amazingfate]

@FumasterLin I also tested the latest develop branch mpp, same issue:

strings /usr/lib/aarch64-linux-gnu/librockchip_mpp.so.0| grep author
eca8325 author: Yandong Lin    2023-02-01 fix parse err with unknown metadata  (HEAD -> develop)
3886846 author: Yandong Lin    2023-02-01 fix split frame issue
93b1cd1 author: He Hua         2022-12-26 [vpu_api]: modify struct VideoFrame  (grafted, origin/develop, origin/HEAD)
eca8325 author: Yandong Lin    2023-02-01 fix parse err with unknown metadata

If you can't reporduce it, I think this issue may be kernel related. The lastest kernel I can get is tag:linux-5.10-gen-rkr3.6, and the latest commit of drivers/video/rockchip/mpp is 42c7305fbcbc7aced4ce1fc9aba1130ce28fcf66: video: rockchip: mpp: rkvdec: add task timing & timeout for rkvdec

[FumasterLin]

The mpp version on your device seems fine. The system on my device is android not linux, i will update a linux system to check again.

[FumasterLin]

I reproduce the issue with gst. The rootcause: the issue video has an unknown meta type obu, and gst parsing error causes the video data not sent to mpp for decoding, so it leads to the subsequent frames decoding error.

[amazingfate]

I reproduce the issue with gst. The rootcause: the issue video has an unknown meta type obu, and gst parsing error causes the video data not sent to mpp for decoding, so it leads to the subsequent frames decoding error.

GST software decoding does not work either. But chromium with software dav1d decoder can play it well. Chromium should has done something to aviod this issue.

[FumasterLin]

It also can play well in android system with mpp decode. I'm not familiar with the processing in gst, I only know that gst will drop a whole buffer when parsing errors, and this buffer happens to have frame data in it.

0:00:05.627850732  1223   0x7fb42a4060 WARN                av1parse gstav1parse.c:1726:gst_av1_parse_handle_to_big_align:<av1parse0> Parse obu error, discard whole buffer 104355.
0:00:10.636505319  1223   0x7fb42a4060 WARN                av1parse gstav1parse.c:1726:gst_av1_parse_handle_to_big_align:<av1parse0> Parse obu error, discard whole buffer 105716.
0:00:15.628383487  1223   0x7fb42a4060 WARN                av1parse gstav1parse.c:1726:gst_av1_parse_handle_to_big_align:<av1parse0> Parse obu error, discard whole buffer 109029.

[amazingfate]

Gstreamer and ffmpeg use libdav1d which just drop the whole data when there is unknow metadata_type: https://code.videolan.org/videolan/dav1d/-/blob/master/src/obu.c#L1533 While chromium handles it as av1 spec: https://chromium.googlesource.com/codecs/libgav1/+/refs/heads/main/src/obu_parser.cc#2472

[amazingfate]

@FumasterLin gstreamer developer gives a patch: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3898/diffs. With this patch gstreamer can play the issue video well via software decoding. But when using mpp hardware decoding it is played the same as chromium. Here is the screenrecord:

https://user-images.githubusercontent.com/5022901/217022527-fc6af945-ed4e-4d92-81ef-a454a0e9721f.mp4

[FumasterLin]

@JeffyCN Could you help to anlyze this issue about gst?

[JeffyCN]

don't know much about it.

if mpp still get the wrong data, please contact gst upstream.

[amazingfate]

don't know much about it.

if mpp still get the wrong data, please contact gst upstream.

This is not a gstreamer only issue, chromium and ffmpeg both have this issue. I think this video has somthing special. But I don't know how to contact developers of bilibili.

[JeffyCN]

This is not a gstreamer only issue, chromium and ffmpeg both have this issue. I think this video has somthing special. But I don't know how to contact developers of bilibili.

i know that, you can contact them as well.

anyway, if mpp gets the correct data, let's check mpp. otherwise, check the upstream(gst chromium ffmpeg), or at least find a way to workaround

[JeffyCN]

by the way, your test video works well on my 3588 evb with: 1/ newest internal mpp(dd1be23a [h264d_api] add compatible for avcc fmt judgement) 2/ gst1-plugins-bad-1.20.3 and replace the related parser code with upstream(aeb262a7e1 pad: Don't leak user_data in gst_pad_start_task) gst-libs/gst/codecparsers/gstav1parser.c gst-libs/gst/codecparsers/gstav1parser.h gst/videoparsers/gstav1parse.c 3/ remove gst_av1_parser_reference_frame_loading() in gst-libs/gst/codecs/gstav1decoder.c to workaround compile error

[amazingfate]

gst-libs/gst/codecparsers/gstav1parser.c

This is the latest mpp I can get: 6ac409a author: Johnson Ding 2023-01-11 [vepu580_hevc]: optimize intra setting. With backported gstreamer 1.20.3 I got this error:

av1d_cbs: mpp_av1_read_fragment_content Failed to read unit 0 (type 5).
av1d_cbs: Assertion obu->header.obu_type == unit->type failed at mpp_av1_read_unit:2760
av1d_cbs: mpp_av1_uncompressed_header No sequence header available: unable to decode frame header.

[JeffyCN]

This is the latest mpp I can get: 6ac409a author: Johnson Ding 2023-01-11 [vepu580_hevc]: optimize intra setting.

i've updated my mirror repo just now, please check it for newest mpp

[amazingfate]

New code works, thank you!