search

JeffyCN / FFmpeg

FFmpeg with rkmpp hwdec - (deprecated due to license issue, check gstreamer instead or https://github.com/nyanmisaka/ffmpeg-rockchip)
https://ffmpeg.org
Other
60 stars 18 forks source link

FFmpeg on rk3568 #1

Closed avafinger closed 2 years ago

avafinger commented 2 years ago

Hi @JeffyCN

I have built your FFmpeg on Rock3a (rk3568), disabled x264,x265,vp8,vp9, but seems not to work with h264_rkmpp. The odd thing, even with x264 disabled I still have h264 working, I think via software. Can you spot what is wrong?

sudo ffmpeg -benchmark -vcodec h264_rkmpp -i Big_Buck_Bunny_1080_10s_30MB.mp4 -map 0:v:0 -f null -
ffmpeg version N-94777-g067b47f66c Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --prefix=/usr --enable-nonfree --enable-gpl --enable-version3 --disable-libx264 --disable-libx265 --enable-libmp3lame --enable-libpulse --enable-libv4l2 --enable-libdrm --enable-rkmpp --enable-libxml2 --enable-librtmp --disable-libxvid --enable-libfreetype --enable-openssl --enable-opengl --enable-libopus --enable-libvorbis --enable-shared
  libavutil      56. 34.100 / 56. 34.100
  libavcodec     58. 56.101 / 58. 56.101
  libavformat    58. 32.104 / 58. 32.104
  libavdevice    58.  9.100 / 58.  9.100
  libavfilter     7. 58.102 /  7. 58.102
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Big_Buck_Bunny_1080_10s_30MB.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    composer        : Sacha Goedegebure
    encoder         : Lavf57.63.100
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    genre           : Animation
  Duration: 00:00:10.00, start: 0.000000, bitrate: 24563 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 24559 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
    Metadata:
      handler_name    : VideoHandler
mpp[104077]: mpp_info: mpp version: 6dadc7e1 author: Ding Wei      2021-09-02 [hal_h265d]: Use MPP_OK default, avoid the api->control is NULL
mpp[104077]: mpp_rt: NOT found ion allocator
mpp[104077]: mpp_rt: found drm allocator
mpp[104077]: hal_h264d_vdpu34x: control info: fmt 7, w 1920, h 1080
mpp[104077]: mpp_buf_slot: set frame info: w 1920 h 1080 hor 1920 ver 1088
mpp[104077]: mpp_dec: setting default w 1920 h 1080 h_str 1920 v_str 1088
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_rkmpp) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[h264_rkmpp @ 0x55b8964da0] Decoder noticed an info change (1920x1080), format=0
[h264_rkmpp @ 0x55b8964da0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
[h264_rkmpp @ 0x55b8964da0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
Output #0, null, to 'pipe:':

Output using h264 (400% cpu):

sudo ffmpeg -benchmark -vcodec h264 -i Big_Buck_Bunny_1080_10s_30MB.mp4 -map 0:v:0 -f null -
ffmpeg version N-94777-g067b47f66c Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --prefix=/usr --enable-nonfree --enable-gpl --enable-version3 --disable-libx264 --disable-libx265 --enable-libmp3lame --enable-libpulse --enable-libv4l2 --enable-libdrm --enable-rkmpp --enable-libxml2 --enable-librtmp --disable-libxvid --enable-libfreetype --enable-openssl --enable-opengl --enable-libopus --enable-libvorbis --enable-shared
  libavutil      56. 34.100 / 56. 34.100
  libavcodec     58. 56.101 / 58. 56.101
  libavformat    58. 32.104 / 58. 32.104
  libavdevice    58.  9.100 / 58.  9.100
  libavfilter     7. 58.102 /  7. 58.102
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Big_Buck_Bunny_1080_10s_30MB.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    composer        : Sacha Goedegebure
    encoder         : Lavf57.63.100
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    genre           : Animation
  Duration: 00:00:10.00, start: 0.000000, bitrate: 24563 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 24559 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    composer        : Sacha Goedegebure
    genre           : Animation
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    encoder         : Lavf58.32.104
    Stream #0:0(und): Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc58.56.101 wrapped_avframe
frame=  300 fps= 35 q=-0.0 Lsize=N/A time=00:00:10.00 bitrate=N/A speed=1.18x    
video:157kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
bench: utime=30.304s stime=0.572s rtime=8.507s
bench: maxrss=138956kB
JeffyCN commented 2 years ago

1/ it was tested in bsp sdk(with chromium/firefox/mpv(custom)/ffplay), and it needs librga to do hardware conversion if not using drm_prime frame 2/ check here: https://github.com/JeffyCN/FFmpeg/blob/debian/4.1.4-1/debian/rules#L180

avafinger commented 2 years ago

I enabled librga and disabled all you mentioned, still the same. I was expecting to receive a drm_prime frame, but looks like not (Decoder noticed an info change (1920x1080), format=0)?

I still have H264_DECODER 1 , it should be 0?

Output:

External libraries:
iconv                   libpulse                libxml2
libdrm                  librtmp                 opengl
libfreetype             libv4l2                 openssl
libmp3lame              libvorbis               rkmpp
libopus                 libxcb                  zlib

External libraries providing hardware acceleration:
librga                  v4l2_m2m
#define CONFIG_V4L2_M2M 1
#define CONFIG_H263_V4L2M2M_DECODER 1
#define CONFIG_H264_V4L2M2M_DECODER 0
#define CONFIG_HEVC_V4L2M2M_DECODER 1
#define CONFIG_MPEG4_V4L2M2M_DECODER 0
#define CONFIG_MPEG1_V4L2M2M_DECODER 1
#define CONFIG_MPEG2_V4L2M2M_DECODER 0
#define CONFIG_VC1_V4L2M2M_DECODER 1
#define CONFIG_VP8_V4L2M2M_DECODER 0
#define CONFIG_VP9_V4L2M2M_DECODER 1
#define CONFIG_H263_V4L2M2M_ENCODER 1
#define CONFIG_H264_V4L2M2M_ENCODER 1
#define CONFIG_HEVC_V4L2M2M_ENCODER 1
#define CONFIG_MPEG4_V4L2M2M_ENCODER 1
#define CONFIG_VP8_V4L2M2M_ENCODER 1

Is there a simple way i could test if mpp and librga is working?

I am still confused about the h264 (native), can you share the config.h of your test?

Enabled decoders:
adpcm_ea_r3             h264                    ra_288
adpcm_ea_xas            h264_rkmpp              ralf
adpcm_g726le            hevc                    realtext
adpcm_ima_amv           hevc_rkmpp              rl2
adpcm_ima_apc           hevc_v4l2m2m            roq
bink                    mpeg1_v4l2m2m           txd
avafinger commented 2 years ago

@JeffyCN

Just in time, i think i am using the BSP kernel, kernel 4.19.193.

avafinger commented 2 years ago

In theory mpp is working:

sudo mpi_enc_test -f 4 -w 1280 -h 720 -t 8 -i sample.yuv -o test.jpg -n 1 -d 1
[sudo] password for rock: 
mpp[42676]: mpi_enc_utils: skip invalid opt d
mpp[42676]: mpi_enc_utils: cmd parse result:
mpp[42676]: mpi_enc_utils: input  file name: sample.yuv
mpp[42676]: mpi_enc_utils: output file name: test.jpg
mpp[42676]: mpi_enc_utils: width      : 1280
mpp[42676]: mpi_enc_utils: height     : 720
mpp[42676]: mpi_enc_utils: format     : 4
mpp[42676]: mpi_enc_utils: type       : 8
mpp[42676]: mpi_enc_test: mpi_enc_test start
mpp[42676]: mpp_rt: NOT found ion allocator
mpp[42676]: mpp_rt: found drm allocator
mpp[42676]: mpp_info: mpp version: 6dadc7e1 author: Ding Wei      2021-09-02 [hal_h265d]: Use MPP_OK default, avoid the api->control is NULL
mpp[42676]: mpi_enc_test: 0x5574e248a0 mpi_enc_test encoder test start w 1280 h 720 type 8
mpp[42676]: mpp_enc: MPP_ENC_SET_RC_CFG bps 3456000 [216000 : 3672000] fps [30:30] gop 60
mpp[42676]: mpi_enc_test: 0x5574e248a0 encoded frame 0    size 49711  
mpp[42676]: mpi_enc_test: 0x5574e248a0 encode max 1 frames
mpp[42676]: mpi_enc_test: 0x5574e248a0 mpi_enc_test success total frame 1 bps 11930640

H264

sudo mpi_enc_test -w 1280 -h 720 -f 4 -t 7 -i sample.yuv  -o out.h264 -n 10
mpp[45423]: mpi_enc_utils: cmd parse result:
mpp[45423]: mpi_enc_utils: input  file name: sample.yuv
mpp[45423]: mpi_enc_utils: output file name: out.h264
mpp[45423]: mpi_enc_utils: width      : 1280
mpp[45423]: mpi_enc_utils: height     : 720
mpp[45423]: mpi_enc_utils: format     : 4
mpp[45423]: mpi_enc_utils: type       : 7
mpp[45423]: mpi_enc_test: mpi_enc_test start
mpp[45423]: mpp_rt: NOT found ion allocator
mpp[45423]: mpp_rt: found drm allocator
mpp[45423]: mpp_info: mpp version: 6dadc7e1 author: Ding Wei      2021-09-02 [hal_h265d]: Use MPP_OK default, avoid the api->control is NULL
mpp[45423]: mpi_enc_test: 0x559163e8a0 mpi_enc_test encoder test start w 1280 h 720 type 7
mpp[45423]: mpp_enc: MPP_ENC_SET_RC_CFG bps 3456000 [216000 : 3672000] fps [30:30] gop 60
mpp[45423]: h264e_api_v2: MPP_ENC_SET_PREP_CFG w:h [1280:720] stride [1280:720]
mpp[45423]: mpp_enc: send header for set cfg change input/format 
mpp[45423]: mpp_enc: mode vbr bps [216000:3456000:3672000] fps fix [30/1] -> fix [30/1] gop i [60] v [0]
mpp[45423]: mpi_enc_test: 0x559163e8a0 encoded frame 0    size 26590   qp 23
mpp[45423]: mpi_enc_test: 0x559163e8a0 loop times 1
mpp[45423]: mpi_enc_test: 0x559163e8a0 encoded frame 1    size 42      qp 26
mpp[45423]: mpi_enc_test: 0x559163e8a0 loop times 2
mpp[45423]: mpi_enc_test: 0x559163e8a0 encoded frame 2    size 39      qp 26
mpp[45423]: mpi_enc_test: 0x559163e8a0 loop times 3
mpp[45423]: mpi_enc_test: 0x559163e8a0 encoded frame 3    size 43      qp 25
mpp[45423]: mpi_enc_test: 0x559163e8a0 loop times 4
mpp[45423]: mpi_enc_test: 0x559163e8a0 encoded frame 4    size 39      qp 25
mpp[45423]: mpi_enc_test: 0x559163e8a0 loop times 5
mpp[45423]: mpi_enc_test: 0x559163e8a0 encoded frame 5    size 44      qp 24
mpp[45423]: mpi_enc_test: 0x559163e8a0 loop times 6
mpp[45423]: mpi_enc_test: 0x559163e8a0 encoded frame 6    size 38      qp 24
mpp[45423]: mpi_enc_test: 0x559163e8a0 loop times 7
mpp[45423]: mpi_enc_test: 0x559163e8a0 encoded frame 7    size 38      qp 24
mpp[45423]: mpi_enc_test: 0x559163e8a0 loop times 8
mpp[45423]: mpi_enc_test: 0x559163e8a0 encoded frame 8    size 38      qp 24
mpp[45423]: mpi_enc_test: 0x559163e8a0 loop times 9
mpp[45423]: mpi_enc_test: 0x559163e8a0 encoded frame 9    size 58      qp 23
mpp[45423]: mpi_enc_test: 0x559163e8a0 encode max 10 frames
mpp[45423]: mpi_enc_test: 0x559163e8a0 mpi_enc_test success total frame 10 bps 647256
avafinger commented 2 years ago

I have rga node and not rga2, is this correct for rk3568?

rock@rock3a:~/kernel$ ls /dev/rga*
/dev/rga
rock@rock3a:~/kernel$ cat .config |grep RGA
CONFIG_VIDEO_ROCKCHIP_RGA=y
# RGA
# CONFIG_ROCKCHIP_RGA is not set
# RGA2
CONFIG_ROCKCHIP_RGA2=y
avafinger commented 2 years ago

Hi @JeffyCN ,

MPP decoding is also working:

sudo mpi_dec_test -t 7 -i freeway_720p.h264 
mpp[7358]: mpi_dec_utils: cmd parse result:
mpp[7358]: mpi_dec_utils: input  file name: freeway_720p.h264
mpp[7358]: mpi_dec_utils: output file name: 
mpp[7358]: mpi_dec_utils: config file name: 
mpp[7358]: mpi_dec_utils: width      :    0
mpp[7358]: mpi_dec_utils: height     :    0
mpp[7358]: mpi_dec_utils: type       : 7
mpp[7358]: mpi_dec_utils: debug flag : 0
mpp[7358]: mpi_dec_utils: max frames : 0
mpp[7358]: mpi_dec_test: mpi_dec_test start
mpp[7358]: mpi_dec_test: input file size 4510750
mpp[7358]: mpp_info: mpp version: 6dadc7e1 author: Ding Wei      2021-09-02 [hal_h265d]: Use MPP_OK default, avoid the api->control is NULL
mpp[7358]: mpi_dec_test: 0x558adfd960 mpi_dec_test decoder test start w 0 h 0 type 7
mpp[7358]: mpp_rt: NOT found ion allocator
mpp[7358]: mpp_rt: found drm allocator
mpp[7358]: mpi_dec_test: 0x558adfd960 decode_get_frame get info changed found
mpp[7358]: mpi_dec_test: 0x558adfd960 decoder require buffer w:h [1280:720] stride [1280:720] buf_size 1843200
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 0
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 1
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 2
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 3
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 4
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 5
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 6
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 7
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 8
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 9
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 10
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 11
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 12
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 13
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 14
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 15
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 16
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 17
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 18
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 19
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 20
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 21
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 22
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 23
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 24
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 25
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 26
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 27
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 28
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 29
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 30
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 31
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 32
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 33
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 34
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 35
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 36
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 37
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 38
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 39
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 40
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 41
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 42
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 43
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 44
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 45
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 46
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 47
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 48
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 49
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 50
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 51
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 52
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 53
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 54
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 55
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 56
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 57
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 58
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 59
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 60
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 61
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 62
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 63
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 64
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 65
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 66
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 67
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 68
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 69
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 70
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 71
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 72
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 73
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 74
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 75
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 76
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 77
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 78
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 79
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 80
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 81
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 82
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 83
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 84
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 85
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 86
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 87
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 88
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 89
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 90
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 91
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 92
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 93
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 94
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 95
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 96
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 97
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 98
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 99
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 100
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 101
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 102
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 103
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 104
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 105
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 106
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 107
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 108
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 109
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 110
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 111
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 112
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 113
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 114
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 115
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 116
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 117
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 118
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 119
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 120
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 121
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 122
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 123
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 124
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 125
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 126
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 127
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 128
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 129
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 130
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 131
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 132
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 133
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 134
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 135
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 136
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 137
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 138
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 139
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 140
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 141
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 142
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 143
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 144
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 145
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 146
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 147
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 148
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 149
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 150
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 151
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 152
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 153
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 154
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 155
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 156
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 157
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 158
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 159
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 160
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 161
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 162
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 163
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 164
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 165
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 166
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 167
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 168
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 169
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 170
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 171
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 172
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 173
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 174
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 175
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 176
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 177
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 178
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 179
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 180
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 181
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 182
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 183
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 184
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 185
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 186
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 187
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 188
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 189
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 190
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 191
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 192
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 193
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 194
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 195
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 196
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 197
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 198
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 199
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 200
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 201
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 202
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 203
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 204
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 205
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 206
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 207
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 208
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 209
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 210
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 211
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 212
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 213
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 214
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 215
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 216
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 217
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 218
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 219
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 220
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 221
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 222
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 223
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 224
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 225
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 226
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 227
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 228
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 229
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 230
mpp[7358]: mpi_dec_test: 0x558adfd960 found last packet
mpp[7358]: mpi_dec_test: 0x558adfd960 decode get frame 231
mpp[7358]: mpi_dec_test: 0x558adfd960 found last packet
mpp[7358]: mpi_dec_test: 0x558adfd960 input 1102 pkt output 233 frm decode 232 frames
mpp[7358]: mpi_dec_test: test success max memory 5.27 MB

Mali Bifrost is also working (GBM), by the way, is this necessary to get DRM_PRIME frame? I am compiling ffmpeg with a mali bifrost set up successfully, just in case.

How can I test the librga? do you have any C code I could try to verify librga? I tried you drm-rga-demo but no idea what is FBPL, don't know how to use it.

Summing up the issues (questions):

  1. mpp frame Is not DRM_PRIME
  2. RGA is then used but failed (not sure if is librga or ffmpeg)

In my setup, it is working:

  1. Mpp encoding
  2. Mpp decoding
  3. Mali Bifrost (GBM)

MPP build is from Rockchip, not the one from HermanChen.

JeffyCN commented 2 years ago

to use drm_prime, you may need to provide custom get_format callback(check mpv for example). and drm_prime needs special rendering, i have a custom mpv repo for it(render drm_prime nv12 with drm/custom x11-xv)

rga2 is correct. to test rga(2), you can use ffplay or chromium/firefox.(make sure the /dev/rga is accessable, chmod 666 /dev/rga)

so please try mpv and ffplay

avafinger commented 2 years ago

@JeffyCN

FFmpeg failed here:

[h264_rkmpp @ 0x557ef60da0] RGA init OK
[h264_rkmpp @ 0x557ef60da0] RGA pitch error
[h264_rkmpp @ 0x557ef60da0] Doing slow software conversion

  if (u_pitch != y_pitch / 2 || v_pitch != y_pitch / 2 ||
        dst_u != dst_y + y_pitch * dst_height ||
        dst_v != dst_u + u_pitch * dst_height / 2) {
        av_log(avctx, AV_LOG_WARNING, "RGA pitch error\n");
        goto bail;
    }

I have seen somewhere that the buffer is adjusted to 1920x1088 (chromium buffer align). Please, X11 or mpv is too complex / bloated, I would like to be able to run this, can you advise or propose a fix?

JeffyCN commented 2 years ago

the coded size is correct: https://github.com/JeffyCN/FFmpeg/commit/f06fbedb6b86ac4fc5902926edf33ef51a650190

if you are testing with chromium, the 1920x1080 should pass this check... please add more log to see which pitch is wrong.

the rga requires contig buffer, so there's also a hack for firefox(replace the default non-contig one): https://github.com/JeffyCN/FFmpeg/commit/72ee109dcc79b560724b50e60d3a8e4ed5853242 so maybe you can force re-alloc buffer like this too

avafinger commented 2 years ago

I am on Ubuntu, CLI. No chromium.

I have the two patches applied, maybe f06fbed is could be the problem? I am on the master branch.

avafinger commented 2 years ago

Does this info helps? [h264_rkmpp @ 0x55741adda0] dst_u: 0x5574603d40 - dst_v: 0x5574683930 [h264_rkmpp @ 0x55741adda0] dst_y + y_pitch dst_height: 0x5574603e30 - dst_u + u_pitch dst_height / 2: 0x5574500d80

avafinger commented 2 years ago

@JeffyCN

[h264_rkmpp @ 0x5585ebeda0] dst_u: 0x5586621550 - dst_v: 0x55866a1140 [h264_rkmpp @ 0x5585ebeda0] dst_y: 0x5586422620 - y_pitch: 1920 - dst_height: 1090 [h264_rkmpp @ 0x5585ebeda0] dst_u: 0x5586621550 - u_pitch: 960 - dst_height / 2: 545

Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

avafinger commented 2 years ago

@JeffyCN

Sorry, here is better info:

[h264_rkmpp @ 0x559685fda0] dst_y = 0x55969b7630
[h264_rkmpp @ 0x559685fda0] dst_u = 0x5596bb6560
[h264_rkmpp @ 0x559685fda0] dst_v = 0x5596c36150
[h264_rkmpp @ 0x559685fda0] width = 1920
[h264_rkmpp @ 0x559685fda0] height = 1080
[h264_rkmpp @ 0x559685fda0] hstride = 1920
[h264_rkmpp @ 0x559685fda0] vstride = 1088
[h264_rkmpp @ 0x559685fda0] y_pitch = 1920
[h264_rkmpp @ 0x559685fda0] u_pitch = 960
[h264_rkmpp @ 0x559685fda0] v_pitch = 960
avafinger commented 2 years ago

I think the dst_height is also wrong:

    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc58.56.101 wrapped_avframe
[null @ 0x5594a8a7b0] Encoder did not produce proper pts, making some up.
[h264_rkmpp @ 0x5594aafda0] dst_y = 0x5595311d30
[h264_rkmpp @ 0x5594aafda0] dst_u = 0x5594f05d40
[h264_rkmpp @ 0x5594aafda0] dst_v = 0x5594f85930
[h264_rkmpp @ 0x5594aafda0] width = 1920
[h264_rkmpp @ 0x5594aafda0] height = 1080
[h264_rkmpp @ 0x5594aafda0] hstride = 1920
[h264_rkmpp @ 0x5594aafda0] vstride = 1088
[h264_rkmpp @ 0x5594aafda0] y_pitch = 1920
[h264_rkmpp @ 0x5594aafda0] u_pitch = 960
[h264_rkmpp @ 0x5594aafda0] v_pitch = 960
[h264_rkmpp @ 0x5594aafda0] dst_height = -2210
[h264_rkmpp @ 0x5594aafda0] dst_height / 2 = -1105
[h264_rkmpp @ 0x5594aafda0] RGA pitch error2
[h264_rkmpp @ 0x5594aafda0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
frame=    1 fps=0.0 q=-0.0 size=N/A time=00:00:00.03 bitrate=N/A speed=0.0494x  [h264_rkmpp @ 0x5594aafda0] dst_y = 0x5595013620
[h264_rkmpp @ 0x5594aafda0] dst_u = 0x5595212550
[h264_rkmpp @ 0x5594aafda0] dst_v = 0x5595292140
[h264_rkmpp @ 0x5594aafda0] width = 1920
[h264_rkmpp @ 0x5594aafda0] height = 1080
[h264_rkmpp @ 0x5594aafda0] hstride = 1920
[h264_rkmpp @ 0x5594aafda0] vstride = 1088
[h264_rkmpp @ 0x5594aafda0] y_pitch = 1920
[h264_rkmpp @ 0x5594aafda0] u_pitch = 960
[h264_rkmpp @ 0x5594aafda0] v_pitch = 960
[h264_rkmpp @ 0x5594aafda0] dst_height = 1090
[h264_rkmpp @ 0x5594aafda0] dst_height / 2 = 545
[h264_rkmpp @ 0x5594aafda0] RGA pitch error2
[h264_rkmpp @ 0x5594aafda0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
[h264_rkmpp @ 0x5594aafda0] dst_y = 0x5594c07630
[h264_rkmpp @ 0x5594aafda0] dst_u = 0x5594e06560
[h264_rkmpp @ 0x5594aafda0] dst_v = 0x5594e86150
[h264_rkmpp @ 0x5594aafda0] width = 1920
[h264_rkmpp @ 0x5594aafda0] height = 1080
[h264_rkmpp @ 0x5594aafda0] hstride = 1920
[h264_rkmpp @ 0x5594aafda0] vstride = 1088
[h264_rkmpp @ 0x5594aafda0] y_pitch = 1920
[h264_rkmpp @ 0x5594aafda0] u_pitch = 960
[h264_rkmpp @ 0x5594aafda0] v_pitch = 960
[h264_rkmpp @ 0x5594aafda0] dst_height = 1090
[h264_rkmpp @ 0x5594aafda0] dst_height / 2 = 545
[h264_rkmpp @ 0x5594aafda0] RGA pitch error2
[h264_rkmpp @ 0x5594aafda0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
frame=    2 fps=1.5 q=-0.0 size=N/A time=00:00:00.10 bitrate=N/A speed=0.0774x  [h264_rkmpp @ 0x5594aafda0] dst_y = 0x5595013620
[h264_rkmpp @ 0x5594aafda0] dst_u = 0x5595212550
[h264_rkmpp @ 0x5594aafda0] dst_v = 0x5595292140
[h264_rkmpp @ 0x5594aafda0] width = 1920
[h264_rkmpp @ 0x5594aafda0] height = 1080
[h264_rkmpp @ 0x5594aafda0] hstride = 1920
[h264_rkmpp @ 0x5594aafda0] vstride = 1088
[h264_rkmpp @ 0x5594aafda0] y_pitch = 1920
[h264_rkmpp @ 0x5594aafda0] u_pitch = 960
[h264_rkmpp @ 0x5594aafda0] v_pitch = 960
[h264_rkmpp @ 0x5594aafda0] dst_height = 1090
[h264_rkmpp @ 0x5594aafda0] dst_height / 2 = 545
[h264_rkmpp @ 0x5594aafda0] RGA pitch error2
[h264_rkmpp @ 0x5594aafda0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
[h264_rkmpp @ 0x5594aafda0] dst_y = 0x5595311d30
[h264_rkmpp @ 0x5594aafda0] dst_u = 0x5594f05d40
[h264_rkmpp @ 0x5594aafda0] dst_v = 0x5594f85930
[h264_rkmpp @ 0x5594aafda0] width = 1920
[h264_rkmpp @ 0x5594aafda0] height = 1080
[h264_rkmpp @ 0x5594aafda0] hstride = 1920
[h264_rkmpp @ 0x5594aafda0] vstride = 1088
[h264_rkmpp @ 0x5594aafda0] y_pitch = 1920
[h264_rkmpp @ 0x5594aafda0] u_pitch = 960
[h264_rkmpp @ 0x5594aafda0] v_pitch = 960
[h264_rkmpp @ 0x5594aafda0] dst_height = -2210
[h264_rkmpp @ 0x5594aafda0] dst_height / 2 = -1105
[h264_rkmpp @ 0x5594aafda0] RGA pitch error2
[h264_rkmpp @ 0x5594aafda0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
frame=    3 fps=1.6 q=-0.0 size=N/A time=00:00:00.16 bitrate=N/A speed=0.0874x  [h264_rkmpp @ 0x5594aafda0] dst_y = 0x5595013620
[h264_rkmpp @ 0x5594aafda0] dst_u = 0x5595212550
[h264_rkmpp @ 0x5594aafda0] dst_v = 0x5595292140
[h264_rkmpp @ 0x5594aafda0] width = 1920
[h264_rkmpp @ 0x5594aafda0] height = 1080
[h264_rkmpp @ 0x5594aafda0] hstride = 1920
[h264_rkmpp @ 0x5594aafda0] vstride = 1088
[h264_rkmpp @ 0x5594aafda0] y_pitch = 1920
[h264_rkmpp @ 0x5594aafda0] u_pitch = 960
[h264_rkmpp @ 0x5594aafda0] v_pitch = 960
[h264_rkmpp @ 0x5594aafda0] dst_height = 1090
[h264_rkmpp @ 0x5594aafda0] dst_height / 2 = 545
[h264_rkmpp @ 0x5594aafda0] RGA pitch error2
[h264_rkmpp @ 0x5594aafda0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
[h264_rkmpp @ 0x5594aafda0] dst_y = 0x5594c07630
[h264_rkmpp @ 0x5594aafda0] dst_u = 0x5594e06560
[h264_rkmpp @ 0x5594aafda0] dst_v = 0x5594e86150
[h264_rkmpp @ 0x5594aafda0] width = 1920
[h264_rkmpp @ 0x5594aafda0] height = 1080
[h264_rkmpp @ 0x5594aafda0] hstride = 1920
[h264_rkmpp @ 0x5594aafda0] vstride = 1088
[h264_rkmpp @ 0x5594aafda0] y_pitch = 1920
[h264_rkmpp @ 0x5594aafda0] u_pitch = 960
[h264_rkmpp @ 0x5594aafda0] v_pitch = 960
[h264_rkmpp @ 0x5594aafda0] dst_height = 1090
[h264_rkmpp @ 0x5594aafda0] dst_height / 2 = 545
[h264_rkmpp @ 0x5594aafda0] RGA pitch error2
[h264_rkmpp @ 0x5594aafda0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
frame=    4 fps=1.6 q=-0.0 size=N/A time=00:00:00.23 bitrate=N/A speed=0.0925x  [h264_rkmpp @ 0x5594aafda0] dst_y = 0x5595013620
[h264_rkmpp @ 0x5594aafda0] dst_u = 0x5595212550
[h264_rkmpp @ 0x5594aafda0] dst_v = 0x5595292140
[h264_rkmpp @ 0x5594aafda0] width = 1920
[h264_rkmpp @ 0x5594aafda0] height = 1080
[h264_rkmpp @ 0x5594aafda0] hstride = 1920
[h264_rkmpp @ 0x5594aafda0] vstride = 1088
[h264_rkmpp @ 0x5594aafda0] y_pitch = 1920
[h264_rkmpp @ 0x5594aafda0] u_pitch = 960
[h264_rkmpp @ 0x5594aafda0] v_pitch = 960
[h264_rkmpp @ 0x5594aafda0] dst_height = 1090
[h264_rkmpp @ 0x5594aafda0] dst_height / 2 = 545
[h264_rkmpp @ 0x5594aafda0] RGA pitch error2
[h264_rkmpp @ 0x5594aafda0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
[h264_rkmpp @ 0x5594aafda0] dst_y = 0x5595311d30
[h264_rkmpp @ 0x5594aafda0] dst_u = 0x5594f05d40
[h264_rkmpp @ 0x5594aafda0] dst_v = 0x5594f85930
[h264_rkmpp @ 0x5594aafda0] width = 1920
[h264_rkmpp @ 0x5594aafda0] height = 1080
[h264_rkmpp @ 0x5594aafda0] hstride = 1920
[h264_rkmpp @ 0x5594aafda0] vstride = 1088
[h264_rkmpp @ 0x5594aafda0] y_pitch = 1920
[h264_rkmpp @ 0x5594aafda0] u_pitch = 960
[h264_rkmpp @ 0x5594aafda0] v_pitch = 960
[h264_rkmpp @ 0x5594aafda0] dst_height = -2210
[h264_rkmpp @ 0x5594aafda0] dst_height / 2 = -1105
[h264_rkmpp @ 0x5594aafda0] RGA pitch error2
[h264_rkmpp @ 0x5594aafda0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
frame=    5 fps=1.6 q=-0.0 size=N/A time=00:00:00.30 bitrate=N/A speed=0.0956x  [h264_rkmpp @ 0x5594aafda0] dst_y = 0x5595013620
[h264_rkmpp @ 0x5594aafda0] dst_u = 0x5595212550
[h264_rkmpp @ 0x5594aafda0] dst_v = 0x5595292140
[h264_rkmpp @ 0x5594aafda0] width = 1920
[h264_rkmpp @ 0x5594aafda0] height = 1080
[h264_rkmpp @ 0x5594aafda0] hstride = 1920
[h264_rkmpp @ 0x5594aafda0] vstride = 1088
[h264_rkmpp @ 0x5594aafda0] y_pitch = 1920
[h264_rkmpp @ 0x5594aafda0] u_pitch = 960
[h264_rkmpp @ 0x5594aafda0] v_pitch = 960
[h264_rkmpp @ 0x5594aafda0] dst_height = 1090
[h264_rkmpp @ 0x5594aafda0] dst_height / 2 = 545
[h264_rkmpp @ 0x5594aafda0] RGA pitch error2
[h264_rkmpp @ 0x5594aafda0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
[h264_rkmpp @ 0x5594aafda0] dst_y = 0x5594c07630
[h264_rkmpp @ 0x5594aafda0] dst_u = 0x5594e06560
[h264_rkmpp @ 0x5594aafda0] dst_v = 0x5594e86150
[h264_rkmpp @ 0x5594aafda0] width = 1920
[h264_rkmpp @ 0x5594aafda0] height = 1080
[h264_rkmpp @ 0x5594aafda0] hstride = 1920
[h264_rkmpp @ 0x5594aafda0] vstride = 1088
[h264_rkmpp @ 0x5594aafda0] y_pitch = 1920
[h264_rkmpp @ 0x5594aafda0] u_pitch = 960
[h264_rkmpp @ 0x5594aafda0] v_pitch = 960
[h264_rkmpp @ 0x5594aafda0] dst_height = 1090
[h264_rkmpp @ 0x5594aafda0] dst_height / 2 = 545
[h264_rkmpp @ 0x5594aafda0] RGA pitch error2
[h264_rkmpp @ 0x5594aafda0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
frame=    6 fps=1.6 q=-0.0 size=N/A time=00:00:00.36 bitrate=N/A speed=0.0976x  [h264_rkmpp @ 0x5594aafda0] dst_y = 0x5595013620
[h264_rkmpp @ 0x5594aafda0] dst_u = 0x5595212550
[h264_rkmpp @ 0x5594aafda0] dst_v = 0x5595292140
[h264_rkmpp @ 0x5594aafda0] width = 1920
[h264_rkmpp @ 0x5594aafda0] height = 1080
[h264_rkmpp @ 0x5594aafda0] hstride = 1920
[h264_rkmpp @ 0x5594aafda0] vstride = 1088
[h264_rkmpp @ 0x5594aafda0] y_pitch = 1920
[h264_rkmpp @ 0x5594aafda0] u_pitch = 960
[h264_rkmpp @ 0x5594aafda0] v_pitch = 960
[h264_rkmpp @ 0x5594aafda0] dst_height = 1090
[h264_rkmpp @ 0x5594aafda0] dst_height / 2 = 545
[h264_rkmpp @ 0x5594aafda0] RGA pitch error2
[h264_rkmpp @ 0x5594aafda0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
[h264_rkmpp @ 0x5594aafda0] dst_y = 0x5595311d30
[h264_rkmpp @ 0x5594aafda0] dst_u = 0x5594f05d40
[h264_rkmpp @ 0x5594aafda0] dst_v = 0x5594f85930
[h264_rkmpp @ 0x5594aafda0] width = 1920
[h264_rkmpp @ 0x5594aafda0] height = 1080
[h264_rkmpp @ 0x5594aafda0] hstride = 1920
[h264_rkmpp @ 0x5594aafda0] vstride = 1088
[h264_rkmpp @ 0x5594aafda0] y_pitch = 1920
[h264_rkmpp @ 0x5594aafda0] u_pitch = 960
[h264_rkmpp @ 0x5594aafda0] v_pitch = 960
[h264_rkmpp @ 0x5594aafda0] dst_height = -2210
[h264_rkmpp @ 0x5594aafda0] dst_height / 2 = -1105
[h264_rkmpp @ 0x5594aafda0] RGA pitch error2
[h264_rkmpp @ 0x5594aafda0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
JeffyCN commented 2 years ago

that log means the dst buf is using non-contig buffers.

so you should look into the custom avctx->get_buffer2 API about how it alloc dst buffers, and make it contig(alloc a big buffer and split into 3 planes(y u v)). if it use the default get_buffer2, it should be hacked in: https://github.com/JeffyCN/FFmpeg/commit/72ee109dcc79b560724b50e60d3a8e4ed5853242#diff-9b62a122bdcc988d8fc3ae9e4ffb26432777816b68fad28fe93f18dae4b337beR454

avafinger commented 2 years ago

Thank you @JeffyCN

avafinger commented 2 years ago

My last question, i get ~80% CPU usage if I pass to ffmpeg: -pix_fmt bgra , how can I know if it is done by the rga?

avafinger commented 2 years ago

Ahh, i think i got the answer: [swscaler @ 0x55a0c26d80] No accelerated colorspace conversion found from yuv420p to bgra.

Can it be done by RGA? If yes, can you advise or point to the right direction?

JeffyCN commented 2 years ago

usually we would provide NV12 drm_prime(or converted I420 for compatible with users like chromium/firefox). and the users should do the conversion(use swscaler/libyuv/pixman in software way or use GPU/RGA in hardware way) if they need other formats.

or maybe you can try to add brga format support in rkmppdec.c, like what we did in: https://github.com/JeffyCN/FFmpeg/commit/2ddb699082bbf7474b1224271c69972a21fc5c23

avafinger commented 2 years ago

Hi @JeffyCN

I am using your contig buffer code but it fails for a similar reason, here is the error: sudo ffmpeg -benchmark -vcodec hevc_rkmpp -i tos-4096x1720-tiles.mkv -map 0:v:0 -f null -

ffmpeg version N-94777-g067b47f66c using your HACK.

mpp[3088]: mpp_info: mpp version: 6dadc7e1 author: Ding Wei      2021-09-02 [hal_h265d]: Use MPP_OK default, avoid the api->control is NULL
mpp[3088]: mpp_rt: NOT found ion allocator
mpp[3088]: mpp_rt: found drm allocator
mpp[3088]: H265D_PARSER: No start code is found.
mpp[3088]: mpp_buf_slot: set frame info: w 4096 h 1714 hor 4352 ver 1720
mpp[3088]: mpp_dec: setting default w 4096 h 1714 h_str 4352 v_str 1720
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_rkmpp) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[hevc_rkmpp @ 0x559a061db0] Decoder noticed an info change (4096x1714), format=0
[hevc_rkmpp @ 0x559a061db0] RGA init OK
rga_api version 1.3.0_[1] (RGA is compiling with meson base: $PRODUCT_BASE)
 RgaBlit(1361) RGA_BLIT fail: Bad address RgaBlit(1362) RGA_BLIT fail: Bad address
fd-vir-phy-hnd-format[10, (nil), (nil), (nil), 0]
rect[0, 0, 4096, 1714, 4352, 1720, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f99c20010, (nil), (nil), 0]
rect[0, 0, 4096, 1714, 4352, 1720, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[hevc_rkmpp @ 0x559a061db0] RGA blit failed
[hevc_rkmpp @ 0x559a061db0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.32.104
    Stream #0:0(eng): Video: wrapped_avframe, yuv420p, 4096x1714 [SAR 1:1 DAR 2048:857], q=2-31, 200 kb/s, 24 fps, 24 tbn, 24 tbc (default)
    Metadata:
      encoder         : Lavc58.56.101 wrapped_avframe
[null @ 0x559a0774f0] Encoder did not produce proper pts, making some up.
frame=    1 fps=0.9 q=-0.0 size=N/A time=00:00:00.04 bitrate=N/A speed=0.038x    RgaBlit(1361) RGA_BLIT fail: Bad address RgaBlit(1362) RGA_BLIT fail: Bad address
fd-vir-phy-hnd-format[21, (nil), (nil), (nil), 0]
rect[0, 0, 4096, 1714, 4352, 1720, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f91b23010, (nil), (nil), 0]
rect[0, 0, 4096, 1714, 4352, 1720, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[hevc_rkmpp @ 0x559a061db0] RGA blit failed
[hevc_rkmpp @ 0x559a061db0] Doing slow software conversion
Error while decoding stream #0:0: Operation not permitted
frame=    1 fps=0.5 q=-0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A speed=0.0189x    
video:1kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
bench: utime=2.117s stime=0.230s rtime=2.212s
bench: maxrss=71136kB
Exiting normally, received signal 2.

But using your custom code (contig buffer) ffmpeg version 'cbda536d19' it can decode just fine.

mpp[3374]: mpp_info: mpp version: 6dadc7e1 author: Ding Wei      2021-09-02 [hal_h265d]: Use MPP_OK default, avoid the api->control is NULL
mpp[3374]: mpp_rt: NOT found ion allocator
mpp[3374]: mpp_rt: found drm allocator
mpp[3374]: mpp: deprecated block control, use timeout control instead
mpp[3374]: mpp: deprecated block control, use timeout control instead
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_rkmpp) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
mpp[3374]: H265D_PARSER: No start code is found.
[hevc_rkmpp @ 0x557e17bbf0] Decoder noticed an info change (4096x1714), format=0
rga_api version 1.3.0_[1] (RGA is compiling with meson base: $PRODUCT_BASE)
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0(eng): Video: wrapped_avframe, nv12, 4096x1714 [SAR 1:1 DAR 2048:857], q=2-31, 200 kb/s, 24 fps, 24 tbn, 24 tbc (default)
    Metadata:
      encoder         : Lavc58.35.100 wrapped_avframe
[null @ 0x557e190920] Encoder did not produce proper pts, making some up.
frame=    8 fps=0.0 q=-0.0 size=N/A time=00:00:00.33 bitrate=N/A speed=0.632x   frame=   22 fps= 21 q=-0.0 size=N/A time=00:00:00.91 bitrate=N/A speed=0.883x   frame=   34 fps= 22 q=-0.0 size=N/A time=00:00:01.41 bitrate=N/A speed=0.917x   frame=   48 fps= 23 q=-0.0 size=N/A time=00:00:02.00 bitrate=N/A speed=0.962x   frame=   62 fps= 24 q=-0.0 size=N/A time=00:00:02.58 bitrate=N/A speed=0.996x   frame=17619 fps= 25 q=-0.0 Lsize=N/A time=00:12:14.12 bitrate=N/A speed=1.06x    
video:9222kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
bench: utime=30.128s stime=152.012s rtime=691.748s
bench: maxrss=58508kB

Unfortunately, the new FFmpeg version has problems with colors when decoding any hevc-10-bit (librga issue?). file: https://jell.yfish.us/media/jellyfish-40-mbps-hd-hevc-10bit.mkv

Is it possible for you to review why https://github.com/JeffyCN/FFmpeg/commit/d28371e2e63b7ed5abfd81defc12c4b380b8c4ed#diff-9b62a122bdcc988d8fc3ae9e4ffb26432777816b68fad28fe93f18dae4b337beR457 does not work with tos-4096x1720-tiles.mkv (possible with any 4k) (contig buffer error above)? file: https://www.libde265.org/hevc-bitstreams/tos-4096x1720-tiles.mkv

JeffyCN commented 2 years ago

4096 is too large, limit is 3840x2160

JeffyCN commented 2 years ago

wait, i was wrong, it could decode up to 4096x4096, but prefer under 3840x2160. and the rga's dst limit is 4096x4096.

the issues you mentions could due to: 1/ the decoder aligned output stride to 4352, and my patch is reusing that as dst stride, which is over rga limit. should use round_up(width, 2) 2/ the conversion code doesn't suppor nv12-10bit, hardcoded to nv12 now. but guessing i can add it later

JeffyCN commented 2 years ago

try: https://github.com/JeffyCN/FFmpeg/commits/issue1 rkmppdec: Fix conversion error for NV12 10BIT … rkmppdec: Drop MPP alignment when doing frame conversion … HACK: fftools/ffmpeg: Use avcodec_default_get_buffer2 directly …

avafinger commented 2 years ago

Thank you for your work!

4096 fixed, I just noticed is a bit slower than the old ffmpeg. But that seems to be bgra conversion, I think. hevc 10-bit is fixed for some files, please, see the one for reference.

fixed for: wget https://jell.yfish.us/media/jellyfish-25-mbps-hd-hevc.mkv jellyfish-25-mbps-hd-hevc

not this one: wget https://jell.yfish.us/media/jellyfish-40-mbps-hd-hevc-10bit.mkv jellyfish-40-mbps-hd-hevc-10bit

Please, can you check the file and see what you find?

avafinger commented 2 years ago

FYI, before the alignment and N12-10bit conversion fix:

before

avafinger commented 2 years ago

Maybe this info can help: [hevc_rkmpp @ 0x55b1fda6b0] before: size = 5474304 [hevc_rkmpp @ 0x55b1fda6b0] before hstride = 2816 [hevc_rkmpp @ 0x55b1fda6b0] before vstride = 1080 [hevc_rkmpp @ 0x55b1fda6b0] new: size = 3110400 [hevc_rkmpp @ 0x55b1fda6b0] new: hstride = 1920 [hevc_rkmpp @ 0x55b1fda6b0] new: vstride = 1080 [hevc_rkmpp @ 0x55b1fda6b0] rga_set_rect(&rct,0,0,1920,1080,2816,1080,mppformat: 1 -> rgaformat: 0

but i have:

define DRM_FORMAT_NV12_10 fourcc_code('N', 'A', '1', '2') / 2x2 subsampled Cr:Cb plane /

double checking again...

avafinger commented 2 years ago

Ok, the reason for the conversion is my kernel header (really odd, I will fix it somehow):

CC  libavcodec/rkmppdec.o
libavcodec/rkmppdec.c: In function ‘rkmpp_get_frameformat’:
libavcodec/rkmppdec.c:96:44: error: ‘DRM_FORMAT_NV12_10’ undeclared (first use in this function); did you mean ‘DRM_FORMAT_NV12’?
   96 |     case MPP_FMT_YUV420SP_10BIT:    return DRM_FORMAT_NV12_10;
      |                                            ^~~~~~~~~~~~~~~~~~
      |                                            DRM_FORMAT_NV12

the missing part then is the alignment for that hevc-10bit file.

avafinger commented 2 years ago

@JeffyCN

Your patch worked!

Thank you!.