关于mpp解码得到的frame的问题

[Kevin111369]

操作

使用mpp_frame_get_buffer()函数得到了解码器输出的frame对应的MppBuffer，然后使用mpp_buffer_get_ptr()函数得到了MppBuffer的地址，然后根据这个地址对图像数据进行操作，例如画框。

问题

对第一帧的数据操作没有问题，但是在第二帧的时候MppBuffer中的数据有残留第一帧中画的框，是不是因为MppBuffer对应的数据空间没有清空的原因。

实验结果

第一帧解码的结果：

第一帧画框的结果：

第二帧解码的结果：

第二帧画框的结果：

[Kevin111369]

解码器使用的frame内存分配模式为半内部分配模式

[Kevin111369]

我在处理完一帧frame后进行了更新，但是frame对应的MppBuffer好像并没有释放，第二帧还是有第一帧画框的残留数据，frame更新代码如下

int dec_frame_update(dec_data_t* mpp_data)
{
    MPP_RET ret = MPP_OK;

    ret = mpp_frame_deinit(&mpp_data->frm_info.frame);
    if (ret != MPP_OK)
    {
        SAMPLE_ERROR_PRT("mpp_frame_deinit failed\n");
        return ret;
    }

    if (mpp_data->frm_info.buffer)
    {
        ret  = mpp_buffer_put(mpp_data->frm_info.buffer);
        if (MPP_OK != ret) {
            SAMPLE_ERROR_PRT("mpp_buffer_put failed\n");
            return ret;
        }
        mpp_data->frm_info.buffer = NULL;
    }

    return ret;
}

[nyanmisaka]

You cannot draw on frames (fd/ptr) used exclusively by the decoder. They must be read-only, modifying them will break the subsequent decoding process.

[Kevin111369]

You cannot draw on frames (fd/ptr) used exclusively by the decoder. They must be read-only, modifying them will break the subsequent decoding process.

So it is impossible to use zero copy to complete the function of the picture frame?

[nyanmisaka]

You cannot draw on frames (fd/ptr) used exclusively by the decoder. They must be read-only, modifying them will break the subsequent decoding process.

So it is impossible to use zero copy to complete the function of the picture frame?

From the corrupted image it seems to be caused by:

1. decoder internal buffer is polluted?
2. cacheable buffer is not synced before accessing by cpu drawing?
3. incorrect frame index?

Did you sync to cpu before and after drawing boxes? e.g. decode -> get_frame -> get_ptr -> sync to cpu (fd) -> draw boxes -> invalidate sync (fd) -> display -> deinit_frame

[Kevin111369]

You cannot draw on frames (fd/ptr) used exclusively by the decoder. They must be read-only, modifying them will break the subsequent decoding process.

So it is impossible to use zero copy to complete the function of the picture frame?

From the corrupted image it seems to be caused by:

1. decoder internal buffer is polluted?
2. cacheable buffer is not synced before accessing by cpu drawing?
3. incorrect frame index?

Did you sync to cpu before and after drawing boxes? e.g. decode -> get_frame -> get_ptr -> sync to cpu (fd) -> draw boxes -> invalidate sync (fd) -> display -> deinit_frame

Sorry, I didn't consider the synchronization issue when using the CPU to draw. I don't know much about sync. Are the corresponding functions for sync to cpu (fd) and invalidate sync (fd) operations mpp_buffer_sync_begin and mpp_buffer_sync_end ?

[Kevin111369]

@nyanmisaka I tried to use the mpp_buffer_sync_begin and mpp_buffer_sync_end to synchronize before and after CPU drawing, but it is still the same. Maybe the MppBuffer corresponding to the frame is read-only.

[HermanChen]

Modify the zero-copy buffer which is still used by decoder as reference frame is dangerous for it may corrupt the original reference frame content. If the box drawing is needed then try RGA to do a A+B->C copy with adding box as osd.

[Kevin111369]

Modify the zero-copy buffer which is still used by decoder as reference frame is dangerous for it may corrupt the original reference frame content. If the box drawing is needed then try RGA to do a A+B->C copy with adding box as osd.

I tried to use the osd function in rga at the beginning, but the osd in rga currently does not support rk3588, so I gave up on rga.

[nyanmisaka]

RGA SRC channel (decoder image) RGA PAT channel (draw boxes on an empty RGBA image) RGA DST channel (output result)

You can use imcomposite() or imblend() instead.

[Kevin111369]

@nyanmisaka @HermanChen Thanks for the tip, I'll give it a try!

[Kevin111369]

RGA SRC channel (decoder image) RGA PAT channel (draw boxes on an empty RGBA image) RGA DST channel (output result)

You can use imcomposite() or imblend() instead.

@nyanmisaka Hello, I'm using imcomposite to implement osd, but I'm having trouble with the process. The SRC channel is the output image of the decoder in the format of NV12. The SRC1 channel is the frame image in the format of RGBA8888, Alpha value = 0xff. The DST channel is the output image in the format of NV12. I set the parameter modein the imcomposite function to IM_ALPHA_BLEND_DST_OVER, and the resulting image SRC1 completely covers SRC. I want to overlay only the area of the rectangle and make the other white areas transparent. How should I set the parameter mode? The experimental images are as follows: src image(NV12): src1 image(RGBA888): dst image(NV12):

[nyanmisaka]

Can you check if the alpha value of the pixels in the blank spaces of the SRC1 image is transparent? It should be 0.

[nyanmisaka]

If they are already 0, then you need to set the premultiplied alpha flag IM_ALPHA_BLEND_PRE_MUL.

[Kevin111369]

Can you check if the alpha value of the pixels in the blank spaces of the SRC1 image is transparent? It should be 0.

Here's what I do: First create src1 and set all the data in buf to 0xff. The code is as follows:

rect_width = frame_info->width;
rect_height = frame_info->height;
rect_hor_stride = frame_info->hor_stride;
rect_ver_stride = frame_info->ver_stride;
rect_format = IMAGE_FORMAT_RGBA8888;

rect_drm_data.drm_buf = (uint8_t *)drm_buf_alloc( );

rect_buf_size = rect_drm_data.actual_size;
rect_drm_fd = rect_drm_data.drm_buffer_fd;
rect_buf = rect_drm_data.drm_buf;

memset(rect_buf, 0xff, rect_buf_size);

Then draw, and the color settings used in the drawing were as follows:

// Color Format ARGB8888
#define COLOR_GREEN     0xFF00FF00
#define COLOR_BLUE      0xFF0000FF
#define COLOR_RED       0xFFFF0000
#define COLOR_YELLOW    0xFFFFFF00
#define COLOR_ORANGE    0xFFFF4500
#define COLOR_BLACK     0xFF000000
#define COLOR_WHITE     0xFFFFFFFF

Finally, I superimposed src and src1. The code is as follows:

src_handle = importbuffer_fd(src_drm_fd, src_buf_size);
rect_handle = importbuffer_fd(rect_drm_fd, rect_buf_size);
dst_handle = importbuffer_fd(dst_drm_fd, dst_buf_size);
if (src_handle == 0 || rect_handle == 0 || dst_handle == 0) {
    SAMPLE_ERROR_PRT("importbuffer failed!\n");
    goto release_buffer;
}

src_img = wrapbuffer_handle(src_handle, src_width, src_height, imageFormat2RgaFormat(src_format), src_hor_stride, src_ver_stride);
rect_img = wrapbuffer_handle(rect_handle, rect_width, rect_height, imageFormat2RgaFormat(rect_format), rect_hor_stride, rect_ver_stride);
dst_img = wrapbuffer_handle(dst_handle, dst_width, dst_height, imageFormat2RgaFormat(dst_format), dst_hor_stride, dst_ver_stride);

ret_rga = imcheck_composite(src_img, dst_img, rect_img, {}, {}, {});
if (IM_STATUS_NOERROR != ret_rga) {
    SAMPLE_ERROR_PRT("%d, check error! %s", __LINE__, imStrError((IM_STATUS)ret_rga));
    ret = -1;
    goto release_buffer;
}

ret_rga = imcomposite(src_img, rect_img, dst_img, IM_ALPHA_BLEND_DST_OVER | IM_ALPHA_BLEND_PRE_MUL);
if (ret_rga != IM_STATUS_SUCCESS) {
    SAMPLE_ERROR_PRT("imcomposite running failed, %s\n", imStrError((IM_STATUS)ret_rga));
    ret = -1;
    goto release_buffer;
}

[nyanmisaka]

First create src1 and set all the data in buf to 0xff.

You should init the image with zero. rgba(0xff, 0xff, 0xff, 0xff) is color white, which is not transparent.

[Kevin111369]

You should init the image with zero. rgba(0xff, 0xff, 0xff, 0xff) is color white, which is not transparent.

Thank you very much for your guidance and patience! I understand the principle of imcomposite. I will try it and wait for my good news!

[Kevin111369]

@nyanmisaka I tried to use rga_alpha_yuv_demo in librga to achieve the synthesis of yuv images and rgba images, but the effect I got was that rgba completely covered the yuv image. Below is my test image and result:BaiDuYun（提取码：zovv） Can you provide a 1920x1080 RGBA image for me to test?

[nyanmisaka]

I don't have a problem blending the first two input images that you provided. in0w1920-h1080-nv12.bin (1920x1078, nv12) in0w1920-h1080-rgba8888.bin (1920x1078, rgba)

It can be reproduced via ffmpeg-rockchip. https://github.com/nyanmisaka/ffmpeg-rockchip ./ffmpeg -init_hw_device rkmpp -f rawvideo -s 1920x1078 -pix_fmt nv12 -i /tmp/in0w1920-h1080-nv12.bin -f rawvideo -s 1920x1078 -pix_fmt rgba -i /tmp/in0w1920-h1080-rgba8888.bin -lavfi "[0:v]hwupload[main];[1:v]hwupload[sub];[main][sub]overlay_rkrga=format=nv12,hwmap=mode=read,format=nv12" -f image2 -y /tmp/out.png

[Kevin111369]

@nyanmisaka I have decided to forgo using the blend functionality of RGA. Instead, I am now obtaining the destination image (dst) by copying the source image (src) using imcopy. Subsequently, I am directly performing drawing operations on the dst image using the CPU.

[nyanmisaka]

That's also feasible, as long as you think it's cheap enough.

[Kevin111369]

That's also feasible, as long as you think it's cheap enough.

Yes, this method is more cost-effective. If I use the imcomposite function, I would need to create a new rect image, use the CPU to add the box to it, and then implement A + B -> C. On the other hand, using the copy + draw method is simpler and more cost-effective.

[Kevin111369]

@nyanmisaka When developing ffmpeg-rockchip, how did you adapt some parameter settings of ffmpeg to the mpp encoder? For example "-tune zerolatency", "-preset ultrafast".

[nyanmisaka]

@nyanmisaka When developing ffmpeg-rockchip, how did you adapt some parameter settings of ffmpeg to the mpp encoder? For example "-tune zerolatency", "-preset ultrafast".

They are just not supported by the MPP runtime. So, not implemented.