How to get buffer from drm_prime?

[FH0]

I use av_hwframe_transfer_data to convert drm_prime to nv12. nv12 has two planes, plane 0 size should be 2073600, plane 1 size should be 1036800, for case 1920x1080. But in fact nv12 plane 0 size is 3133568 and plane 1 will cause Segmentation fault.

I use it in my Qt project.

void onFrame(AVFrame *frame) {
    if (videoSink == nullptr) {
        return;
    }

    auto        fmt = getFormat(static_cast<AVPixelFormat>(frame->format));
    QVideoFrame videoFrame({QSize(frame->width, frame->height), fmt});
    videoFrame.map(QVideoFrame::WriteOnly);
    for (int i = 0; i < videoFrame.planeCount(); i++) {
        auto buf = av_frame_get_plane_buffer(frame, i);
        memcpy(videoFrame.bits(i), buf->data, buf->size);
    }
    videoFrame.unmap(); // TODO use drm prime

    emit videoSink->videoFrameChanged(videoFrame);
}

[FH0]

the second linesize should be 960.

04-08 03:43:42.924 I videoFrame.planeCount: 2, format: nv12
04-08 03:43:42.924 I i: 0 linesize: 1920 height: 1080
04-08 03:43:42.926 I i: 1 linesize: 1920 height: 1080

If I copy with hard code, it will be OK.

void onFrame(AVFrame *frame) {
    if (videoSink == nullptr) {
        return;
    }

    auto        fmt = getFormat((AVPixelFormat)frame->format);
    QVideoFrame videoFrame({QSize(frame->width, frame->height), fmt});
    videoFrame.map(QVideoFrame::WriteOnly);
    memcpy(videoFrame.bits(0), frame->data[0], 1920 * 1080);
    memcpy(videoFrame.bits(1), frame->data[1], 1920 * 1080 / 2);
    videoFrame.unmap(); // TODO use drm prime

    emit videoSink->videoFrameChanged(videoFrame);
}

[FH0]

Finally solved by

void onFrame(AVFrame *frame) {
    if (videoSink == nullptr) {
        return;
    }

    auto        fmt = getFormat((AVPixelFormat)frame->format);
    QVideoFrame videoFrame({QSize(frame->width, frame->height), fmt});
    videoFrame.map(QVideoFrame::WriteOnly);
    for (int i = 0; i < videoFrame.planeCount(); i++) {
        memcpy(videoFrame.bits(i), frame->data[i], videoFrame.mappedBytes(i));
    }
    videoFrame.unmap(); // TODO use drm prime

    emit videoSink->videoFrameChanged(videoFrame);
}

But I still don't know how to get buffer from avframe by ffmpeg way.

[nyanmisaka]

@FH0

the second linesize should be 960.

It is a semi-planar format, U and V are interleaved, so the 2nd linesize should be 1920.

1.5: bytes_per_pixel of NV12 Y buf: 1920xAlign(1080)x1.5=3133440 (+128=3133568) //<= calc by MPP runtime UV buf: 1920xAlign(540)x1.5=1566720

See this func for more info https://github.com/nyanmisaka/ffmpeg-rockchip/blob/ba84e56c51d9cde1f3b1fead2a21e4d271028709/libavutil/hwcontext_rkmpp.c#L192