Decoding frame data delay

[1204465458]

hello friend, a problem, I use ffmpeg api av_read_frame() read a packet from rtsp stream, then decode by onevpl, but the output yuv data has a delay, and the longer the testing time, the greater delay, follow picture show, the picture save time is 2024-06-05 11:17:47, but rtsp camera shou the time is 2024-06-05 11:17:42, 5 seconds delay before and after decoding, thanks for providing some suggestions.

folllow is init decoder code:

mfxVideoParam param{}; param.mfx.CodecId = MFX_CODEC_AVC; param.IOPattern = MFX_IOPATTERN_OUT_SYSTEM_MEMORY; param.mfx.FrameInfo.FourCC = MFX_FOURCC_NV12; param.mfx.FrameInfo.ChromaFormat = MFX_CHROMAFORMAT_YUV420; param.mfx.FrameInfo.PicStruct = MFX_PICSTRUCT_PROGRESSIVE; param.mfx.FrameInfo.FrameRateExtN = fps; param.mfx.FrameInfo.FrameRateExtD = 1; param.mfx.FrameInfo.CropX = 0; param.mfx.FrameInfo.CropY = 0; param.mfx.FrameInfo.CropW = width; param.mfx.FrameInfo.CropH = height; param.mfx.FrameInfo.Width = ALIGN16(width); param.mfx.FrameInfo.Height = ALIGN16(height); param.AsyncDepth = 8; ret = MFXVideoDECODE_Init(session, &param); if (ret != MFX_ERR_NONE) { std::cerr << "MFXVideoDECODE_Init failed, error code: " << ret << std::endl; return -1; }

follow is decode code:

memmove(bitstream.Data, bitstream.Data + bitstream.DataOffset, bitstream.DataLength); bitstream.DataOffset = 0; memcpy(bitstream.Data + bitstream.DataLength, data, size); bitstream.DataLength += size;

auto ret = MFXVideoDECODE_DecodeFrameAsync(session, &bitstream, &surfPool[index], &surfaceOut, &sync);
switch (ret) {
    case MFX_ERR_NONE:
        while (true) {
            ret = MFXVideoCORE_SyncOperation(session, sync, waitTime);
            if (ret == MFX_ERR_NONE) {
                return RawFrame(surfaceOut);
            }

            if (ret == MFX_WRN_IN_EXECUTION) {
                continue;
            } else {
                std::cerr << "decode MFXVideoCORE_SyncOperation failed, error code: " << ret << std::endl;
                return nullptr;
            }
        }
    case MFX_ERR_MORE_DATA:
        std::cout<<"Expect more data at input"<<std::endl;
        return nullptr;
    case MFX_ERR_MORE_SURFACE:
        // The function requires more frame surface at output before decoding can proceed.
        // This applies to external memory allocations and should not be expected for
        // a simple internal allocation case like this
        index = GetFreeSurfaceIndex();
        if (index < 0) {
            std::cerr << "GetFreeSurfaceIndex failed" << std::endl;
        }
        std::cout << "Expect more surface at output" << std::endl;
        return nullptr;
    case MFX_ERR_DEVICE_LOST:
        std::cout << "Lose the hardware acceleration device" << std::endl;
        return nullptr;
    case MFX_WRN_DEVICE_BUSY:
        std::cout << "decode the hardware acceleration device is busy" << std::endl;
        //usleep(5 * 1000);   ///< maybe don't need
        return nullptr;
    case MFX_WRN_VIDEO_PARAM_CHANGED:
        // The decoder detected a new sequence header in the bitstream
        // video parameters may have changed, In external memory allocation case, might need to reallocate the output surface
        //std::cout<<"The video parameters are changed during decoding"<<std::endl;
        return nullptr;
    case MFX_ERR_INCOMPATIBLE_VIDEO_PARAM:
        // The function detected that video parameters provided by the application
        // are incompatible with initialization parameters, the application should close the component and then reinitialize it
        std::cout << "Incompatible video parameters" << std::endl;
        return nullptr;
    case MFX_ERR_REALLOC_SURFACE:
        // Bigger surface_work required. May be returned only if
        // mfxInfoMFX::EnableReallocRequest was set to ON during initialization,
        // this applies to external memory allocations and should not be expected for
        // a simple internal allocation case like this
        std::cout << "Bigger output surface required" << std::endl;
        return nullptr;
    default:
        std::cerr << "unknown status: " << ret << std::endl;
        return nullptr;
}

[akwrobel]

This appears to be a runtime issue and it should be filed here: https://github.com/intel/vpl-gpu-rt/issues