Decoding midway into a stream

[kestrelm]

Hello,

I am implementing the decoder for a livestream type application. Using your reference hello-decode example, I noticed the following:

    // Prepare input bitstream and start decoding
    bitstream.MaxLength = BITSTREAM_BUFFER_SIZE;
    bitstream.Data      = (mfxU8 *)calloc(bitstream.MaxLength, sizeof(mfxU8));
    VERIFY(bitstream.Data, "Not able to allocate input buffer");
    bitstream.CodecId = MFX_CODEC_HEVC;

    // Pre-parse input stream
    sts = ReadEncodedStream(bitstream, source);
    VERIFY(MFX_ERR_NONE == sts, "Error reading bitstream\n");

    decodeParams.mfx.CodecId = MFX_CODEC_HEVC;
    decodeParams.IOPattern   = MFX_IOPATTERN_OUT_SYSTEM_MEMORY;
    sts                      = MFXVideoDECODE_DecodeHeader(session, &bitstream, &decodeParams);
    VERIFY(MFX_ERR_NONE == sts, "Error decoding header\n");

    // input parameters finished, now initialize decode
    sts = MFXVideoDECODE_Init(session, &decodeParams);
    VERIFY(MFX_ERR_NONE == sts, "Error initializing decode\n");

This only works if the decoder starts from the beginning of the stream at which point it proceeds to decode properly for each subsequent MFXVideoDECODE_DecodeFrameAsync() call.

However, what happens if you are not at the beginning of the stream ( aka join it halfway, like during a live streaming event? ) In this case, my testing indicates that the:

    // input parameters finished, now initialize decode
    sts = MFXVideoDECODE_Init(session, &decodeParams);
    VERIFY(MFX_ERR_NONE == sts, "Error initializing decode\n");

will fail with "Error initializing decode". What should I be doing in order to join a stream midway if I do not have the data from the very start of the stream? Should I be caching the start stream data somehere and use it to "boot up/start up" the decoder before proceeding to decode the actual stream if I were to join the stream in the middle?

Thanks

[jeffreymcallister]

Sorry for the delayed reply. You are right. The hello-decode example is simplistic and does not cover cases like starting mid-stream.

However, the API is designed to handle this scenario. As you can see from the oneVPL spec for MFXVideoDECODE_DecodeHeader ( https://spec.oneapi.io/onevpl/latest/API_ref/VPL_func_vid_decode.html?highlight=mfxvideodecode_decodeheader#mfxvideodecode-decodeheader), this function will return MFX_ERR_MORE_DATA if a header cannot be parsed with the bitstream data available.

A more complete implementation would have DecodeHeader in a loop, with the MORE_DATA error indicating that the app must supply more data from the bitstream to continue. This will skip frame data in the bitstream until a suitable header can be found.

The code below keeps reading in 100 byte increments until an IDR frame is parsed, even if started in the middle of a GOP:

 while (isStillGoing) {
      sts = MFXVideoDECODE_DecodeHeader(session, &bitstream, &decodeParams);
      if (MFX_ERR_MORE_DATA == sts) {
        sts = ReadEncodedStreamBytes(bitstream, source, 100);
        if (MFX_ERR_NONE!=sts) goto end;
        continue;
      }
      isStillGoing=false;
    }

This was tested in hello-decode, with the ReadEncodedStreamBytes function here:

mfxStatus ReadEncodedStreamBytes(mfxBitstream &bs, FILE *f, mfxU32 read_incr) {
    mfxU8 *p0 = bs.Data;
    mfxU8 *p1 = bs.Data + bs.DataOffset;
    if (bs.DataOffset > bs.MaxLength - 1) {
        return MFX_ERR_NOT_ENOUGH_BUFFER;
    }
    if (bs.DataLength + bs.DataOffset > bs.MaxLength) {
        return MFX_ERR_NOT_ENOUGH_BUFFER;
    }
    for (mfxU32 i = 0; i < bs.DataLength; i++) {
        *(p0++) = *(p1++);
    }
    bs.DataOffset = 0;
    mfxU32 read_bytes=(mfxU32)fread(bs.Data + bs.DataLength, 1, read_incr, f);
    bs.DataLength += read_bytes;
    if (bs.DataLength == 0)
        return MFX_ERR_MORE_DATA;

    return MFX_ERR_NONE;
}

Let us know if this does not answer your question.

[kestrelm]

Thanks! I will follow your example to address the issue.