Frame rendering without buffering, minimizing delay

[hrvthzs]

Is it possible to immediately render received frames? I am encoding frames using NVidia Video Codec SDK which already adds 1 frame delay. Encoded frames are send via websocket and fed to the player via decode. At this point I am at 3 frames behind the actually rendered.

I had a similar issue with JMuxer but it was even more behind. However I was able to send multiple (same) frames at once which could minimize the delay, when starting the stream feed enough data to the player. With Broadway I need to feed the frames separately, otherwise it breaks the decoding and rendering is not working.

Thanks for the help.

[soliton4]

there should not be any delay with broadway and the renderer. as long as you chose a low latency profile in your encoder, every nal unit should result in a frame. you get it immediately in the onframedecoded callback and can paint it using the renderer or whatever you prefer to render frames. also the video codec sdk should not add a delay

[hrvthzs]

Hi thanks for the fast response. Even if I output the encoded frames to a file and play it in VLC it is delayed by 1. Here is my code, I am using ultra low latency.

Initialization

NV_ENC_INITIALIZE_PARAMS initializeParams = { NV_ENC_INITIALIZE_PARAMS_VER };
NV_ENC_CONFIG encodeConfig = { NV_ENC_CONFIG_VER };
auto encodeCLIOptions = NvEncoderInitParam("", NULL, true);

initializeParams.encodeConfig = &encodeConfig;  
// Use ultra low lactency tuning
mEncoder->CreateDefaultEncoderParams(&initializeParams, encodeCLIOptions.GetEncodeGUID(), encodeCLIOptions.GetPresetGUID(), NV_ENC_TUNING_INFO_ULTRA_LOW_LATENCY);
// Profile must be Baseline when using Broadway player decoder
initializeParams.encodeConfig->profileGUID = NV_ENC_H264_PROFILE_BASELINE_GUID;
encodeCLIOptions.SetInitParams(&initializeParams, mImpl->mFormat);  
mEncoder->CreateEncoder(&initializeParams);

Encoding

const NvEncInputFrame* encoderInputFrame = mImpl->mEncoder->GetNextInputFrame();
NvEncoderCuda::CopyToDeviceFrame(mImpl->mContext, image, 0, (CUdeviceptr)encoderInputFrame->inputPtr,
    (int)encoderInputFrame->pitch,
    mImpl->mEncoder->GetEncodeWidth(),
    mImpl->mEncoder->GetEncodeHeight(),
    CU_MEMORYTYPE_DEVICE,
    encoderInputFrame->bufferFormat,
    encoderInputFrame->chromaOffsets,
    encoderInputFrame->numChromaPlanes);

std::vector<std::vector<uint8_t>> vPacket;
mImpl->mEncoder->EncodeFrame(vPacket);

After encoding the packet goes to the client via websocket and to the Broadway player. Initialization

const broadwayPlayer = new Player({useWorker: false});

Decoding

windowManager.onVideoFrameChanged(props.node,
    (frame:Uint8Array) =>
    {
        broadwayPlayer.decode(frame, new Object());
    }
);

NVEncoderer parameters

Encoding Parameters:
    codec        : h264
    preset       : p3
    tuningInfo   : ultralowlatency
    profile      : baseline(h264)
    chroma       : yuv420
    bitdepth     : 8
    rc           : cbr
    fps          : 30/1
    gop          : INF
    bf           : 0
    multipass    : 1
    size         : 1528x1316
    bitrate      : 0
    maxbitrate   : 0
    vbvbufsize   : 0
    vbvinit      : 0
    aq           : disabled
    temporalaq   : disabled
    lookahead    : disabled
    cq           : 0
    qmin         : P,B,I=0,0,0
    qmax         : P,B,I=0,0,0
    initqp       : P,B,I=0,0,0
NV_ENC_INITIALIZE_PARAMS:
    encodeGUID: h264
    presetGUID: p3
    tuningInfo: ultralowlatency
    encodeWidth: 1528
    encodeHeight: 1316
    darWidth: 1528
    darHeight: 1316
    frameRateNum: 30
    frameRateDen: 1
    enableEncodeAsync: 1
    reportSliceOffsets: 0
    enableSubFrameWrite: 0
    enableExternalMEHints: 0
    enableMEOnlyMode: 0
    enableWeightedPrediction: 0
    maxEncodeWidth: 1528
    maxEncodeHeight: 1316
    maxMEHintCountsPerBlock: 000000FB695FD708
NV_ENC_CONFIG:
    profile: baseline(h264)
    gopLength: 4294967295
    frameIntervalP: 1
    monoChromeEncoding: 0
    frameFieldMode: 1
    mvPrecision: 3
NV_ENC_RC_PARAMS:
    rateControlMode: 0x2
    constQP: 28, 31, 25
    averageBitRate:  0
    maxBitRate:      0
    vbvBufferSize:   0
    vbvInitialDelay: 0
    enableMinQP: 0
    enableMaxQP: 0
    enableInitialRCQP: 0
    enableAQ: 0
    qpMapMode: disabled
    multipass: qres
    enableLookahead: 0
    disableIadapt: 0
    disableBadapt: 0
    enableTemporalAQ: 0
    zeroReorderDelay: 0
    enableNonRefP: 0
    strictGOPTarget: 0
    aqStrength: 0
    minQP: 0, 0, 0
    maxQP: 0, 0, 0
    initialRCQP: 0, 0, 0
    temporallayerIdxMask: 0
    temporalLayerQP: 0, 0, 0, 0, 0, 0, 0, 0
    targetQuality:
    lookaheadDepth: 0
NV_ENC_CODEC_CONFIG (H264):
    enableStereoMVC: 0
    hierarchicalPFrames: 0
    hierarchicalBFrames: 0
    outputBufferingPeriodSEI: 0
    outputPictureTimingSEI: 0
    outputAUD: 0
    disableSPSPPS: 0
    outputFramePackingSEI: 0
    outputRecoveryPointSEI: 0
    enableIntraRefresh: 0
    enableConstrainedEncoding: 0
    repeatSPSPPS: 0
    enableVFR: 0
    enableLTR: 0
    qpPrimeYZeroTransformBypassFlag: 0
    useConstrainedIntraPred: 0
    level: 0
    idrPeriod: 4294967295
    separateColourPlaneFlag: 0
    disableDeblockingFilterIDC: 0
    numTemporalLayers: 0
    spsId: 0
    ppsId: 0
    adaptiveTransformMode: 2
    fmoMode: 0
    bdirectMode: 0
    entropyCodingMode: 1
    stereoMode: 0
    intraRefreshPeriod: 0
    intraRefreshCnt: 0
    maxNumRefFrames: 0
    sliceMode: 0
    sliceModeData: 0
NV_ENC_CONFIG_H264_VUI_PARAMETERS:
    overscanInfoPresentFlag: 0
    overscanInfo: 0
    videoSignalTypePresentFlag: 0
    videoFormat: 5
    videoFullRangeFlag: 0
    colourDescriptionPresentFlag: 0
    colourPrimaries: 2
    transferCharacteristics: 2
    colourMatrix: 2
    chromaSampleLocationFlag: 0
    chromaSampleLocationTop: 0
    chromaSampleLocationBot: 0
    bitstreamRestrictionFlag: 0
    ltrNumFrames: 0
    ltrTrustMode: 0
    chromaFormatIDC: 1
    maxTemporalLayers: 0

Do you see any problem with the config?

[soliton4]

sorry im not here to debug your code. streaming to vlc via file is not the best solution when you want low latency. and it is outside of the scope of this project

[hrvthzs]

I am sorry, wasn't asking for debugging. I am new to H264, could you please have a brief look at the encoding parameter if I set a wrong value somewhere? Thank you.

[soliton4]

i would recommend using worker threads. the decoder blocks for a considerable amount of time and during that time your network feed as well as your rendering are blocked as well. i dont see anything where you would lose a frame to latency

[hrvthzs]

Issue was caused by the NVEncoder, which has an extraOutputDelay parameter, which is larger than 0 by default. Now it works perferctly. Thanks.