SDK version[3.1.0] Dropped Frame/Fragment[QUESTION]

[derek1875]

The 3.1.0 SDK did not resolve this issue (SDK version[3.0.0] Dropped Frame/Fragment[QUESTION] #642). The issue shows a few video/audio frames at the beginning of a stream may be dropped, So is there anything wrong with my configuration?

SDK3.1.0.log Attached is the verbose log. From this log, we could see

1. [KVS] contentViewTrimTilItems(): ContentView is not big enough to contain a single fragment.
2. Error code 0x52000084, it means STATUS_SETTING_KEY_FRAME_FLAG_WHILE_USING_EOFR . It seems to be an error while using both the key frame and the eofr frame.

Here is my pseudo streaming code:

PStreamInfo pstreamInfo;
CLIENT_HANDLE clientHandle;
PDeviceInfo pDeviceInfo;
STREAM_HANDLE streamHandle;
PClientCallbacks pClientCallbacks;
PCHAR defaultRegion = (PCHAR) "us-west-2";
UINT64 storageSize = 9 * 1024 * 1024;

void kinesis_video_init()
{
    CHK_STATUS(createRealtimeAudioVideoStreamInfoProvider(streamName,
                                                              2 * HUNDREDS_OF_NANOS_IN_AN_HOUR,
                                                              60 * HUNDREDS_OF_NANOS_IN_A_SECOND,
                                                              &pstreamInfo));
    for (i = 0; i < pstreamInfo->streamCaps.trackInfoCount && pstreamInfo->streamCaps.trackInfoList[i].trackType != MKV_TRACK_INFO_TYPE_AUDIO; ++i);
    CHK(i < pstreamInfo->streamCaps.trackInfoCount, STATUS_TRACK_INFO_MISSING);
    pTrackInfoAudio = pstreamInfo->streamCaps.trackInfoList + i;
    STRNCPY(pTrackInfoAudio->codecId, (PCHAR) "A_MS/ACM", MKV_MAX_CODEC_ID_LEN);
    pTrackInfoAudio->codecPrivateData = audioCpd;
    pTrackInfoAudio->codecPrivateDataSize = SIZEOF(audioCpd);

    STRNCPY(pstreamInfo->streamCaps.contentType, (PCHAR) "video/h264,audio/alaw", MAX_CONTENT_TYPE_LEN);
    pstreamInfo->streamCaps.frameRate = 130; 
    pstreamInfo->streamCaps.nalAdaptationFlags = NAL_ADAPTATION_ANNEXB_NALS | NAL_ADAPTATION_ANNEXB_CPD_NALS;
    pstreamInfo->streamCaps.absoluteFragmentTimes = TRUE;

    CHK_STATUS(createDefaultDeviceInfo(&pDeviceInfo));
    CHK_STATUS(setDeviceInfoStorageSize(pDeviceInfo, storageSize));
    pDeviceInfo->clientInfo.loggerLogLevel = LOG_LEVEL_ERROR;//LOG_LEVEL_VERBOSE;
    CHK_STATUS(createDefaultCallbacksProviderWithIotCertificate(endPoint,
                                                                        IOT_CERT_PATH,
                                                                        IOT_PRIVATE_KEY_PATH,
                                                                        IOT_CA_CERT_PATH,
                                                                        roleAlias,
                                                                        thingName,
                                                                        defaultRegion,
                                                                        NULL, NULL, &pClientCallbacks));

    CHK_STATUS(createStreamCallbacks(&pStreamCallbacks));
    pStreamCallbacks->customData = (UINT64)NULL;
    pStreamCallbacks->bufferDurationOverflowPressureFn = NULL;
    pStreamCallbacks->streamLatencyPressureFn = NULL;
    pStreamCallbacks->streamConnectionStaleFn = NULL;
    pStreamCallbacks->droppedFrameReportFn = defaultDroppedFrameReportCallback;
    pStreamCallbacks->droppedFragmentReportFn = defaultDroppedFragmentReportCallback;
    pStreamCallbacks->streamErrorReportFn = defaultStreamErrorReportCallback;
    pStreamCallbacks->fragmentAckReceivedFn = defaultFragmentAckReceivedCallback;
    pStreamCallbacks->streamReadyFn = NULL;
    pStreamCallbacks->streamClosedFn = NULL;
    pStreamCallbacks->streamShutdownFn = NULL;
    pStreamCallbacks->freeStreamCallbacksFn = NULL;

    CHK_STATUS(addStreamCallbacks(pClientCallbacks, pStreamCallbacks));

    PlatformCallbacks platformCallbacks = {0};
    platformCallbacks.logPrintFn = myLogging;
    CHK_STATUS(setPlatformCallbacks(pClientCallbacks, &platformCallbacks));
    CHK_STATUS(createKinesisVideoClient(pDeviceInfo, pClientCallbacks, &clientHandle));
    CHK_STATUS(freeDeviceInfo(&pDeviceInfo));

    createKinesisVideoStream(clientHandle, pstreamInfo, &streamHandle);
}

int main()
{
    kinesis_video_init();

    while(1) {
      //video fps is 15, so actually we are pushing a 12s video. 
      for(int i=0; i < 180; i++) {
        get_frame(frame, type);
        ...
        if (type == MEDIA_TYPE_VIDEO)
          frame.frame.flags = FRAME_FLAG_KEY_FRAME;
        else
          frame.flags = FRAME_FLAG_NONE;
        putKinesisVideoFrame(streamHandle, frame);
      }
      sleep(30);
    }
}

I noticed that I could not use key-frame flag in this version. (#am azon-kinesis-video-streams-producer-sdk-cpp-3.1.0/dependency/libkvscproducer/kvscproducer-src/dependency/libkvspic/kvspic-src/src/client/src/FrameOrderCoordinator.c:287) So should I remove the flag FRAME_FLAG_KEY_FRAME? But I have not found a proper sample to use. What's your suggestions?

[hassanctech]

Can you please try:

pstreamInfo->streamCaps.frameOrderingMode = FRAME_ORDERING_MODE_MULTI_TRACK_AV_COMPARE_DTS_ONE_MS_COMPENSATE_EOFR;

?

[derek1875]

Your suggestion resolved my issue and the frames have not been dropped over the last 24 hours, Thanks a lot.

[hassanctech]

Great! I do want to make one comment, so what you've identified is a bug in our release, I'm working on the fix now and there will likely be a new release that goes out to address this. The default value for frameOrderingMode should be set to what I suggested in the case that you have > 1 track.

[derek1875]

It's good to know your plan, I will keep on testing once new version being released.