[QUESTION] lack of `pCodecPrivate` for video track causes `TRACK_NUMBER_MISMATCH - 4010` error

[mitchellhansen-wyze]

Hello!

I have two implementations of KVS streaming in a project, one is using the https://github.com/awslabs/amazon-kinesis-video-streams-producer-c library, the other is using this one.

For the producer-c implementation I use the createRealtimeAudioVideoStreamInfoProvider to create the 2 tracks, and then manually set up the codecPrivateData via a hardcoded byte array for the audio track. I do not provide a codecPrivateData for the video track.

For the embedded-c implementation I attempted to do the same, I configure the audio tracks CPD via the same hardcoded byte array, and leave the video CPD as :

    video_track_info->pCodecPrivate = NULL;
    video_track_info->uCodecPrivateLen = 0;

When using these tracks to create the stream I don't get any errors :

    if ((stream_handle = Kvs_streamCreate(video_track_info, audio_track_info)) == NULL)
    {
        LOG_ERROR("Failed to create kvs stream");
        return EXIT_FAILURE;
    }

But, when I start actually streaming video, the Kvs_putMediaDoWork will error out with :

Error: Time:Mon Nov 21 16:23:42 2022 File:<path>/third/aws-embedded-c-kvs/src/source/restful/kvs/restapi_kvs.c Func:prvLogFragmentAck Line:533 PutMedia session error id:4010
ERROR: Failed in Kvs_putMediaDoWork [-FFFFF056]

This works just fine with the producer-c though, I do not have to provide a CPD for the video track in order for it to stream. Is there something I'm missing here? Or does producer-c just make assumptions for me in regards to the CPD that this lib does not?

Thanks!

[weichihl]

@mitchellhansen-wyze In my understanding, an empty codec private data should work fine on uploading videos. It only affects the decoding side.

And the error "TRACK_NUMBER_MISMATCH" happens when there is a frame whose track number is not listed in the MKV header.

In the sample_config.h, there is a compile flag DEBUG_STORE_MEDIA_TO_FILE. Enable it will store all the MKV data that has been sent out. Could you please share the MKV data to me so that I can check the MKV format? Thanks.

[mitchellhansen-wyze]

@weichihl As far as I can tell, DEBUG_STORE_MEDIA_TO_FILE only applies to the sample application? I'm working with a full integration of this library, so it isn't incredibly easy to instrument an environment in which I can run the sample code.

I did look through the sample application though, and noticed that it seems to just be dumping the MKV/frame information received from Kvs_dataFrameGetContent and Kvs_putMediaUpdate, I modified my integration to dump out this same information as hex to the console and got the following (Be aware that in my current environment I send blank audio data. So audio frames will be composed completely as 2A bytes) :

frame_dump_hex.txt

(if you paste this hex into a hex editor it will quickly reconstitute it as a binary file, apologies for the inconvenience)

Does this give you the information that you need? Thanks for your help!

[mitchellhansen-wyze]

@weichihl this might help as well, I tweaked my program to add the CPD back to the video track and dumped the hex from that as well

frame_dump_hex_good.txt

There is a difference in the headers between the two :

Bad :  1F43B675FFE78800000184A11F354DA78100A3010000000000935E81000080
Good : 1F43B675FFE78800000184A1A78429A78100A3010000000000935E81000080
                               ^^^^^^

~I have no clue as to what those 3 bytes encode though~ Ah, they encode a timestamp. Not sure what's going on here then... The hex looks very very similar between the two, mostly timestamp differences as far as I can tell

[mitchellhansen-wyze]

@weichihl This is a very suspicious bit of code :

https://github.com/aws-samples/amazon-kinesis-video-streams-producer-embedded-c/blob/main/src/source/mkv/mkv_generator.c#L420

I moved the

            PUT_UNALIGNED_4_byte_BE(pHeader + MKV_SEGMENT_TRACK_ENTRY_LEN_OFFSET, MKV_LENGTH_INDICATOR_4_BYTE | (uHeaderLen - MKV_SEGMENT_TRACK_ENTRY_HEADER_SIZE));

            *ppBuf = pHeader;
            *pBufLen = uHeaderLen;

down out of the scope of the if (bHasCodecPrivateData) (similar to how the prvCreateAudioTrackEntry works), and the putMedia no longer gives me a 4010 error, but the stream is now unplayable from the play URL that is generated.

Looking at the logs on the cloud end, it's giving me a Missing codec private data in fragment for track 1 error when trying to generate a play URL for this stream. But recall that this same thing worked with the producer-c version of the library...