MushMal
commented
3 years ago @wlinInspire the code in frame order coordinator is very involved. It has to handle multiple edge cases to "massage" the frame timestamps in the right order in a streamable fashion and handle different track frame timestamp collisions and handle A/V sync issue. This is all done so we can have a streamable multi-track MKV which is syntactically correct and semantically valid. The frame order coordinator can indeed modify the timestamps of the frames in order to handle frames from different tracks. The default mode in Producer C for multi-track Audio Video is FRAME_ORDERING_MODE_MULTI_TRACK_AV_COMPARE_PTS_ONE_MS_COMPENSATE which will compare and reorder frames according to the presentation timestamp. The reordering of the frames between different tracks is done so they can be monotonically increasing (across the tracks) as long as they are in decode order (MKV requirement) and the timestamps are self-contained. This works OK for most of the pipelines but for some reason for your pipeline it seems that the DTS will cause an issue.
Can you dump out the pts and dts of the frames that cause this issue? I would expect that for your case the pts and dts of the frames should be the same. We have a certain handling of the pts/dts in the GStreamer plugin to adapt for different behaviors for different pipelines. Looking at the pts and dts of the frames, it's likely that the pts is not "properly" set.
There is also a discussion in here: https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp/issues/594
wlinInspire
Hi KVS team,
I am streaming RTSP video and audio to KVS using GStreamer plugin built from the master branch. I see "putKinesisVideoFrame(): Failed to submit frame to Kinesis Video client" with error code
0x30000005(STATUS_CONTENT_VIEW_INVALID_TIMESTAMP). I added more logging and found that although frame DTS is indeed monotonically increasing inside frameOrderCoordinatorPutFrame(), it becomes out of order inside putFrame() sometimes. Below is the sample debugging info and you can see thatputFrame(): putting Frame DTS: 29490462comes beforeputFrame(): putting Frame DTS: 29489342, which should be the other way around. (Note the warning message is not referring to the frame DTS causing the problem)The logging code I added is as below at this
and this
After some debugging, I think the problem is caused by this .
The default frame ordering mode is
FRAME_ORDERING_MODE_MULTI_TRACK_AV_COMPARE_PTS_ONE_MS_COMPENSATE, so the code will compare PTS instead of DTS. If I force to compare DTS, the error 0x30000005 is gone and frames are put in the order of DTS. However, new errors come out from backend with error code0x52000060(STATUS_ACK_ERR_FRAGMENT_TIMECODE_NOT_MONOTONIC)Is the backend comparing PTS instead of DTS? Is there any more debugging I can do?