search

awslabs / amazon-kinesis-video-streams-producer-sdk-cpp

Amazon Kinesis Video Streams Producer SDK for C++ is for developers to install and customize for their connected camera and other devices to securely stream video, audio, and time-encoded data to Kinesis Video Streams.
Apache License 2.0
379 stars 334 forks source link

HLS stream not consistent #441

Closed js1972 closed 4 years ago

js1972 commented 4 years ago

I am using the kvssink to send a camera stream to kinesis video. The camera is set to 10fps and i-frame interval of 10.

I have a mobile app build in Flutter. The video player is a simple wrapper around iOS AVPlayer. So in the app I get the HLS Streaming URL successfully every time and pass it to the video player. About 1 in every 2 or 3 tries, the video player just shows a white screen and nothing happens. If I exit out and try again it will work and play a stable stream. Every so often I also see the following errors message on ios:

Failed to load video: The operation couldn't be completed.
CoreMediaErrorDomain error -12312 - Media Entry discontinuity value does not match previous playlist for MEDIA-SEQUENCE 0

Here is the gstreamer pipeline I am using to send to Kinesis:

gst-launch-1.0 rtspsrc location={} short-header=true ! rtph264depay ! video/x-h264, format=avc, alignment=au ! h264parse ! kvssink stream-name={}-{} storage-size=128 retention-period=1 max-latency=60 framerate=10 key-frame-fragmentation=false fragment-duration=2000"

I'm dynamically building the location and stream name above as you can see with "{}".

By adding framerate=10 above to match my cameras and also forcing a fragment duration seems to help. Instead of seeing the errors roughly 50% of the time. I no only see them maybe once in 5 times.

Note that if I use the AWS Kinesis Video preview it works fine all the time.

Is there any advice on how to set the kvssink parameters to get stable HLS streaming on iOS ?

ps. The same issues occurs on Android - which makes me think its more of a kinesis thing that an ios/android thing. ;-)

MixMasterMitch commented 4 years ago

It is possible that the parameters used to create the HLS session are to blame, not the gstreamer parameters.

Are you using SERVER_TIMESTAMP or PRODUCER_TIMESTAMP mode? You probably want PRODUCER_TIMESTAMP.

And what DiscontinuityMode are you using? You probably want ON_DISCONTINUITY.

See https://docs.aws.amazon.com/kinesisvideostreams/latest/dg/API_reader_GetHLSStreamingSessionURL.html

js1972 commented 4 years ago

I have tried setting those values and there is an improvement but I still get the issue every so often. It's hard to say how often. As it will fail a couple of times, then work the next 9 or 10 times in a row.

Interestingly it seems to be the first time I look at a stream in a while. If I stop then start it again it works. Here's my settings from the getHLSStreamingSessionURL call:

         'StreamName': streamName,
          'PlaybackMode': 'LIVE',
          'DiscontinuityMode': 'ON_DISCONTINUITY',
          'HLSFragmentSelector': {
            'FragmentSelectorType': 'PRODUCER_TIMESTAMP',
          },

On the kvssink side. Would max-latency=60 and fragment-duration=2000 have anything to do with it? If I remove key-frame-fragmentation=false fragment-duration=2000 it definitely performs more poorly. As in - the streaming client wont work maybe 30% of the time.

Thanks for the help.

Maybe I need to reload the entire player in iOS when its not streaming... Kind of a brute-force retry mechanism.

chehefen commented 4 years ago

@js1972, can you export this env variable KVS_DEBUG_DUMP_DATA_FILE_DIR=$PWD, and then run this pipeline? 

gst-launch-1.0 rtspsrc location={} short-header=true ! rtph264depay ! video/x-h264, format=avc, alignment=au ! h264parse ! kvssink stream-name={}-{} storage-size=128 retention-period=1"

it will dump a mkv file in $PWD, you can send that file to us at kinesis-video-support@amazon.com. Thanks.

MushMal commented 4 years ago

@js1972 do you have any update on this?

js1972 commented 4 years ago

Sorry have not got back to this yet. Will do now...

So what happens with the improved parameters above is that generally - the first time I try to watch a stream from the hls client on iOS it fails and I just see a blank screen with no error messages. So I can out and go in to watch again and it will work - over and over. So at most if fails one time for each camera stream I may look at.

js1972 commented 4 years ago

Have sent the recorded file to: kinesis-video-support@amazon.com

MushMal commented 4 years ago

@js1972 thanks for the clip. Looking at the clip I don't see anything that could cause the issue. The frames look normal. The first cluster has a single frame with size 119 - not yet sure whether this could cause the problem.

MushMal commented 4 years ago

One more thing...

2020-05-20 12:12:00 [547381821936] ERROR - getKinesisVideoStreamData(): operation returned status code: 0x5200002f

this is not an error, it's getting printed as an error (which is fixed in PIC since) but it's one of the control statuses that's being passed from the lower layer to indicate no more data available to the networking layer. You can ignore that.

MixMasterMitch commented 4 years ago

@js1972 I have looked at the provided MKV file with mkvinfo and I see the following at the start of the file:

|+ Cluster
| + Cluster timestamp: 441652:11:38.302000000
| + Cluster position: 0
| + Simple block: track number 1, 1 frame(s), timestamp 441652:11:38.302000000
|  + Frame with size 119
|+ Cluster
| + Cluster timestamp: 441652:11:38.381000000
| + Cluster position: 0
| + Simple block: key, track number 1, 1 frame(s), timestamp 441652:11:38.381000000
|  + Frame with size 33535
| + Simple block: track number 1, 1 frame(s), timestamp 441652:11:38.497000000
|  + Frame with size 243
| + Simple block: track number 1, 1 frame(s), timestamp 441652:11:38.571000000
|  + Frame with size 82
| + Simple block: track number 1, 1 frame(s), timestamp 441652:11:38.681000000
|  + Frame with size 49
| + Simple block: track number 1, 1 frame(s), timestamp 441652:11:38.748000000
...

The first cluster is starting with a single non-keyframe. Most video players will fail if they receive a cluster like this. Perhaps this is just a result of the KVS_DEBUG_DUMP_DATA_FILE_DIR flag, but I don't think so. You need to make sure you start pulling from the rtsp source starting from a keyframe. Perhaps @MushMal can provide some guidance on how to do that with gstreamer.

js1972 commented 4 years ago

Would love to know how to start it on a key frame. Will start researching. The interesting thing is that the hls client works perfectly a lot of the time.

js1972 commented 4 years ago

What is it in mkvinfo that tells you the file does not start with a key frame?

chehefen commented 4 years ago

Would love to know how to start it on a key frame. Will start researching. The interesting thing is that the hls client works perfectly a lot of the time.

you can do it by setting this to TRUE: https://github.com/awslabs/amazon-kinesis-video-streams-pic/blob/8f0db7900d6e847ae8072ae5e8205ef81551c685/src/client/src/Stream.c#L127

the package is in the submodule.

js1972 commented 4 years ago

Setting this to true: pKinesisVideoStream->skipNonKeyFrames = FALSE; Looks like it will skip ALL non-key frames?? Thats not what we want is it? We just want to start the stream with a key frame.

MushMal commented 4 years ago

@js1972 it will skip the non-key frames on startup. This is an internal state variable that's actually used for other purposes. We just want to see whether this is the root cause of the issue you are seeing.

js1972 commented 4 years ago

oh ok. So if I change that I need to build the sources again I assume. I have tried setting the kvssink parameter key-frame-fragmentation true but when I do this I cannot stream from hls at all. Let me change the source and I'll come back.

MushMal commented 4 years ago

Resolving due to staleness. Please reopen or cut a new issue with more details if needed

js1972 commented 4 years ago

@MushMal Just wanted to let you know tat I finally got around to testing the advice to change the source and set:

you can do it by setting this to TRUE: https://github.com/awslabs/amazon-kinesis-video-streams-pic/blob/8f0db7900d6e847ae8072ae5e8205ef81551c685/src/client/src/Stream.c#L127

It works. On iOS the stream now works first time every time. Before setting this var to TRUE it would only work on iOS on the second attempt which was quite annoying.

Note that it always worked on Android though - so Android must be more tolerant of the first frame being a key frame or not.

You mentioned this was just a test. Can I leave it like this or would you now push a change?

MushMal commented 4 years ago

Interesting.. I would imagine that the media pipeline/application would vet out the "bad" frames that would come in first but perhaps initializing the skip indicator to true and skipping all non-key-frames initially is a good default.

In order to land this change we need to:

MushMal commented 4 years ago

@js1972 John, could I ask you to attach the first frame (likely an SEI NALu) or the MKV that contains to the GitHub issue so others can look into it as well? I would like to understand what the bits are and whether they might have any downstream effect?

js1972 commented 4 years ago

Hi @MushMal is it possible to re-open this incident while the testing is taking place. Also your previous comment - I can't tell if that is asking me to do something or someone called "John" ?

MushMal commented 4 years ago

Yeah, of course I will reopen the issue. Sorry, for some odd reason I must have typed the GitHub alias and then added John :) Sorry Jason. Please help us get that first frame so we can see what that is. Blank removing it might break some of the analytics or playback engines down stream. We are trying to see whether removing non-key-frame vs adapting the media pipeline is the best.

js1972 commented 4 years ago

How do I get this for you... Save a new mkv file like from here : https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp/issues/441#issuecomment-630437212

MushMal commented 4 years ago

Yeah, that will work. We don't preserve the earlier sent files. I can then extract the frame directly from the MKV

js1972 commented 4 years ago

Ok I have resent you the mkv file that I previously sent above to this email kinesis-video-support@amazon.com. This is the file that shows that the stream starts without a keyframe.

MushMal commented 4 years ago

Perfect, thank you. Let me take a closer look and I will report back

js1972 commented 4 years ago

Hi @MushMal any news on this?

MushMal commented 4 years ago

Sorry, this has been dropped from the list. Will hopefully look at it at some stage

MushMal commented 4 years ago

OK, sorry this was not prioritized somewhat.

Here are the bits of the frame I see:

00 00 00 73 61 e0 00 a0 01 45 71 1f 00 da be 54 b0 13 96 48 80 5b 11 9b d6 01 1e 12 96 3c 5b b0 5f ae dd c6 b4 08 8b 7c 1b 47 0a a8 38 71 9a 28 91 06 c9 44 04 c9 8a 0e e2 4c 1a a9 96 2b 44 f1 21 47 64 ce ee 8b 68 f5 7a c6 17 88 17 59 6f dc 75 66 08 7d 0d 4c e0 2d d1 b0 d2 49 c0 be 0e b6 02 fd 18 8c a8 be 4c b0 c8 16 a1 d2 38 68 45 77 17 24 7a d9 64 c7 e0

The frame is properly packaged but I just can't tell what exactly it represents as its NALu type is 1 which is a Coded slice of a non-IDR picture VCL. I don't have a way of knowing what the encoder has produced. Perhaps you could check the manufacturer spec of your rtsp source.

This said, I think we should not process and skip first non-key frames: https://github.com/awslabs/amazon-kinesis-video-streams-pic/issues/76

MushMal commented 4 years ago

I am going to resolve this issue for now as there is a mitigation and we are still unsure what's the NALu type. We are tracking the skipping non-key-frame NALus in a separate issue: https://github.com/awslabs/amazon-kinesis-video-streams-pic/issues/76

MushMal commented 4 years ago

The changes are in PIC. Those will trickle to C Producer and CPP Producer SDKs but if anyone needs them ASAP then they can reference this commit: 1988f99fe77a3b37e9ed33643c20190a5c2e1689

js1972 commented 4 years ago

@MushMal is there a way for me to tell when theses changes will filter through to master?

MushMal commented 4 years ago

We started migration of the changes. Currently, they are in C Producer: https://github.com/awslabs/amazon-kinesis-video-streams-producer-c/commit/81305d2796b5d571a8c19edc59c0463975f97c8c

Hope to move this tomorrow to cpp

js1972 commented 4 years ago

Thankyou. pKinesisVideoStream->skipNonKeyFrames = FALSE is now set to TRUE when I clone master and build.

However I have also found that streaming via HLS to an iOS device is still very flakey unless I also set this property on kvssink:

key-frame-fragmentation=false

This value defaults to true. So it looks like that even with pKinesisVideoStream->skipNonKeyFrames = TRUE you also need key-frame-fragmentation=false to ensure streams start with a key frame...

MushMal commented 4 years ago

Key frame fragmentation drives an entirely different aspect of the SDK. If it specified as false, the SDK will cut a fragment on a key-frame only after a fragment-duration has passed. This is useful for scenarios like most of the audio encoders where every frame is like an I-frame but we need the SDK to not cut a fragment with every frame. In your case I would recommend understanding how your encoder behaves as for a normal video case the encoder should drive the fragmentation