RTSP: Streaming and Player Improvements.

[MrArca9]

So I have some code I have merged into the library and I wanted to share it in hopes of being able to have smoother playback.

Let's start with the chunk from FFmpeg.Media (Keep in mind this is all lazy man coding)

private void ExtractMediaFramesContinuously() {
  decimal renderTime = StartTime;
  decimal lastTime = 0;
  bool internalMute = false;
  decimal oVolume = this.Volume;
  bool retrying = false;
  while (IsCancellationPending == false) {
   var wasPlaying = this.IsPlaying;
   if (lastTime - (Decimal) 5 > 0) {
    var frame = AudioFramesCache.GetFrame(lastTime - (Decimal) 0.05, CheckFrameBounds);
    var frame0 = AudioFramesCache.GetFrame(renderTime, CheckFrameBounds);
    if (frame0 == frame) {
     Debug.WriteLine("REPEATING AUDIO FRAMES!!");
     internalMute = true;
     if (this.Volume != 0) {
      oVolume = this.Volume;
     }
     this.Mute();
    } else if (internalMute == true) {
     Debug.WriteLine("NONREPEATING AUDIO FRAMES!!");
     internalMute = false;
     this.Volume = 1;
    }
   }
   //        // Lock up changes
   MediaFramesExtractedDone.Reset();
   //
   //        // Extract state
   lastTime = renderTime;
   renderTime = RealtimeClock.PositionSeconds;
   //
   //  Load frames
   //  Debug.WriteLine(lastTime);

   // Debug.WriteLine(renderTime);                                          

   InternalLoadFrames(renderTime);
   //
   //        // Unlock
   MediaFramesExtractedDone.Set();
   //

   if (wasPlaying && this.IsPlaying == false) {
    Debug.WriteLine("ATTEMPTING TO PLAY AGAIN");
    this.Play();
    retrying = true;
   }
   if (lastTime == renderTime && lastTime > 10 && retrying == false) {
    Debug.WriteLine("REPEATING FRAMES!!");
    this.Play();
   }
   //            // HasMediaEnded will most likely contain an "old value" for the current cycle. That's why we call the method to re-evaluate.
   //            if (InternalGetHasMediaEnded() == false)
   //            {
   //              ErrorOccurredCallback(this, new MediaPlaybackException(MediaPlaybackErrorSources.ExtractMediaFramesContinuously, MediaPlaybackErrorCode.FrameExtractionLoopForcedPause,
   //                   string.Format("WARNING: Something did not go smoothly. Wall clock paused @ {0:0.000} Call the Play method to resume playback.",
   //                        renderTime)));
   //                //Attempt To Start Playing Again
   //                this.Play();
   //            }
   //        }
   //
   //        // give waiter methods a chance to execute before a new lock is set.
   retrying = false;
   Thread.Sleep(1);

  }

The main changes here is fixing the audio stutter that happens when the player tries to repeat a frame in the queue. It tries to set the volume to 0 then return back to what ever the user had before setting it to 0. It works alright, has some delay before kicking in.

The other changes is the player will try to "replay" itself if it becomes desynced rather than throwing an error. This is a personal change to try and keep the video player from crashing all the time with the streams I use. As for some reason after about 2-3 mins it becomes desynced quite often.

This next part is very important and should be added ASAP for m3u8 links.

FFmpegMedia.Decoding.cs line 104 ffmpeg.av_dict_set_int(&optionsDict, "multiple_requests", 1, 0); Without this the player WILL NOT have continues playback on m3u8 files. This should be a toggle option within the interface for the player because I can understand some users not needing it but for my personal application without this line I can never have continuous playback from a streamed m3u8 file.

The below lines are icing and should help a little. ffmpeg.av_dict_set_int(&optionsDict, "reconnect", 1, 0); ffmpeg.av_dict_set_int(&optionsDict, "reconnect_at_eof", 1, 0); ffmpeg.av_dict_set_int(&optionsDict, "reconnect_streamed", 1, 0); ffmpeg.av_dict_set_int(&optionsDict, "reconnect_delay_max", 200, 0);

My main issue here is trying to buffer some video into ffmpeg before playing so it is overall smoother and more reliable. I have yet to figure out how to do so. If you could provide any assistance that would be amazing. We (My team and I) Have decided to use this project because it loads and offloads video streams SOOOO much faster than VLC for .Net and load times are very important to us. We just need an increase in stream reliability and stability. (Unfortunately they left me to do all this)

Also if you should add anything from my personal changes, I would add in the audio repeater identifier and muter and the multiple connections option.

[mariodivece]

THANK YOU!!! m3u8 DOES in fact work much better with theses params! Very nice contributions. I will see how to avoid re-rendering repeated audio frames in a more encapsulated manner.

[mariodivece]

For reference -- Test HLS streams http://g33ktricks.blogspot.mx/2016/04/list-of-hls-streaming-video-sample-test.html

[MrArca9]

That's great to hear. Like I said before this is an absolutely awesome project for our team and I am super excited to see how well we can get this to work.

If I stumble across anything else I will let you know.

My primary issue right now is the desnycing of the player and the clock? Could also just be the stream source. Might have to step in and manually manipulate the stream to smooth it out (IE buffering when needed ect)

if we could somehow pool the information into a memory stream and play using that memory stream do you think we could effectively "buffer" by letting the stream fill up with more information before playing it?

I know very little about ffmpeg and video libs(All this was from massive amounts of searching) so if we could figure out a way to buffer incoming streams I would be super happy :)

[mariodivece]

For reference, I need to look at the synchronization code of ffplay and see how ffmpeg devs do it. The below is coming from issue #12

I still have not looked very closely at streaming scenarios -- see my code comments regarding media buffering events -- but it seems they require special handling as ffplay does it https://www.ffmpeg.org/doxygen/3.0/ffplay_8c_source.html#l03009 I will try to take a look at this in the following days. In the meantime it would be helpful if you could get me a couple more streaming samples... maybe some hls and rtsp samples that work corrctly with ffplay would be ideal for testing.

[mariodivece]

@9Arca9 -- In fact the AudioFramesCache and the VideoFramesCache are exactly meant for that. I need tor eview the code for ffplay and see how they do it. It just takes some time...

[MrArca9]

That sounds great! Don't mind how long it takes as long as we know we can do it.

yea the documentation on this is rather limited so I had to guess most of the changes I made and hope it works.

I will also look into that and post if i have any solutions.

My suggestion, just off the top of my head, is to pause playback if we reach below the allowed threshold in the cache, for instance:

buffer x amount of data in cache. Start playback once buffer is filled, Should buffer drop below y% pause(?) playback until buffer reaches x again,

Something like that.

[MrArca9]

Points of interest in FFPlay's page.

3.5 Advanced options -framedrop Drop video frames if video is out of sync. Enabled by default if the master clock is not set to video. Use this option to enable frame dropping for all master clock sources, use -noframedrop to disable it.

(We should play around with setting. Setting this to video when doing HLS/RTP streams)

-infbuf Do not limit the input buffer size, read as much data as possible from the input as soon as possible. Enabled by default for realtime streams, where data may be dropped if not read in time. Use this option to enable infinite buffers for all inputs, use -noinfbuf to disable it.

(Curious to see if this is enabled, could help with buffering)

[MrArca9]

These options might have an impact. Not thoroughly tested though.

ffmpeg.av_dict_set(&optionsDict, "sync", "video", 0);
ffmpeg.av_dict_set_int(&optionsDict, "framedrop", 1, 0);

EDIT: Nope, don't seem to do anything.

Using nasa m3u8 link for testing. Still having issues with frame freezing and locking. http://nasatv-lh.akamaihd.net/i/NASA_103@319271/master.m3u8

[MrArca9]

Is it possible to change the capacity of the buffer to allow more information to be stored inside so that the player can be smoother?

I am digging around and looking at FFmpegMediaFrameCache seeing if i can increase from 60

If i can increase the buffer and read when the buffer is no longer getting information in I might be able to react in code.

What I am ultimately hoping for is: 1) Access to max buffer size to change how much we can store before playing the video. (Higher storage = more ram usage but streaming video should be smoother) 2) An event that is fired off when the video has reached the end of the cache and now has to "Buffer" that is to allow the cache to reach max buffer size before resuming,

Hopefully i am making sense.

[DuHastMeinSkill]

This wasn't working at all for RTSP streaming. It would constantly stutter and freeze for long periods of time. There are a few issues. The first is that for streaming video "av_read_frame" and "avcodec_decode_video2" can take a long time. The way you are doing thread synchronization is causing drawing to wait for very long times when either of those methods takes a long time to return. This causes the video to stutter, and frames will be removed from the cache without having ever been drawn. Your thread synchronization needs to only wait at critical sections instead of doing a blanket wait. It would probably be best to refactor the code so it is only using the frame cache at the beginning and end of the loop when it prepares to fill it and then when it adds the frames to fill it.

If you fix the above, the other issue is you are removing the first frame in each loop, and then loading a single frame. If the loop takes a long time because either of the above functions took a long time the video will stutter. Now for a live stream you can pretty much just draw the frame and then throw it out. However, obviously for a video file a different method is needed. You will probably need an update thread running at the desired framerate of your video that manages time synchronization and removing frames, and a background thread that is just always filling the cache if it isn't full.

[mariodivece]

Hey @Wizzardman thanks for your comments! All Pull Requests are welcome!

[mariodivece]

The new decoding login is able to handle RTSP streaming very well. Could you please provide with some sample streams?

[mariodivece]

I am closing this issue now because the new decoding logic is handling rendering very efficiently and issue no longer applies. Also, there was no further feedback. Closing.