Repeatedly Calling Seek fails to seek past a certain frame within Seekable Range

[hellovertex]

Hello, I am using mediaPlayer.OpenMedia() on a HLS Url and process each frame individually using mediaPlayer.ExtractFrame() method.

I now want to provide a Rewind button to go back two seconds. In my Update() function, I call mediaPlayer.ExtractFrame(). So when the button is clicked, successive Update() calls should return the video-frames starting from the seeked position two seoncds ago, for further processing.

The issue is with when the button is clicked multiple times: Repeatedly calling Seek(), keeps seeking to almost the same time. So I can not get further past a certain image, even if it is well within seeking range: I expect successive calls to Seek() to "add on to each other" (minus the time passed between two consecutive clicks)

My Setup:

• Unity version: 2020.3.39f1
• AVPro Video version: v2.5.5.f1 core
• Operating system version: Windows 10 Pro for Play Mode (build goes to Android28+, but we are not there yet)
• Device model: Lenovo Thinkpad P15
• Video specs: 1920x1080, 30fps, h264, no audio

This is the code of the corresponding OnClick function that triggers the rewind and produced above screenshot.

private DateTime lastTimeClicked = DateTime.MinValue;
public float jumpDeltaTime = -2f;

public void OnClick() {
    if (lastTimeClicked != DateTime.MinValue)
    {
        var lastClick = DateTime.Now - lastTimeClicked;
            Debug.Log($"Rewind Button Clicked. Last Click was {lastClick.Milliseconds/1000.0} seconds ago");
    }
    // Get Seekable time range
    var timelineRange = Helper.GetTimelineRange(
            _videoInput.mediaPlayer.Info.GetDuration(), 
            _videoInput.mediaPlayer.Control.GetSeekableTimes());

    // Time before
    var t0 = _videoInput.mediaPlayer.Control.GetCurrentTime();

    // Jump 2 seconds
    double time = _videoInput.mediaPlayer.Control.GetCurrentTime() + jumpDeltaTime;
    time = System.Math.Max(time, timelineRange.startTime);
    time = System.Math.Min(time, timelineRange.startTime + timelineRange.duration);
    _videoInput.mediaPlayer.Control.Seek(time);  // SeekToFrameRelative() leads to same issue
Debug.Log($"Seekable Times = {timelineRange.StartTime} to {timelineRange.EndTime}\n" +
          $"Backward Seeked to time: {time}");

    // Time after
    var t1 = _videoInput.mediaPlayer.Control.GetCurrentTime();
Debug.Log($"GetCurrentTime() before Seek: {t0}, GetCurrentTime() after Seek: {t1}");
        lastTimeClicked = DateTime.Now;
}

For completeness the class that calls ExtractsFrame:

// _videoInput: contains media player and triggers frame processing
Texture2D _tex;
private void Start() {
    // ...
    _tex = new Texture2D(1920, 1080);
    _tex.hideFlags = HideFlags.HideAndDontSave;
    mediaPlayer.OpenMedia(new MediaPath(_HLSStreamUrl, MediaPathType.AbsolutePathOrURL));
    mediaPlayer.Control.Play();
    // ...
}

public void Update() {
    // ...
    _tex = mediaPlayer.ExtractFrame(_tex);
    ProcessFrame(_tex)  // Sometimes Slightly slower than 30fps, Synchronous (blocking) --> 3rd party, I cant change it
    // ...
}

Note: I think I can not use mediaPlayer.ExtractFrame(timeSeconds) because on subsequent calls, I wouldnt know timeSeconds-offset, because of the blocking frame processing procedure.

Do you have an idea what might be wrong? Any pointer would be much appreciated.

If it helps, the hls stream is produced via gst-launch-1.0 rtspsrc location=$RTSP_URL short-header=TRUE ! rtph264depay ! video/x-h264, width=1920, height=1080, framerate=30, format=avc,alignment=au ! kvssink stream-name=$STREAM_NAME storage-size=512 access-key=$AWS_ACCESS_KEY_ID secret-key=$AWS_SECRET_ACCESS_KEY aws-region=$AWS_REGION and ffmpeg -re -stream_loop -1 -i $VIDEOFILE_PATH -c copy -f rtsp rtsp://localhost:8554/mystream. I could ask to change these too, if needed.

[hellovertex]

We tried -g 1 flag with ffmpeg, as your documentation on Seeking suggested, to seek directly for iframes, but it would not let us: Codec AVOption g (set the group of picture (GOP) size) specified for input file #0 (FILENAME) is not a decoding option.

[Chris-RH]

Which Video API are you using (Media Foundation, DirectShow or WinRT)? Have you tried different test streams from other sources (https://www.renderheads.com/content/docs/AVProVideo/articles/feature-streaming.html#test-streams)?

[Chris-RH]

Which Video API are you using (Media Foundation, DirectShow or WinRT)? Have you tried different test streams from other sources (https://www.renderheads.com/content/docs/AVProVideo/articles/feature-streaming.html#test-streams)?

[hellovertex]

Sorry for the delay, due to internal requirements, we built an in-house solution using a custom framebuffer that is sufficient for our purposes. In case you might still want the information: We used Media Foundation. If I get the time to test different streams, I will re-open, but that depends on whether we get funding. Thanks for your quick help though!