Media source range removal does not stop the playback

[jaroslaw-bojko-red]

Problem: The client reported that the audio language change during playback takes up to 1 minute.

Description: Problem spotted on two different JS players: dash.js and https://github.com/videojs/http-streaming (they use a different streaming dash, hls). Both of the playback using single audiotrack. The language switch is performed by flushing the current buffer and sending samples with a new language with PTS pointed to the current position.

How the switch is performed in those apps:

1. Site removes old language data on the audio sourcebuffer via 4.5.6 Range Removal method, but the operation is asynchronous, so we postpone it.
2. Immediately perform seek to the current position or 0.0 without waiting for data removal. (the second app does not perform any seek, only data removal)
3. Data removal from sourcebuffer is being executed
4. Resume playback while we still have cached audio samples in the buffer.

Play suisse does not wait for Range removal to execute fully. According to html5 standard range removal is asynchronous operation. Play Suisse playback still should be stopped according to 4.5.9 Coded Frame Removal point 5:

_If this object is in activeSourceBuffers, the current playback position is greater than or equal to start and less than the remove end timestamp, and HTMLMediaElement.readyState is greater than HAVE_METADATA, then set the HTMLMediaElement.readyState attribute to HAVEMETADATA and stall playback.

The problem is that we continue the playback even though we have been ordered to remove all samples, for example: Site check current playback position 143829311111ns(143829ms) and removes data removed 384025 bytes, start(136.021333), end(168.021332). We confirmed that the data had been removed. After this site, check if we have buffered 145.152000, and we state yes. Even though a second earlier, we confirmed that we removed data.

Range removal successfully emptied the buffers from MediaSource, but the samples already passed to the decoder remain untouched. The code does not perform gstreamer pipeline flush. That is why the browser continues playback.

Solution: Wpe 2.22:

1. Get SourceBufferPrivate::reenqueueMediaIfNeeded method into wpe 2.22 to perform gstreamer pipeline flush. (from: https://github.com/WebPlatformForEmbedded/WPEWebKit/blob/wpe-2.38/Source/WebCore/platform/graphics/SourceBufferPrivate.cpp
2. Revert commit: commit 8dfa02e0439332db9f79771001fa832ddcd9da3f Author: Enrique Ocaña González eocanha@igalia.com Date: Fri Jul 7 12:18:38 2017 +0000

[GStreamer][MSE] Don't flush audio streams, only video streams

That prevents the audio track from being flushed.

With those changes, the language is switched immediately.

Wpe 2.38: The SourceBufferPrivate::reenqueueMediaIfNeeded is present in the code, and constraints that we cannot flush audio streams have been lifted. Still, the audio language is switched for about a minute. To fix this, we need:

Comment code: // if (!m_playerPrivate.hasAllTracks()) { // GST_DEBUG_OBJECT(m_playerPrivate.pipeline(), "Source element has not emitted tracks yet, so we only need to clear the queue. trackId = '%s'", trackId.string().utf8().data()); // MediaSourceTrackGStreamer* track = m_tracks.get(trackId); // track->clearQueue(); // return; // }

From: SourceBufferPrivateGStreamer.cpp fo perform a full flush of audiotrack and decoder buffers.

We have a question regarding this issue and fixes: Wpe 2.22 - Why commit "Don't flush audio streams, only video streams" explicitly disabled the audiotrack flush? What changed that this constraint has been lifted in new browser? Wpe 2.38 - Why hasAllTracks() does not return true while all tracks and playback are progressing? I do not see the bool variable change to true in the code like there is a lack of implementation. What do you think is the correct solution?

[calvaris]

Would you please provide more information on how to test this? We would need the stream and steps we need to follow with that stream.

[calvaris]

Even better than steps, an HTML file to reproduce and information about the platform you're running on.

[modeveci]

@jaroslaw-bojko-red Could you please provide info regarding questions? Thank you!

[rychem]

@modeveci , @calvaris there is no html file available to show issue reproduction. Issue is visible on BlueTV or PlaySuisse applications. Can the issue be investigated based on real app? On last RDK Browser Working Group the access to real applications as collaboration faciliatation was discussed as one of the point. CC: @oscaruitenbroek

[calvaris]

I don't have access to those apps.

[modeveci]

@rychem I will check with @oscaruitenbroek.

[rychem]

@modeveci , @calvaris we found way to reproduce the issue by using dash.js reference player. Please use dash.js reference player. http://reference.dashif.org/dash.js/latest/samples/multi-audio/multi-audio.html Switching audio tracks from web inspector reproduces the issue document.getElementById('trackSwitchBtn').click() and e.g. document.getElementById('audio-track-listItem_2').click()

However during the verification we found out issue is reproducable only on WPE2.22 (which is not expected to be fixed). WPE2.38 does not reproduce issue anymore (probably after recent branch update). Please use aforementioned method to check 2.38 on your platforms. If also not reproducable, we can close the ticket.

[calvaris]

Now that you managed to reproduce it there, could you please write a test on your own without the need of using the inspector, please?

[calvaris]

Any progress on this? I still need the use case as an HTML that does not need to use the inspector.

[calvaris]

Well, it only happens in 2.22 so it is not expected to be fixed and it works in 2.38 so I am closing this.