Handling missing segments "gracefully"

[orcaman]

Hi there,

Currently, if a segment is missing, when the playback reaches a certain point, it just halts (the buffer controller seems to think there is not enough data buffered to keep going).

This could be handled more nicely I guess, for example, by skipping the interval of the missing data (we know the duration of the missing segment so we should be able to seek to skip it).

This is especially helpful if the last segment is missing. For example, each segment may have a duration of 3 seconds, except for the last one which might be less (for example, 100ms). In this case, we will halt 3 seconds before the video ends, just because we're missing the last 100ms (it would be nicer to just play to end - if the missing segment is the last one, forget about it).

Cheers, Or

[bwidtmann]

Hi @orcaman

I don't really understand 'segment is missing'. But if you have specific trouble at the last seconds of the stream, try this PR #680 maybe it solves your problem.

[orcaman]

Hi @bwidtmann, thanks for the ref to the PR.

I'll try to give an example for what I mean by missing segment.

So for example we have an MP4 file that has been transcoded to DASH, and we now have segments 1-100 in a certain adaptation set. But segment 5 is missing for some reason (because someone deleted it or because there is a bug with the transcoder or because the server gives a 404 over http://ourserver/video/segment4.m4s for some reason).

Let's assume that every segment represents 1 second of video. In this scenario, we can play 99% of the stream. However, because segment 5 is missing, dash.js will play only the first 4 seconds, and then it will get stuck.

My suggestion is that dash.js will skip the missing segments it runs into (for example if the fragment loader gets 404, the segment is missing, skip it). Since we know the duration of each segment, we can seek to the next non-missing one, and thus improve playback in case there are errors.

[dsparacio]

I think the MSE will not move forward if we are missing a segment. I do agree that we need much more comprehensive retry logic for 404. But if a segment is just missing, my understanding is that the MSE source buffer will not decode and render past the missing segment. It must be pushed in at some point to move forward.

[bwidtmann]

@AkamaiDASH MSE will definitely stall and stop playback in case of missing segment. What @orcaman is suggesting is manually seeking to the next buffered range, but I think this leads to unpredictable side effects and breaks user experience.

We had a similar use case when there were broken chunks in certain quality levels caused by transcoding. Our solution was decreasing quality level when there was a 4xx or 5xx error and try to reload the same segment with lower quality again.

You can see this on another player: https://github.com/Orange-OpenSource/hasplayer.js/pull/11

I am working on a PR to add this fallback functionality for dash.js as well.

[dsparacio]

Ah missed the skip and seek. That is what our best effort fetch logic does for Akamai Flash HDS. If 4/5XX status code then try all different quality at same idx , if still no good response try next idx at current quality up to 10 indices forward before giving up.

[dsparacio]

More info on the feature https://github.com/Dash-Industry-Forum/dash.js/issues/1211

[sandersaares]

Ah missed the skip and seek. That is what our best effort fetch logic does for Akamai Flash HDS. If 4/5XX status code then try all different quality at same idx , if still no good response try next idx at current quality up to 10 indices forward before giving up.

I would just like to confirm that such behavior would be perfect for us when audio or video is concerned. For noncritical adaptation set types (e.g. text), ignoring missing segments entirely would be preferable.

[Victordmdb]

Hi guys! I was wondering if there was an update on this? Or maybe a common workaround? I'm working with gpac Dashcast to product a live stream, and it sometimes skips segments, which makes dashjs stop.

[dsparacio]

@Victordmdb There was not time to get that feature in and I am very eager to do this so I promise it will be part of 2.4. I have updated the tags. Are you interesting is participating in the design ?

[dsparacio]

More info on request https://github.com/Dash-Industry-Forum/dash.js/issues/1570

[Victordmdb]

@AkamaiDASH Sure, I could give it a try, but I'm no great js programmer.

[dsparacio]

@Victordmdb I was meaning design as in talking over how the feature should work and what use cases to program for etc. I can do the programming or share the work with others/you if that is also something you are able to contribute. I just want to avoid going off in a corner and doing the work.

[Victordmdb]

@AkamaiDASH Oh right, then yes of course! Just let me know what you need. I don't think I'll much be able to contribute code-wise though.

[dsparacio]

Another duplicate with some more important info #1310

[ThiagoMiranda]

@AkamaiDASH Is there any workaround for this? I´ve tried to switch the quality ( using the setQualityFor('video', e.event.request.quality - 1) ) on the error listener but it didn´t work.

[dsparacio]

I am going to to start work on this and wanted to see if anyone is interested in helping with the plan. I will write up a spec for implementation then submit it here for review.

Does anyone have an MPD where this is the case. I will create something otherwise by deleting a segment for a static presentation.

@TobbeEdgeware how hard would it be to set up a live sim scenario with one missing segment every minutes or at some interval?

[davemevans]

I think we need to be extremely careful with the design of this feature.

In the static case, skipping segments seems like a passable approach, but I do agree with @bwidtmann that switching representation would give significantly better UX.

In the dynamic case, I don't think this is a particularly good solution, in the first instance at least. If a segment is unavailable, you need to take into account that the player could have either a) miscalculated the segment availability window and is ahead of itself, either because it was wrong to start with, or the clock has drifted or b) failed to update the manifest after it has changed, for whatever reason. So the first things to try should be to resynchronise the time, and a reload of the manifest.

Regardless of live or on-demand, in the presence of alternate BaseURLs, an endpoint switch should always be tried first.

On a related note, currently the retry interval defaults to 500ms * 4 retries (though both of these are configurable). I would imagine a sensible CDN configuration would cache 404s for a few seconds which renders the default configuration mostly useless.

That said, simply hammering the endpoint with retries isn’t going to make the segment appear and a large player population behaving in this way could cause further issues. In an on-demand presentation you can assume it will never appear and not bother retrying that exact URL at all. If it’s a live presentation and the player has resynchronised its clock and reloaded the manifest and the segment still isn’t available, the same probably applies since it’s most likely either some upstream element has failed or the presentation is misconfigured and the segment will never appear.

Just my opinion, of course ...

[sandersaares]

That does indeed sound like a good set of tricky aspects all mixed up together! An intimidating topic!

I imagine this is an area that could also deserve documenting further in IOP, so perhaps some set of guidelines & a reference with regard to client behavior expectations could also be created and integrated into IOP as part of this dash.js work?

[davemevans]

That sounds sensible. The DVB DASH profile has some good stuff around player resiliency - see section 10.8

[TobbeEdgeware]

Regarding using multiple baseURLs for reliability, the Nomor team created a mode of dash-live-source-simulator with 2 different baseURLs which are fluctuating in reliability, see https://github.com/Dash-Industry-Forum/dash-live-source-simulator/wiki/Usage-for-BaseURL

In general, it should be fairly easy to add a parameter to turn on emulating that a segment is missing from one representation of a live stream but not from others.

[dsparacio]

@TobbeEdgeware Great info for baseURL test stream thanks!.

In general, it should be fairly easy to add a parameter to turn on emulating that a segment is missing from one representation of a live stream but not from others.

This would be so great to dev with. Any way to set it up the same for both dynamic and static MPD?

[TobbeEdgeware]

The live-simulator only outputs dynamic MPDs at the moment. The static MPD links I have are just to static content. That said, one could make a vod simulator which also does some manipulation. I will probably not have time to implement something like that in the near future.

[Victordmdb]

Signals has an app called dashcastx which allows one to generate a live mpeg dash stream. I think it would be possible to increment the segment index randomly or "ignore" a segment, but would need someone who understands the code more thoroughly.

[dsparacio]

The live-simulator only outputs dynamic MPDs at the moment.

Fair enough. I think for static it is easy for me to corrupt a few segments in any rep so I can set that test stream up soon. The live scenario with missing seg would be great to have.

[sandersaares]

This topic is also being looked into by Shaka Player developers FYI, linked above.

[dsparacio]

https://docs.google.com/document/d/1jF6tZa23cDraIOQrWG4Tel4V9J9JFq6TunYqBe-XWbY/edit?usp=sharing

OK I have attempted to put together a draft spec for this issue. . I am trying to outline all the stuff we should consider and how to solve sub-use-cases. The PLAN section is in progress and will fill in as we discuss the issues. On that note I would like to setup a call where key contributors can discuss this over the phone. I will be setting up a poll today for Dates / Time that work for all of us.

@sandersaares I do not have your email but I want to add you to the doc to edit. let me know if you would like that access or if comments/suggestions are enough.

[sandersaares]

I will message you my email address on Slack.

[dsparacio]

If you are interested in openly reviewing the spec to finalized it so work can start then please pick the best time and date for you offered on this poll http://doodle.com/poll/ut4fmbmw42qxxkp3

Thanks for your participation!

[charlielye]

Does anyone have a suggestion, or a patch I can hack in, that could enable the player to try the next best representation in the event of a 404? In my setup there will always be at least one representation on the server, and I want the player to make a best effort to not stall...

[mwexpway]

I am facing the same issue : dash.js stucks as soon as one segment is missing. Does anyone have a workaround or a patch to integrate dash.js code in order to skip a missing segment and continue to request the next ones?

[bbert]

Please check PR #2539 that should resolve this issue.

[mwexpway]

I tried the proposed correction #2539 but it is not sufficient to resolve the issue about missing segment. Additionnaly to clearing the URL blacklist as in #2539, I also commented the line which add the URL to blacklist :

        if (e.error) {
            if (e.request.mediaType === _constantsConstants2['default'].AUDIO || e.request.mediaType === _constantsConstants2['default'].VIDEO) {
                // add service location to blacklist controller - only for audio or video. text should not set errors
                //eventBus.trigger(_coreEventsEvents2['default'].SERVICE_LOCATION_BLACKLIST_ADD, { entry: e.request.serviceLocation });
            }
        }

Then on application side I added a callback function notified on error to restart playing the MPD URL :

function cb_error(e)
{
    if (e.error=='download' && e.event && e.event.id=='content')
    {
        // Try to reload URL
        mediaplayer.attachSource(playing_url);
    }
}

mediaplayer = dashjs.MediaPlayer().create();
mediaplayer.on(dashjs.MediaPlayer.events.ERROR, cb_error, null);

It seems to be working: when a segment is missing, the video starts again. The workaround is on the application side, and requires disabling the URL blacklisting. Maybe it would be better to make the correction in the DASH player, this would avoid parsing the MPD again.

[ChrisMash]

I'm seeing this issue on a live stream that's been paused for a while (dash 2.6.6)

[Sureshyvn]

Dear @mwexpway function cb_error(e) {............................................ in this method there is a parameter 'e' so what is 'e' here that you are passing???

[mwexpway]

The e in the function definition is a parameter. This function is a callback, called by the DASH.js which defines the argument value for e.

DASH js is not well documented. I based the code on first answer to question here: https://stackoverflow.com/questions/35631329/dash-js-trapping-404-errors-from-mpeg-dash-player

[yogevNisim]

I'm having the same problem here for 2.7.0.

Is there any solution for that?

[amitmiz]

after investigation I made on my live stream, the player seems to stop because of the following scenario

SEGMENT 111 TRY DOWNLOAD
SEGMENT 111 RECIEVE 404
DASH JS RETRY 111
SEGMENT 112 TRY DOWNLOAD
SEGMENT 112 DOWNLOADED AND APPEND TO MSE BUFFER
SEGMENT 111 DOWNLOADED AND APPEND TO MSE BUFFER

idk if this is what everyone here expirencing but it seems that this is what causing the player to halt at least for me @yogevNisim

[yogevNisim]

@amitmiz yes, that seems to be the same problem here.

any solution found for that?

[timur-gilauri]

I am having this issue when stream video as VOD. So there is finite number of fragments to download, but the player know nothing about it and keep trying to download fragments that are beyond the boundaries, gets 404 and then crush.

As I know the duration of the video, I can count amount of downloaded fragments, depends on start time and if next fragment is not needed I reset request. It is prevent a player crush but cause some other problems.

event.sender.abortRequests()
event.sender.reset()

It could be a good feature if we can manually set a video duration, so player can calculate how many fragments it has to download, and trigger 'ended' event, when it reaches the last one.

[sandersaares]

but the player know nothing about it

Why not? The MPD should have information about the duration of the video. Is this not the case? Can you show an example of the video you use?

[timur-gilauri]

@sandersaares there is dynamic manifest, that does not contains info neither about duration of the video or amount of segments, but only has information about segment duration. Even limited time video is stream as live. In this case there is no difference. And it is no way to change a giving manifest.

So, it would be nice feature to manually set a video duration and let player know how many segments it should download.

[sandersaares]

From this description, it sounds to me like your manifest is not valid DASH then. When a dynamic MPD stops getting new data, the server must update the MPD to say so - by defining a Period@duration and MPD@mediaPresentationDuration that limits it to the data that really exists.

The MPD is the authority - if it says a segment is available, it must be available.

[dsilhavy]

@amitmiz @timur-gilauri : Do you still have problems with your streams in the latest version of the player: http://reference.dashif.org/dash.js/v3.0.1/samples/dash-if-reference-player/index.html

Please also validate that your streams are correct: https://conformance.dashif.org/

I will leave this issue open for 3 days and close it if there is no feedback by then