search

Dash-Industry-Forum / dash.js

A reference client implementation for the playback of MPEG DASH via Javascript and compliant browsers.
http://reference.dashif.org/dash.js/nightly/samples/dash-if-reference-player/index.html
Other
5.15k stars 1.68k forks source link

Captioning support - out-of-band captions #33

Closed wilaw closed 11 years ago

wilaw commented 11 years ago

Support the

  1. Parsing of caption data from the manifest (not 608/708 in-segment data)
  2. Loading and parsing of SMPTE-TT caption files (per DASH-AVC/264 guidelines)
  3. UI component to display and synchronize captions with video.

Per DASH-AVC/264, the captions can be signalled either as a xml file, or a ISOBMFF file. See the table below. We should supoort both types. A test vector forstpp exists at http://dash.edgesuite.net/dash264/TestCases/4b/sony/SNE_DASH_CASE4B_SD_REVISED.mpd .

captions

kirkshoop commented 11 years ago

As discussed in a call I think the default approach should be to send the caption stream into a texttrack StreamBuffer. To support scenarios that will render captions in html/css instead, the same extension model used in the rest of the player should be used.

SorraTheOrc commented 11 years ago

Captioning implemented in the 1.0 release

tieleman commented 9 years ago

Sorry about reviving this old issue, but it doesn't seem to be that the encapsulation in ISOBMFF is currently supported. Can anyone give a comment about that? If I add an ISOBMFF file in my manifest (using MIME-type "application/mp4" and codec "stpp") I don't ever see the player perform a request for it. Is this correct?

As you may know this is part of the EBU-TT-D spec (EBU Tech 3381, Carriage of EBU-TT-D in ISOBMFF) as well. Recently the BBC has shown a first proof-of-concept using this:

http://www.bbc.co.uk/rd/blog/2015/01/subtitles-in-an-ip-world-1 (Sample content here: http://rdmedia.bbc.co.uk/dash/ondemand/elephants_dream)

I'd imagine it would require the use of an ISOBMFF demuxer in the Dash.js project as you cannot simply off-load this to the browser media stack.

jeroenwijering commented 9 years ago

What would be the use case for putting the TTML inside MP4 versus as plain XML?

wilaw commented 9 years ago

Live event captioning?

Irrespective of the use case, we already need to parse the mp4 boxes coming in, in order to extract embedded messages, events and media timestamps which are a core part of the DASH standard. So once we have that parser in place, extracting the TTML is almost a by-product. GPAC submitted c ode for mp4 parsing, as did Fraunhofer I believe.

@Kirk - where are we on evaluating the two parsers and implementing one? Since that gives us event support, I think it should be a high priority for 1.4

Cheers

Will

From: Jeroen Wijering notifications@github.com<mailto:notifications@github.com> Reply-To: "Dash-Industry-Forum/dash.js" reply@reply.github.com<mailto:reply@reply.github.com> Date: Thursday, February 19, 2015 at 9:27 AM To: "Dash-Industry-Forum/dash.js" dash.js@noreply.github.com<mailto:dash.js@noreply.github.com> Cc: Will Law wilaw@akamai.com<mailto:wilaw@akamai.com> Subject: Re: [dash.js] Captioning support - out-of-band captions (#33)

What would be the use case for putting the TTML inside MP4 versus as plain XML?

Reply to this email directly or view it on GitHubhttps://github.com/Dash-Industry-Forum/dash.js/issues/33#issuecomment-75096244.

tieleman commented 9 years ago

Exactly. Also, new standards such as DVB-DASH actively reference EBU-TT-D as a means to provide subtitling using the ISOBMFF approach.

johnluther commented 9 years ago

@tieleman and @wilaw , are you saying that a full-blown MP4/ISOBMFF parser needs to be implemented in the library simply to extract 'emsg' boxes and encapsulated TTML data? If not, my apologies, I'm new here. :-)

If so, @kirkshoop, is code size impact part of the parser evaluation? Any idea how much it might add? Would it be possible to implement only enough code to parse the required atoms and ignore everything else?

JL

wilaw commented 9 years ago

@jluther "are you saying that a full-blown MP4/ISOBMFF parser needs to be implemented in the library simply to extract 'emsg' boxes and encapsulated TTML data? I" No, don't extrapolate "full-blown" out of a need to read certain boxes. The mp4 box/atom structure is relatively simple to walk. We need to only understand and further parse a subset of ftyp that we are interested in.

"Would it be possible to implement only enough code to parse the required atoms and ignore everything else?" That is the intention from the beginning.

Panic ye not :)

Cheers

Will

From: John Luther notifications@github.com<mailto:notifications@github.com> Reply-To: "Dash-Industry-Forum/dash.js" reply@reply.github.com<mailto:reply@reply.github.com> Date: Thursday, February 19, 2015 at 5:05 PM To: "Dash-Industry-Forum/dash.js" dash.js@noreply.github.com<mailto:dash.js@noreply.github.com> Cc: Will Law wilaw@akamai.com<mailto:wilaw@akamai.com> Subject: Re: [dash.js] Captioning support - out-of-band captions (#33)

@tielemanhttps://github.com/tieleman and @wilawhttps://github.com/wilaw , are you saying that a full-blown MP4/ISOBMFF parser needs to be implemented in the library simply to extract 'emsg' boxes and encapsulated TTML data? If not, my apologies, I'm new here. :-)

If so, @kirkshoophttps://github.com/kirkshoop, are code size impact part of the parser evaluation? Any idea how much it might add? Would it be possible to implement only enough code to parse the required atoms and ignore everything else?

JL

Reply to this email directly or view it on GitHubhttps://github.com/Dash-Industry-Forum/dash.js/issues/33#issuecomment-75172324.

kirkshoop commented 9 years ago

Mikhail investigated the two libraries and reported back. Size was a concern. No further action has been taken.

Kirk

Developer Microsoft Open Technologies, Inc.

From: Will Law [mailto:notifications@github.com] Sent: Thursday, February 19, 2015 8:14 PM To: Dash-Industry-Forum/dash.js Cc: Kirk Shoop (MS OPEN TECH) Subject: Re: [dash.js] Captioning support - out-of-band captions (#33)

@jluther "are you saying that a full-blown MP4/ISOBMFF parser needs to be implemented in the library simply to extract 'emsg' boxes and encapsulated TTML data? I" No, don't extrapolate "full-blown" out of a need to read certain boxes. The mp4 box/atom structure is relatively simple to walk. We need to only understand and further parse a subset of ftyp that we are interested in.

"Would it be possible to implement only enough code to parse the required atoms and ignore everything else?" That is the intention from the beginning.

Panic ye not :)

Cheers

Will

From: John Luther notifications@github.com<mailto:notifications@github.com<mailto:notifications@github.com%3cmailto:notifications@github.com>> Reply-To: "Dash-Industry-Forum/dash.js" reply@reply.github.com<mailto:reply@reply.github.com<mailto:reply@reply.github.com%3cmailto:reply@reply.github.com>> Date: Thursday, February 19, 2015 at 5:05 PM To: "Dash-Industry-Forum/dash.js" dash.js@noreply.github.com<mailto:dash.js@noreply.github.com<mailto:dash.js@noreply.github.com%3cmailto:dash.js@noreply.github.com>> Cc: Will Law wilaw@akamai.com<mailto:wilaw@akamai.com<mailto:wilaw@akamai.com%3cmailto:wilaw@akamai.com>> Subject: Re: [dash.js] Captioning support - out-of-band captions (#33)

@tielemanhttps://github.com/tieleman and @wilawhttps://github.com/wilaw , are you saying that a full-blown MP4/ISOBMFF parser needs to be implemented in the library simply to extract 'emsg' boxes and encapsulated TTML data? If not, my apologies, I'm new here. :-)

If so, @kirkshoophttps://github.com/kirkshoop, are code size impact part of the parser evaluation? Any idea how much it might add? Would it be possible to implement only enough code to parse the required atoms and ignore everything else?

JL

Reply to this email directly or view it on GitHubhttps://github.com/Dash-Industry-Forum/dash.js/issues/33#issuecomment-75172324.

— Reply to this email directly or view it on GitHubhttps://github.com/Dash-Industry-Forum/dash.js/issues/33#issuecomment-75187032.

tieleman commented 9 years ago

I understand the size concern. Is the Fraunhofer code publicly available at this time? It might seem feasible to just have a very small generic ISOBMFF box parser (based on ArrayBuffers/DataView) and just build tiny specific parsers for the boxes we are interested in to extract the TTML samples.

For other timed media functionality it could be augmented with different box parsers as well.

There already is some minimal parsing going on in src/streaming/protection/CommonEncryption.js to parse the pssh data. Something similar could be done in a more generic way I think for ISOBMFF.

tieleman commented 9 years ago

Hi @wilaw, @kirkshoop, is there any public reference to the Fraunhofer code, or is it proprietary? Also, should this issue be re-opened, or should there be submitted new issues with regards to captions in ISOBMFF?

squapp commented 9 years ago

@tieleman I believe you are referring to our code to parse 'emsg' boxes. This was implemented to support basic ad-insertion based on DASH-IF guidelines. Let us know if you have any questions