search

CCExtractor / ccextractor

CCExtractor - Official version maintained by the core team
https://www.ccextractor.org
GNU General Public License v2.0
708 stars 422 forks source link

[Proposal] Adding an option to enable/disable X-TIMESTAMP-MAP in WebVTT #1127

Open ozt860071 opened 4 years ago

ozt860071 commented 4 years ago

Please prefix your issue with one of the following: [BUG], [PROPOSAL], [QUESTION].

CCExtractor version (using the --version parameter preferably) : X.X

In raising this issue, I confirm the following (please check boxes, eg [X] - and delete unchecked ones):

My familiarity with the project is as follows (check one, eg [X] - and delete unchecked ones):

Necessary information

Video links (replace text below with your links)

Additional information Currently when WebVTT is specified as output, X-TIMESTAMP-MAP is automatically added. And if no subtitle text is found in the input, it generates an empty WebVTT, with just one line "WEBVTT" in the file.

There are potentially two issues with this implementation. (a) X-TIMESTAMP-MAP is not a WebVTT requirement, but Apple HLS requirement, this may cause a problem if it is used in non-HLS environment. It should be an option to turn it off. (b) On the other hand, Apple HLS spec says "an X-TIMESTAMP-MAP metadata header SHOULD be added to each WebVTT header", which includes the empty ones. The spec also says "If a WebVTT segment does not have the X-TIMESTAMP-MAP, the client MUST assume that the WebVTT cue time of 0 maps to an MPEG-2 timestamp of 0.". But unfortunately, not all clients follow this rule. It is safer to add X-TIMESTAMP-MAP in all VTT files, if it is enabled.

So, my proposal is to add an option for WebVTT, whether X-TIMESTAMP-MAP is enabled or disabled. And if it is enabled, it will be added in every VTT file it generates, including one with no caption cues.

(cf. "HTTP Live Streaming": https://tools.ietf.org/html/rfc8216 section 3.5)

NilsIrl commented 4 years ago

What happens if X-TIMESTAMP-MAP is disabled and the WebVTT cue time 0 doesn't map to an MPEG-2 timestamp of 0?

NilsIrl commented 4 years ago

What would be the reason to not have the X-TIMESTAMP-MAP?

NilsIrl commented 4 years ago

Wouldn't it be better to have the X-TIMESTAMP-MAP header present all the time?

ozt860071 commented 4 years ago

You’ll lose caption, and in worst case, video freeze.

On Dec 4, 2019, at 3:34 PM, Nils ANDRÉ-CHANG notifications@github.com wrote:

What happens if X-TIMESTAMP-MAP is disabled and the WebVTT cue time 0 doesn't map to an MPEG-2 timestamp of 0?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/CCExtractor/ccextractor/issues/1127?email_source=notifications&email_token=AB4ZYEQBI7FXHMYUFSFJIE3QXA5B3A5CNFSM4JQUKKPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF64S7I#issuecomment-561891709, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4ZYERROT4I2YN2DRAMEYTQXA5B3ANCNFSM4JQUKKPA.

ozt860071 commented 4 years ago

Since it is HLS requirement, this may affect negatively in non-HLS use cases. I hope not. I don’t mind having it all the time, though.

On Dec 4, 2019, at 3:35 PM, Nils ANDRÉ-CHANG notifications@github.com wrote:

What would be the reason to not have the X-TIMESTAMP-MAP?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/CCExtractor/ccextractor/issues/1127?email_source=notifications&email_token=AB4ZYERUXAUFA2NTLMFRT43QXA5FLA5CNFSM4JQUKKPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF64VVY#issuecomment-561892055, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4ZYERROQQFDJSCDSCW4I3QXA5FLANCNFSM4JQUKKPA.

ozt860071 commented 4 years ago

I am fine with this. But I don’t know how it affects in other use cases where this is not required.

Thanks

On Dec 4, 2019, at 3:40 PM, Nils ANDRÉ-CHANG notifications@github.com wrote:

Wouldn't it be better to have the X-TIMESTAMP-MAP header present all the time?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/CCExtractor/ccextractor/issues/1127?email_source=notifications&email_token=AB4ZYEXAGJ4VZMFAJXCOZELQXA5V7A5CNFSM4JQUKKPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF65C2A#issuecomment-561893736, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4ZYEXCUV3XZKLDLRY7BXTQXA5V7ANCNFSM4JQUKKPA.

NilsIrl commented 4 years ago

@cfsmp3 What should I do? I feel like the best here is to always use X-TIMESTAMP-MAP.

NilsIrl commented 4 years ago

There is also the general problem that ccextractor creates files even when it can't find anything making this problem more general and not limited to .vtt. @cfsmp3

ozt860071 commented 4 years ago

But for HLS at least, I need an output even if no caption text is contained in the .vtt file. The client needs the get the timing info from X-TIMESTAMP-MAP for synchronization.

Thanks

On Dec 5, 2019, at 12:09 PM, Nils ANDRÉ-CHANG notifications@github.com wrote:

There is also the general problem that ccextractor creates files even when it can't find anything making this problem more general and not limited to .vtt. @cfsmp3 https://github.com/cfsmp3 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/CCExtractor/ccextractor/issues/1127?email_source=notifications&email_token=AB4ZYEUVS7QIUBJILRT5MBLQXFNZBA5CNFSM4JQUKKPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGB7OQQ#issuecomment-562296642, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4ZYERHWKLVUU5W2PXJVT3QXFNZBANCNFSM4JQUKKPA.

NilsIrl commented 4 years ago

The client needs the get the timing info from X-TIMESTAMP-MAP for synchronization.

why would you need to synchronize if you don't have subtitles?

cfsmp3 commented 4 years ago

The client needs the get the timing info from X-TIMESTAMP-MAP for synchronization.

why would you need to synchronize if you don't have subtitles?

It's part of the HLS specification. If there are subtitles in WebVTT:

Subtitles

  1. Subtitle requirements 5.1. Subtitles MAY be provided. 5.2. Subtitles MUST be WebVTT (according to the HLS specification) or IMSC1 in fMP4. 5.3. WebVTT subtitles MUST be in text files, with an X-TIMESTAMP-MAP according to the HLS specification.

References: https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices https://tools.ietf.org/html/rfc8216#page-13

Note that Apple there says MUST (in 5.2) but the RPC8216 says SHOULD. But still, if we generate a .vtt file, even if it has no subtitles, it should have the required header. Otherwise players may refuse to play anything at all.

NilsIrl commented 4 years ago

The thing is that you can't synchronize between 2 things when you're missing 1 of the 2, in this case subtitles.

cfsmp3 commented 4 years ago

An empty subtitle file is not the same as no subtitle file. If there's a subtitle file even if it has no actual subtitles inside you need to be able to know where it starts relative to the video.

In any case if the specs say that's the way it is then that's what we have to implement, unless you want to get in touch (you can) with the RFC8216 writers and discuss :-)

Otherwise we're just not standard compliant and that's not a good thing to be. In fact, if you read the RFC you will see that in a bunch of places it says that clients MUST refuse to play media that is not compliant. Not that they should do their best to play it anyway, but the opposite.

And well, there's a good reason for that: To force all programs generating media to follow the specs.

NilsIrl commented 4 years ago

If there's a subtitle file even if it has no actual subtitles inside you need to be able to know where it starts relative to the video.

  1. As in where the subtitles are located in the video file in bytes.
  2. Or as in the timestamp at which the subtitles start in seconds (or the duration unit that is used).
cfsmp3 commented 4 years ago

You want to sync using time, not bytes. There's some reasons for this, one of them being that you might be starting the stream in the middle and not have access to anything before you started playing - for example, when you turn on your TV your TV needs to start playing the stream from the point you turn it on, Also, the stream never ends...

Second, even if we were talking about actual files with a beginning and an end, they will contain at the very least 1 video track and 1 audio track (possibly more of audio), so what byte offset would you use?

NilsIrl commented 4 years ago

In case 1, the value would just be metadata that would kinda be useless without the exact same video file.

For example in the East Enders file here is the header:

X-TIMESTAMP-MAP=MPEGTS:1090971027,LOCAL:00:00:00.000

MPEGTS is the duration since the start of the stream (before the start of the video file)?

cfsmp3 commented 4 years ago

Not exactly. The MPEGTS is a clock that runs at 90,000 Hz and that rolls over roughly every 26 hours. Forget about the "start of the stream", for all purposes assume the stream doesn't have a beginning or an end, it's 24x7 (assuming we're talking about a TV channel). You need to sync the audio, video and packets. Of course packets interleaved in a way that they are playable without having a huge buffer (which would also cause a delay, and for example when watching sports you want the less possible latency), but still you need to sync everything.

.srt and other video formats do start at 00:00 assuming that you are going to attach them to a file that also starts at 00:00. That's fine, we love .srt and CCExtractor produces timing relative to the start of a file :-) But professional use, in broadcast media, is just different.

If someone from a media company tells us they need the MPEG clock to sync, they're probably know what they are doing.

On Sun, Jan 12, 2020 at 1:28 PM Nils ANDRÉ-CHANG notifications@github.com wrote:

In case 1, the value would just be metadata that would kinda be useless without the exact same video file.

For example in the East Enders file here is the header:

X-TIMESTAMP-MAP=MPEGTS:1090971027,LOCAL:00:00:00.000

MPEGTS is the duration since the start of the stream (before the start of the video file).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CCExtractor/ccextractor/issues/1127?email_source=notifications&email_token=ABNMTWOWJ6GPZBRW7CK7IITQ5ODRDA5CNFSM4JQUKKPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIXEWQQ#issuecomment-573459266, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNMTWNKZLSG6XMWWIYD3BDQ5ODRDANCNFSM4JQUKKPA .

kdrag0n commented 4 years ago

Shouldn't this issue be closed since #1176 has been merged?

NilsIrl commented 4 years ago

Well the underlying issue is still present so it doesn't solve the original problem

(THE header doesn't show up all the time). I'm going to reopen my PR

NilsIrl commented 4 years ago

Is there a file I can test it with? (a file without subtitles).

cfsmp3 commented 4 years ago

Is there a file I can test it with? (a file without subtitles).

You can use any file - just manually select --datapid pointing to a PID that is not present in the stream.

NilsIrl commented 4 years ago

You can use any file - just manually select --datapid pointing to a PID that is not present in the stream.

This shouldn't affect the header right?

NilsIrl commented 4 years ago

https://github.com/NilsIrl/ccextractor/blob/db646f50ac14a772de9f374b2b40e05dbc6c0da9/src/lib_ccx/general_loop.c#L991-L1006

If you look at the above snippet, data_node == NULL when a wrong -datapid is set. The thing is that in order to get the value for X-TIMESTAMP-MAP, data_node needs to be used. I don't know if the problem has to do with the fact -datapid is used. From what I understand it's not the fact we don't have subtitles that prevents from having a correct X-TIMESTAMP-MAP so it should be possible.

cfsmp3 commented 4 years ago

OK so cheating with datapid is not an an option :-(

Check out our list of samples. https://drive.google.com/drive/folders/0B_61ywKPmI0TUUk5LXJPeG1feFE?usp=sharing

Specifically the UK TV one has one that says "no subs"

ozt860071 commented 3 years ago

Sorry not to see this in 0.89. (from CHANGES.TXT)

cfsmp3 commented 3 years ago

Sorry not to see this in 0.89. (from CHANGES.TXT)

  • New: Added an option to disable timestamps for WebVTT (In response to issue #1127) This is not implemented, either.

https://github.com/CCExtractor/ccextractor/pull/1176/files

ozt860071 commented 3 years ago

As you mentioned,

Note that Apple there says MUST (in 5.2) but the RPC8216 says SHOULD. But still, if we generate a .vtt file, even if it has no subtitles, it should have the required header. Otherwise players may refuse to play anything at all.

This needs to be resolved. I think the code is ready.