search

ffmpeginteropx / FFmpegInteropX

FFmpeg decoding library for Windows 10 UWP and WinUI 3 Apps
Apache License 2.0
205 stars 52 forks source link

External subtitle parser #415

Closed brabebhin closed 6 months ago

brabebhin commented 6 months ago

This allows creation of TimedMetadataTracks from subtitle files. It introduces a new class which allows parsing a single subtitle file, and exposes the related functionalities, SubtitleStreamInfo and subtitle delay.

This TimedMetadataTrack can then be used with any MediaPlaybackItem, created by any MediaSource type

softworkz commented 6 months ago

Is it meant to be complete? If yes, what does it do actually?

Is it supposed to allow adding a subtitle track to an arbitrary (non-FFMpegInteropX MediaSource)?

brabebhin commented 6 months ago

It introduces a new class in the namespace, SubtitleParser. This subtitle parser allows you to read a subtitle stream and exposes the resulting SubtitleStreamInfo for you to use as you see fit.

I wouldn't say it is 100% complete, looking forward to feedback ^^

softworkz commented 6 months ago

It introduces a new class in the namespace, SubtitleParser. This subtitle parser allows you to read a subtitle stream and exposes the resulting SubtitleStreamInfo for you to use as you see fit.

But for which case would I need this? I can already add external subtitles to the FFmpegMediaSource already..?

brabebhin commented 6 months ago

The FFmpegMediaSource adds the external subtitle to the internal MediaPlaybackItem it created.

This PR helps you add a subtitle to a MediaPlaybackItem created from an AdaptiveMediaSource.

softworkz commented 6 months ago

The FFmpegMediaSource adds the external subtitle to the internal MediaPlaybackItem it created.

This helps you add a subtitle to a MediaPlaybackItem created from an AdaptiveMediaSource.

LOL, I don't get it. Even those two sentences don't go together for me. Could you perhaps show some pseudo-code for how to set up playback in the way you mean?

brabebhin commented 6 months ago

Yeah i see how that could have been confusing lol.

I've added 2 more letters to clarify.

softworkz commented 6 months ago

Oh, these two letters make a difference indeed!

And the second thing was that I had overlooked the most important member in the IDL:

SubtitleStreamInfo SubtitleTrack{get;};

Guess why I overlooked it.....no doc comments hehe...

softworkz commented 6 months ago

It's working..!

brabebhin commented 6 months ago

I'll also add a convenience method for URI. But I'm bit busy over the weekend. Glad to know it works.

lukasf commented 6 months ago

I just thought, this won't work for true live streams, right? The implementation tries to read the stream till its end, which is fine for finite files. But if this is a true live stream, the end might never come. Not sure if live streams use text subs like that. Could this be a problem? Do we need to safeguard against this somehow?

softworkz commented 6 months ago

I just thought, this won't work for true live streams, right? The implementation tries to read the stream till its end, which is fine for finite files. But if this is a true live stream, the end might never come. Not sure if live streams use text subs like that. Could this be a problem? Do we need to safeguard against this somehow?

I'd say no. I'm not aware of any live-streaming subtitle delivery method of that kind. For actual live streams (HLS), there's just one supported subtitle format: WebVTT (segmented!). There are three other possibilities (yet not per HLS spec) how an HLS live stream could carry subtitles:

All those are can be contained in the MPEGTS stream: DVB and Teletext subs are separate elementary streams (having their own PID) and ATSC closed captions are included as side data in the video stream. Often, players can show these because they have implementation for it anyway (for playing MPEGTS which is not delivered via HLS).

Where the "external subtitle parser" helps (us) are those cases where the server needs to transcode. This means in our case that it's delivered via HLS, but as VOD type HLS stream. When we would convert the subtitles to HLS at the server side, a lot of presentation aspects can get lost (especially in case of ASS), and there it's preferable to feed the subtitles separately (via this external subtitle parser).

brabebhin commented 6 months ago

I just thought, this won't work for true live streams, right? The implementation tries to read the stream till its end, which is fine for finite files. But if this is a true live stream, the end might never come. Not sure if live streams use text subs like that. Could this be a problem? Do we need to safeguard against this somehow?

The URI overloads should be the equivalent of UWP TimedTextSource URI APIs. As far as I can tell, those will also read the stream until the end.

I am on the fence about this, should we offer a cancellation token to prevent hanging up forever?

lukasf commented 6 months ago

If live stream subtitles are always provided within the video stream as @softworkz wrote, this method will never be used for that. Adding a CancellationToken would allow to break processing. But the question is, would anyone really pass one in (and cancel it after a certain timeout)? Probably not. So maybe we should just leave it as is. From my POV, this is ready for merge.