Custom interface for rendering subtitles on an RGBA texture(s) | (XySubFilter for madVR) [Part 3]

[GoogleCodeExporter]

Part 1: http://code.google.com/p/xy-vsfilter/issues/detail?id=40

Part 2: http://code.google.com/p/xy-vsfilter/issues/detail?id=91

Topic to continue discussions about the subtitle interface we created along 
with madshi, which is now supported by madVR as well as xy-VSFilter in the form 
of XySubFilter.

Latest revision of the new subtitle interface:
http://madshi.net/SubRenderIntf.h

Original issue reported on code.google.com by cyber.sp...@gmail.com on 16 Jul 2013 at 6:50

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Can anybody write up a short summary of what kind of control/information 
abilities media players need from the new subtitle interface/implementation? 
I'm not sure right now. I think we first need to make a list of requirements, 
and once we have that we can brainstorm about how to best realize that.

It would be great if the media player devs joining this discussion could read 
the first 3 short sections of the interface header to get a good understanding 
of how the new subtitle interface works exactly: (1) introduction, (2) basic 
interface principle, (3) auto-loading, external subtitles.

Thanks!!

Original comment by mad...@gmail.com on 16 Jul 2013 at 7:39

[GoogleCodeExporter]

I'm already fully aware of the interface and we will consider integrating it 
into our own renderers. Probably it's best if we can do that in the next 30 
days so we can get some discussion going, but we also have a lot of other 
priorities at the moment.

I'll discuss it with Underground78 and get back to you when he's around.

Original comment by Jules.B...@gmail.com on 16 Jul 2013 at 10:31

[GoogleCodeExporter]

That would be very nice, if you could integrate it into your own renderers, 
too! You don't have to do that in the next 30 days, I think. But we should in 
the near future try to find a solution that gives the media player some control 
over the new interface. So your input on what kind of control the media player 
needs exactly would be much welcome...

Original comment by mad...@gmail.com on 16 Jul 2013 at 10:41

[GoogleCodeExporter]

Concerns were expressed to us about how the interface defines that the Subtitle 
Consumer always auto-load all Subtitle Providers installed on a system, which 
makes it impossible for a Media Player to block and/or control how subtitle 
rendering is handled.

Any major changes which break backwards compatibility with the current version 
1.0.6 of SubRenderIntf.h, I'd really like to see occur before XySubFilter 
reaches wide circulation. We are quickly reaching the point where core portions 
of the interface such as required loading behavior, really need to be finalized 
very soon if any changes are desired. Actual support of such changes by media 
players or otherwise could come at some later date.

> But we should in the near future try to find a solution that gives the media 
player 
> some control over the new interface. So your input on what kind of control 
the media player
> needs exactly would be much welcome...

Basically what madshi said.

Original comment by cyber.sp...@gmail.com on 16 Jul 2013 at 12:24

[GoogleCodeExporter]

Is the auto-loading the only problem?

Original comment by mad...@gmail.com on 16 Jul 2013 at 12:34

[GoogleCodeExporter]

> Is the auto-loading the only problem?

I really don't know, the media player developers would need to answer that.

The problem which auto-loading causes, is that if MPC-HC/MPC-BE for example 
sets XySubFilter to "Block" in External Filters. This causes XySubFilter own 
auto-loading mechanism for embedded subtitles to get blocked, but then madVR 
would load XySubFilter into the graph anyway.

Then there is the issue with external subtitles, since in this case XySubFilter 
requires the assistance of a subtitle consumer like madVR to enter the graph. 
This results in XySubFilter loading external subtitles and begin rendering them 
with madVR, irrespective on any other subtitle renderer which may or may not be 
present.

clsid @Doom9 mentioned that MPC-HC/MPC-BE have subtitle functionality (subtitle 
downloads, drag/drop, etc) which only work with the built-in ISR. This could 
potentially result in both an ISR subtitle renderer from a media player, as 
well as a subtitle provider like XySubFilter being active and rendering 
subtitles at the same time.

The issue seem to come down to how we explicitly designed the SubRenderIntf.h 
to function outside of DirectShow, making traditional methods a media player 
would use to take control over the situation useless. This suggests that we 
need to ofter "something" in SubRenderIntf.h which allows media players to 
regain some of that control. What form that "something" takes would really 
depend on the explicit needs of media player developers, and is currently up 
for debate.

Original comment by cyber.sp...@gmail.com on 17 Jul 2013 at 12:02

[GoogleCodeExporter]

Instead of relying on the renderer to preform auto-loading of the subtitle 
renderer I'm in favor of relying on the player to provide a subtitle pin on the 
filter graph for any available external subtitles.

Original comment by Jules.B...@gmail.com on 17 Jul 2013 at 11:33

[GoogleCodeExporter]

@Jules, that would work nicely for all media players which have explicit 
support for the new subtitle interface, and which are able to automatically 
load external subtitles. But if you want to use it in a media player which 
doesn't have such support, there's a problem. Maybe the solution would be for 
the media player to somehow "signal" that it supports the new subtitle 
interface, and if it does, then the consumer should not auto-load the subtitle 
providers?

Original comment by mad...@gmail.com on 17 Jul 2013 at 5:43

[GoogleCodeExporter]

In my opinion (but I'm biased) making this interface work for all players 
without having them add any support for it will just result in a lot of hacks 
and shouldn't be part of the specification.

Original comment by Jules.B...@gmail.com on 17 Jul 2013 at 5:54

[GoogleCodeExporter]

I could live with that, but let's wait what the others have to say about this.

Original comment by mad...@gmail.com on 17 Jul 2013 at 5:58

[GoogleCodeExporter]

Wouldn't that just make things worse for subtitle filters like VSFilter.dll and 
FFDShow which already auto-load external subtitles themselves? How would 
multiple external subtitles + embedded subtitles for a video be handled? 
Overall, I don't like the idea of requiring every media player implement 
loading of and support of every possible external subtitle format. Doing this 
would just seem to create more problems than it solves.

In the scope of the current revision of SubRenderIntf.h, multiple Subtitle 
Providers are intended to be handled by a Subtitle Manager. The Subtitle 
Manager would accept connections from multiple Subtitle Providers, and handle 
switching between Providers, and/or combining subtitle bitmaps from multiple 
Providers before passing them along to a Subtitle Consumer like madVR for 
display.

> // HKEY_CLASSES_ROOT\Autoload.SubtitleProvider\YourSubProvName REG_SZ %GUID%

This exists for multiple purposes:

Allowing multiple Subtitle Providers to connect to a Subtitle Manager.
Allowing Subtitle Providers to find External Subtitles
Allowing Subtitle Providers to find Embedded Subtitles if they do not support 
auto-loading themselves.

If heading towards the direction of explicit Media Player support, why not just 
move forward and implement a Subtitle Manager as we were planning for in 
SubRenderIntf.h? We could then add new Subtitle Mangers specific fields which 
signal connected Providers if they should be idle, as well as new callbacks for 
media players to communicate with the Subtitle Manager?

Original comment by cyber.sp...@gmail.com on 17 Jul 2013 at 11:25

[GoogleCodeExporter]

>This exists for multiple purposes:
>
>Allowing multiple Subtitle Providers to connect to a Subtitle Manager.
>Allowing Subtitle Providers to find External Subtitles
>Allowing Subtitle Providers to find Embedded Subtitles if they do not 
>support auto-loading themselves.

Also:

Allowing multiple Subtitle Providers a chance to load their supported subtitle 
formats, since format support may not always overlap between different 
Providers.

Allowing Subtitle Providers which only support external subtitles.
(We were actually leaning towards this direction with XySubFilter in our 
external_detector branch. The Consumer auto-loaded our external subtitle 
detector which have the job of manually loading XySubFilter itself and acting 
as a pass-through for both internal/external subtitles. Our developer prefers 
this method for XySubFilter, and was planing on migrating to it sometime after 
our first Beta release. Though that opinion may change, if the interface itself 
changes how loading is handled.)

Original comment by cyber.sp...@gmail.com on 18 Jul 2013 at 12:54

[GoogleCodeExporter]

Ok, how do you guys like the following solution?

(1) Keep the auto-loading logic exactly as it is. So consumers still auto-load 
all registered filters.
(2) However, the filters which get auto-loaded are *NOT* allowed to actually 
*render* the external subtitles. The only thing it is allowed to do is to add 
the external subtitles to the DirectShow graph as additional subtitle pins.

This solution would nicely solve all auto-loading problems, wouldn't it?

However, would it maybe conflict with the external subtitle loading of 
VSFilter.dll and ffdshow, resulting in all external subtitles tracks being 
offered twice?

Original comment by mad...@gmail.com on 18 Jul 2013 at 7:18

[GoogleCodeExporter]

Duplicates would indeed be an issue. Such a method would turn in a mess real 
quickly if you had multiple subtitle providers all adding external subtitles 
with different naming syntax and sorting methods to the DirectShow Graph.

@madshi

I feel more than anything we need to add a centralized control hub for media 
players, likely in the Subtitle Consumer:

A) "On/Off switch" for the subtitle interface which can be flipped by the media 
player at will during playback. For example, for media players like MPC-HC and 
MPC-BE which contain an ISR.

B) Offer media players a way to perform entirely manually graph building, where 
nothing is allowed to auto-load, and the media player can load Subtitle 
Providers manually. For example, so XySubFilter can be easily supported by the 
"SmartPlay" option of Zoom Player.

Any such control commands for the subtitle interface could be communicated my 
the Media Player to the Subtitle Consumer and then propagated downstream. If a 
media player manually loads the video renderer Subtitle Consumer first into the 
graph, such commands could be issued before any other filters are loaded. As 
part of such a solution, we should likely remove XySubFilter's built-in 
auto-load functionality, and depend entirely on either the media player or 
consumer to load Subtitle Providers. By default a Consumer would still autoload 
all Providers, unless a media player desires otherwise.

Possible commands could be:

Enable/Disable Provider auto-loading
Enable/Disable Consumer
Block Provider "ProviderName"
Prefer Provider "ProviderName"

(Provider names could be acquired from the "Autoload.SubtitleProvider" key)

Original comment by cyber.sp...@gmail.com on 18 Jul 2013 at 8:53

[GoogleCodeExporter]

There is no guarantee that the video renderer is the consumer. E.g. at some 
point nevcairiel was also considering making LAV Video Decoder a consumer. 
There could be multiple consumers in the graph. It would make things a bit 
painful for the media player if it had to browse through the graph and find all 
consumers and tell *all of them* which providers should be blocked or not 
blocked.

> Prefer Provider "ProviderName"

For a "prefer" solution to work, first of all we would have to know when two 
providers try to handle the same external subtitle file. And there's no way of 
knowing that at the moment.

However, this shows a problem with the current logic: Since nobody knows which 
provider auto-loads which external subtitles, there can always be duplicates. 
E.g. if both LibAss and XySubFilter providers are registered, both might 
auto-load the same external subtitles tracks, so every external track would 
have one duplicate. Unfortunately we can't simply have consumers only loading 
one provider, because there might also be some providers which are specific to 
certain subtitle formats. E.g. there might be one subtitle provider which only 
supports external .srt subtitles and a different provider which only supports 
external .sup subtitles. In that case both providers would have to be loaded.

-------

The only proper way of solving the "duplicates" problem with external subtitles 
is to have a central piece of software which controls the loading of external 
subtitles. This could either be the media player, or we create an external 
filter/dll which does all the dirty work.

One solution could be to let providers register themselves to the registry with 
a list of supported subtitle file extensions. Then this central piece of 
software could enumerate the registered providers and their supported subtitle 
file extensions and then make decisions based on that information. This logic 
would also easily allow blocking and prefering functionality. However, if we go 
this way, the central piece of software would need to be able to somehow tell 
the auto-load providers to handle only one specific external subtitle file.

Original comment by mad...@gmail.com on 18 Jul 2013 at 9:34

[GoogleCodeExporter]

> There is no guarantee that the video renderer is the consumer.
> E.g. at some point nevcairiel was also considering making LAV Video Decoder
> a consumer. 

Create a registry key to list the names and GUIDs of all installed Consumers?

> It would make things a bit painful for the media player if it had to browse
> through the graph and find all consumers and tell *all of them* which 
providers 
> should be blocked or not blocked.

Create a global registry key for the Subtilte Interface with subkeys optionally 
added by each media player to list their default preferences?

> For a "prefer" solution to work, first of all we would have to know when
> two providers try to handle the same external subtitle file. 

Why? Prefer/Block are both overrides, which imply to use a particular provider 
if available. From a Consumer standpoint, it would mean to only load and/or 
accept connections from the "preferred" Provider if installed.

Original comment by cyber.sp...@gmail.com on 18 Jul 2013 at 9:55

[GoogleCodeExporter]

> From a Consumer standpoint, it would mean to only
> load and/or accept connections from the "preferred"
> Provider if installed.

As I wrote in my previous comment:

> Unfortunately we can't simply have consumers only
> loading one provider, because [...]

Original comment by mad...@gmail.com on 18 Jul 2013 at 10:01

[GoogleCodeExporter]

I don't see the problem. If a media player is telling use it wants a particular 
Provider to be used, we do it, no questions asked. This of course would have 
the implication that other Providers (which may support different formats) 
would not get used, but this is what the media player is asking to occur if it 
sets "prefer". Media players should normally never set "prefer" because of 
this. Setting "Block Provider" or "Disable Provider auto-loading" would be much 
safer, and what should normally be used unless forcing a particular provider 
with "Prefer Provider" is desired.

Original comment by cyber.sp...@gmail.com on 18 Jul 2013 at 10:14

[GoogleCodeExporter]

I think you should not over-complicate this thing, the whole topic is already 
full of over-complications for 0.1% use-cases.

Give the player an option to disable any auto-loading so it can control what 
happens, and be done with it. Preferably without doing that through some 
registry keys, because that just feels awful.

Original comment by h.lepp...@gmail.com on 18 Jul 2013 at 10:16

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Re #21:

Sorry, but this is totally different to how "prefer" and "block" works for 
DirectShow filters in MPC-HC. As such I find it a nearly useless solution. If 
you're happy with a very limited solution which just gives MPC-HC the power to 
disable the new interface completely, then we should skip "prefer"/"block" and 
instead offer an option to simply totally disable/enable the new subtitle 
interface. But if you want to support "prefer" and "block" then it should be 
implemented properly, similar to how it works for DirectShow filters in MPC-HC.

Re #22:

I could certainly live with that. And I agree that we should keep registry use 
at a minimum.

Original comment by mad...@gmail.com on 18 Jul 2013 at 10:26

[GoogleCodeExporter]

At the bare minimum we would need:

"Disable Auto-loading" to allow media players to manually load Subtitle 
Providers (or not).

"Disable Consumer" command if an built-in ISR gets enabled in the middle of 
playback (subtitle downloads, drag/drop, etc).

I'm fine with scrapping the idea of allowing media players to "Force/Prefer" or 
"Block" a particular Provider, unless we get a specific request for such a 
functionality. I only tossed them in as extras if we did want to offer full 
fledged management of providers, and I like the idea of writing all supported 
internal and external subtitle formats to the registry if we go the route of 
mimicking MPC-HC DirectShow "Prefer", instead of more dumb "Force/Prefer" 
functionality like I had suggested.

Original comment by cyber.sp...@gmail.com on 18 Jul 2013 at 10:46

[GoogleCodeExporter]

The whole "prefer"/"block" functionality in MPC-HC only shows effect if you 
rebuild the graph. So I think a simple "disable auto-loading" would do. The 
"Disable Consumer" functionality would be nice, but not really necessary, I 
think. If the end user wants to switch subtitle renderers, why should that work 
any faster than changing decoder filters? I think it's acceptable if we require 
the user to rebuild the graph in both cases...

Opinions?

Original comment by mad...@gmail.com on 18 Jul 2013 at 10:50

[GoogleCodeExporter]

> If the end user wants to switch subtitle renderers, why should that work 
> any faster than changing decoder filters? I think it's acceptable if we 
> require the user to rebuild the graph in both cases...

You've obviously never used this functionality before in MPC-HC, try it.

Play a video with subtitles with XySubFilter with the MPC-HC ISR disabled, then 
drag an external subtitle into the MPC-HC window. MPC-HC will seamlessly enable 
the MPC-HC ISR in the middle of playback without even pausing the graph, and 
suddenly both the ISR and XySubFilter will be rendering different subtitles at 
the same time. Why you even allow this in madVR is another issue. If you don't 
want such a "Disable Consumer" functionality triggered by the media player, 
then you should consider handling this automatically when the ISR starts 
rendering subtitles when a Subtitle Consumer is connected.

Original comment by cyber.sp...@gmail.com on 18 Jul 2013 at 11:10

[GoogleCodeExporter]

I'm not sure if we should really go out of our way to make every current MPC-HC 
feature work nicely which is still based on the ISR. In the long run we hope to 
replace the built in ISR, right? So IMHO the question we should ask ourselves 
is: How can we make it possible for MPC-HC to drag & drop a subtitle file so 
that it renders with XySubFilter? Probably the clean solution would be for 
MPC-HC to add a subtitle pin to the graph and connect that to XySubFilter 
during playback. That should already work without us needing to do any extra 
work, right?

Original comment by mad...@gmail.com on 18 Jul 2013 at 11:18

[GoogleCodeExporter]

Here's a suggested (not decided yet!) new version of the interface header:

http://madshi.net/SubRenderIntf2.h

It's similar to the old/official one, but I've added two global static 
functions to the header:

// ---------------------------------------------------------------------------
// auto-loading helper functions
// ---------------------------------------------------------------------------

// This global function makes it possible for media players to globally
// enable/disable the auto-loading of subtitle providers (for external
// subtitle tracks).
static void EnableSubtitleAutoLoading(bool enable);

// This global function should be called by subtitle consumers, the first
// time the upstream filter connects (in the CompleteConnect method).
// This function auto-loads all registered subtitle providers, unless the
// auto-loading was disabled by the media player.
static void AutoLoadSubtitleProviders();

// ---------------------------------------------------------------------------

This makes is extremely easy for the media player to completely disable the 
auto-loading. The media player just needs to call 
"EnableSubtitleAutoLoading(false)" before building the graph.

The disadvantage of this new header version is that it contains some real code 
in it, not just definitions. That isn't "nice", normally header files are 
supposed to only contain definitions. If you guys prefer, I could put the code 
into a cpp file instead. But having only one file has its advantages, too.

Any thoughts on this?

We can still discuss other functionality (like disabling consumers on the fly, 
if it's *really* necessary) in addition to this. Or we could use a completely 
different solution. But I thought I'd give this a first try, and the new header 
suggestion would already work quite nicely. It would need no changes in 
XySubFilter at all. Only a new madVR build compiled with the new header...

Original comment by mad...@gmail.com on 18 Jul 2013 at 11:27

[GoogleCodeExporter]

> Wouldn't that just make things worse for subtitle filters like VSFilter.dll 
and FFDShow which already auto-load external subtitles themselves? How would 
multiple external subtitles + embedded subtitles for a video be handled?

We have control over VSFilter.dll, so we can just change the behaviour if 
necessary. I don't know how we will handle external + embedded subtitles or 
FFDShow compatibility, but it doesn't really seem like a blocker to me.

> "Disable Consumer" command if an built-in ISR gets enabled in the middle of 
playback (subtitle downloads, drag/drop, etc).

You're overcomplicating things for obscure use cases, but a basic use case like 
subtitle downloads or delayed subtitle loading is not handled by the interface? 
The subtitle pin idea fails here though since the graph is already built.

Original comment by Jules.B...@gmail.com on 18 Jul 2013 at 11:41

[GoogleCodeExporter]

> The subtitle pin idea fails here though
> since the graph is already built.

A graph can be modified/adjusted in real time, IIRC? Can't you just add a 
subtitle pin on the fly and connect it to XySubFilter? I think in the worst 
case you have to stop the graph and restart it, but for the end user it 
shouldn't be a problem?

Original comment by mad...@gmail.com on 18 Jul 2013 at 11:57

[GoogleCodeExporter]

You have to stop the graph to modify it like that, and that will cause a 
noticeable playback hiccup, so users may care.

Original comment by h.lepp...@gmail.com on 18 Jul 2013 at 12:00

[GoogleCodeExporter]

According to this:

http://msdn.microsoft.com/en-us/library/windows/desktop/dd388735(v=vs.85).aspx

... it should be possible on the fly without stopping the graph. Of course both 
the subtitle pin and XySubFilter would need to support dynamic graph building 
for this to work.

Original comment by mad...@gmail.com on 18 Jul 2013 at 12:24

[GoogleCodeExporter]

I've not seen anything that supports this, ever.

Original comment by h.lepp...@gmail.com on 18 Jul 2013 at 12:29

[GoogleCodeExporter]

In any case, I do not think the video renderer should be in charge of loading 
the subtitle filter. Connecting filters should be the responsibility of the 
player. This way we can cross out auto-loading, merits and the registry from 
the interface which should simplify it considerably.

It is true that you need explicit external subtitle support from the player, 
since you don't connect to the video pin anymore. But you already need a 
renderer that supports this interface. It wouldn't seem that strange to me that 
you also need a player that supports it.

Original comment by Jules.B...@gmail.com on 18 Jul 2013 at 12:37

[GoogleCodeExporter]

> I've not seen anything that supports this, ever.

Maybe so, but it shouldn't be that hard to implement. And not every filter in 
the graph needs to support it, but only those 2 which need to be dynamically 
reconnected. (And the media player.)

> In any case, I do not think the video renderer
> should be in charge of loading the subtitle filter.

In a perfect world, I agree. But external subtitles are quite common and so 
rather important. If we remove auto-loading from the interface completely, 
basically that would mean that maybe some day MPC-HC and MPC-BE support 
external subtitles with the new subtitle interface properly. But what about 
other media players? ZoomPlayer, KMPlayer, PotPlayer, J.River MC etc etc. I 
don't want to be a position where I have to "beg" every media player developer 
to implement proper support for external subtitles. It's *so* much easier just 
supporting this the way we currently do. It's already working nicely right now, 
automatically, in all media players.

If you MPC-HC developers don't like the auto-loading, the new interface 
suggestion I've posted earlier allows you to completely turn off auto-loading 
with just one simple function call. Shouldn't that be good enough for your 
needs?

> Connecting filters should be the responsibility
> of the player. This way we can cross out auto-
> loading, merits and the registry from the interface
> which should simplify it considerably.

Registry and auto-loading: Yes.
Merit: No.

Merits are still needed in any case. There could be multiple subtitle consumers 
in the graph, and the provider needs a way to decide which one to serve.

Original comment by mad...@gmail.com on 18 Jul 2013 at 1:28

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

> Merits are still needed in any case. There could be multiple subtitle 
consumers in the graph, and the provider needs a way to decide which one to 
serve.

Wouldn't you rather just limit the interface to connecting subtitle renderers 
and video renderers? Instead of complicating things with post processors and 
video decoders?

Original comment by Jules.B...@gmail.com on 18 Jul 2013 at 1:41

[GoogleCodeExporter]

> I don't want to be a position where I have to "beg" every media player 
developer to implement proper support for external subtitles.

You wouldn't have to beg, if the users of those players want sharp subtitles 
then the developers of that player will implement the functionality without you 
doing anything. Ofcourse there are some players like WMP which will never 
implement such changes, so if you insist on supporting them then it's okay as 
long as you can turn it off.

> If you MPC-HC developers don't like the auto-loading, the new interface 
suggestion I've posted earlier allows you to completely turn off auto-loading 
with just one simple function call. Shouldn't that be good enough for your 
needs?

I don't know yet if I agree with the new interface, I'll get back to you on 
that later.

Original comment by Jules.B...@gmail.com on 18 Jul 2013 at 1:53

[GoogleCodeExporter]

> Wouldn't you rather just limit the interface
> to connecting subtitle renderers and video
> renderers? Instead of complicating things with
> post processors and video decoders?

Well, we were trying to think a bit ahead. If the new subtitle interface is 
liked by many developers, there's a chance more subtitle providers could exist 
some day in the future. E.g. there could be one based on LibAss, as an 
alternative to XySubFilter. Or maybe at some point a follow-up-spec to ASS is 
released, then there might be a subtitle provider just for ASS2 rendering. Also 
nevcairiel considered using the new subtitle interface for DVD subtitles as 
part of the LAV Video Decoder. There might be some overlap of supported 
subtitle formats between those providers, but maybe some providers will only 
handle very specific formats.

On the other end, there might be multiple consumers, too. E.g. some users are 
using LAV Video Decoder in their reencoding processing chain, making use of 
hardware accelerated decoding and DXVA2 or YADIF deinterlacing. For such 
purposes it might be helpful if LAV Video Decoder could also be a subtitle 
consumer. At least I think nevcairiel at some point considered this.

If there are multiple provider and multiple consumers in the graph, the best 
way to handle all that is to have one subtitle mixer in between which could 
then serve as the central switching hub. In order for all of this to work 
properly, we had to add the whole merit system.

Of course we could dumb the interface down and limit it to really only connect 
subtitle renderers with video renderers. It would simplify the interface a bit. 
But then some of the things mentioned above would not be possible.

Original comment by mad...@gmail.com on 18 Jul 2013 at 2:19

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Okay I'm now working on adding support for this interface. Since the 
functionality of our internal subtitle renderer interfaces and SubRenderIntf.h 
are quite similar I'll try to support it by implementing a compatibility layer 
in between. If that doesn't work out I'll implement them in parallel, but that 
will cost a lot more time. I'm currently working on a ISubPicProvider which 
will serve subtitle frames from XySubFilter instead of rendering them itself.

Looking at both interfaces I do have some questions. I'm not an expert on 
VSFilter, so warn me if I stop making sense.

> STDMETHOD(RequestFrame)(REFERENCE_TIME start, REFERENCE_TIME stop, LPVOID 
context) = 0;

RequestFrame expects a start and stop time, but provides no information about 
when certain subtitles start or stop. For animated subtitles this is not really 
a problem since you request them frame-by-frame anyway and can guess the start 
and stop times yourself from the fps. But for static subtitles our internal 
interface wants to request the frame using the start and end time of the 
subtitle itself and thus render the subtitles per subtitle instead of per 
frame. I propose adding a function to ISubRenderProvider that allow you to give 
a REFERENCE_TIME and it'll give you the start and stop times of the subtitle at 
that position.

And then there is the IsAnimated flag, which basically communicates whether 
we're dealing with animated subtitles like ASS or static ones like vobsub. I 
propose to add this information as a field to the subtitle provider. However 
due to a well-known "feature" internally SRT is considered an animated 
subtitle. You are ofcourse not forced to reproduce this and can make your own 
decision about whether you consider SRT an animated subtitle. Since it seems 
that the ISR is able to process this flag per-subtitle you can also choose to 
add it to the function I described in the previous paragraph, but this 
functionality is not used internally so decide what makes most sense for your 
interface.

I think these additions make sense and it will greatly help compatibility 
between our ISubPicProvider to your ISubRenderProvider. What is your opinion on 
these additions?

Original comment by Jules.B...@gmail.com on 21 Jul 2013 at 10:20

[GoogleCodeExporter]

I don't think its a good idea, a consumer should simply ask on a frame-by-frame 
basis, that way you don't need to know if its animated or not, and can simply 
draw the subs as you get them.

If you want to implement caching/buffering/reusing of the subs, the sub 
provider is supposed to identify identical subs with the same id, so you can 
use that to identify a sub that stays the same for multiple frames.

Original comment by h.lepp...@gmail.com on 21 Jul 2013 at 10:46

[GoogleCodeExporter]

I understand, but our current caching scheme is based on subtitle timings. 
Others implementing the interface may be interested in such information too. I 
don't think it'd be unreasonable to add.

If these are not added another option for me is to implement a new subtitle 
queue based on the id-scheme instead of reusing the current one which is based 
on subtitle timings.

Original comment by Jules.B...@gmail.com on 21 Jul 2013 at 11:00

[GoogleCodeExporter]

So basically it's up to the devs of xy to decide if they also want to allow 
people to use a subtitle queue based on subtitle timings instead of subtitle 
ids with their filter.

Original comment by Jules.B...@gmail.com on 21 Jul 2013 at 11:04

[GoogleCodeExporter]

I'm not sure I understand the issue, and what this has to do with Subtitle 
Providers.

XySubFilter has more advanced animation detection than MPC-HC, and we also 
internally queue unique subtitle Bitmap IDs based on start/end timings. When 
the Subtitle Consumer requests subtitles frame-by-frame, the Subtitle Provider 
informs the Consumer of memory pointers for all active Bitmap IDs for that 
frame. Subtitle Consumers are expected to handle identical Bitmap IDs for 
static subtitles in a logical manner.

Rather than us, I doubt that madshi would be in favor of supporting what you 
are suggesting in madVR, as keeping track of start/end times sounds like 
unneeded complexity for Consumers. After all you could potentially have 
hundreds if not more active subtitle bitmaps in a frame, all with different 
durations. It sounds like a nightmare for implementing a simple Subtitle Queue 
(i.e. subtitle pre-buffering) like madVR uses for XySubFilter.

> I'm currently working on a ISubPicProvider which will serve subtitle 
> frames from XySubFilter instead of rendering them itself.

> So basically it's up to the devs of xy to decide if they also want to allow 
people to use
> a subtitle queue based on subtitle timings instead of subtitle ids with their 
filter

Maybe I'm just confused at what you are trying to do here? Could you explain 
further?

Are you working on:

A) Subtitle Provider for the MPC-HC ISR to use with madVR

B) Subtitle Consumer (EVR-CP modification?) to use with XySubFilter

C) Both A) & B)

D) Subtitle Consumer which translates XySubFilter bitmaps to the interface 
currently used by the MPC-HC ISR?
It's worth mentioning that our subtitle interface bitmaps use pre-multipled 
alpha, which require pre-multiplied alpha-blending.

Original comment by cyber.sp...@gmail.com on 22 Jul 2013 at 1:58

[GoogleCodeExporter]

@Jules.Blok

Subtitles may be invalidated because users edit it or new samples receive from 
source filter or a new subtitle track is selected. Anyway if you want to cache 
a subtitle based on its display time, you'll need a way for provider to inform 
you to invalidate the cache. It does not seem to be a good idea.

Original comment by YuZhuoHu...@gmail.com on 22 Jul 2013 at 6:37

[GoogleCodeExporter]

> RequestFrame expects a start and stop time, but provides
> no information about when certain subtitles start or stop.

True. Basically we tried to logically free the consumer from having to know 
*anything* about the underlying subtitle track/format. Basically the consumer 
just tells the provider that the consumer needs a subtitle for a specific video 
frame and that specific video frame is identified by the video frame's 
start/stop times. The consumer doesn't know (nor does it care) when the 
subtitles start/stop or whether they're animated. The consumer is really naive 
and simply requests a new rendered subtitle for every video frame.

However, to make this more efficient, the provider is allowed (actually 
encouraged) to reuse bitmap IDs to signal to the consumer whether a subtitle 
has changed compared to the previous video frame or not, allowing the consumer 
to optimize it's work flow accordingly. E.g. madVR only uploads every subtitle 
bitmap ID to the GPU once, which saves quite a bit of PCI Express bandwidth and 
CPU power.

Of course we could add more methods to let the consumer ask information like 
start/stop time of subtitles and whether they're animated or not. But really, 
from a logical point of view the consumer doesn't need to know, so I'm not sure 
if we should add this. I do understand that adding such things would make 
compatability easier, but on the other hand I also value a clean and simple 
interface. So I'm not sure what to do.

What I'm not sure about right now is whether those added provider fields you're 
suggesting would require changes in the consumer behaviour as well? I'm not 
sure I'd like that, because it would make development of every future consumer 
more complex.

What do you think?

Original comment by mad...@gmail.com on 22 Jul 2013 at 7:08

[GoogleCodeExporter]

> Rather than us, I doubt that madshi would be in favor of supporting what you 
are suggesting in madVR, as keeping track of start/end times sounds like 
unneeded complexity for Consumers.

It would not be mandatory to do so, it's just needed for our subpic buffer 
which requests subtitles from the provider ahead of time.

> Maybe I'm just confused at what you are trying to do here? Could you explain 
further?

I'm working on B) Subtitle Consumer to use with XySubFilter. But instead of 
implementing it as a completely new interface I'm trying to implement a class 
that provides compatibility with our current ISR interface, using you 
XySubFilter instead of our internal VSFilter.

> Subtitles may be invalidated because users edit it or new samples receive 
from source filter or a new subtitle track is selected. Anyway if you want to 
cache a subtitle based on its display time, you'll need a way for provider to 
inform you to invalidate the cache. It does not seem to be a good idea.

True our ISR has subtitle invalidation calls which take time as a parameter 
indicating after which point subtitles are invalidated (-1 indicating all of 
them). So to make it work correctly the Subtitle Consumer also needs to 
optionally implement a call to invalidate subtitles on any buffering it's doing.

> Of course we could add more methods to let the consumer ask information like 
start/stop time of subtitles and whether they're animated or not. But really, 
from a logical point of view the consumer doesn't need to know, so I'm not sure 
if we should add this.

Subtitle timings and invalidation is needed to buffer the subtitle frames ahead 
of time. If you do not want the subtitle consumer to be able to buffer them 
then I'll need to implement a naive subtitle queue which does not do any 
buffering ahead of time.

Original comment by Jules.B...@gmail.com on 22 Jul 2013 at 10:01

[GoogleCodeExporter]

> What I'm not sure about right now is whether those added provider fields 
you're suggesting would require changes in the consumer behaviour as well? I'm 
not sure I'd like that, because it would make development of every future 
consumer more complex.

Subtitle buffering would be completely optional and the subtitle consumer can 
ignore it if it wants to just request them when processing the frame.

Original comment by Jules.B...@gmail.com on 22 Jul 2013 at 10:04