Custom interface for rendering subtitles on an RGBA texture(s) | (XySubFilter for madVR) [Part 3]

[GoogleCodeExporter]

Part 1: http://code.google.com/p/xy-vsfilter/issues/detail?id=40

Part 2: http://code.google.com/p/xy-vsfilter/issues/detail?id=91

Topic to continue discussions about the subtitle interface we created along 
with madshi, which is now supported by madVR as well as xy-VSFilter in the form 
of XySubFilter.

Latest revision of the new subtitle interface:
http://madshi.net/SubRenderIntf.h

Original issue reported on code.google.com by cyber.sp...@gmail.com on 16 Jul 2013 at 6:50

[GoogleCodeExporter]

I don't like how the YUV Matrix is being used by XySubFilter. As I understand 
ot, it will always correct the color values unless explicitly specified that it 
doesn't have to. I realize the importance of it, but subtitles in the ASS 
format should really just be specified in RGB and we should encourage that by 
making it the default.

For this I propose a new version of the ASS standard where RGB values that 
compensate for the YUV matrix are deprecated and will be displayed without 
color correction unless the yuv matrix field is explicitly specified in the 
subtitle file. This new version would therefore also make the yuv matrix field 
an official part of the standard. Files that still specify that they're written 
in an older ASS version will still be color corrected by default as they are 
now.

This way we have a nice clean standard without the complicated color correction 
while we still maintain support for correctly displaying old subtitles with 
color correction.

What are your opinions on this?

Original comment by Jules.B...@gmail.com on 27 Jul 2013 at 9:24

[GoogleCodeExporter]

@Jules, we had discussed all this back in the day when we all tried to agree on 
how to solve the color matrix thing. Discussions were done between all 
interested parties, invitations to the MPC-HC team and other media player 
developers to join the dicussion were also issued, IIRC.

The key problems are that:

(1) 99.9% of all HD ASS files out there require color correction to look as 
intended by the subtitle author.
(2) If new subtitles were written in "simple RGB", that would make it harder 
for VSFilter style rendering to produce correct colors, because VSFilter 
renders the subtitles in YCbCr directly onto the video stream, and VSFilter 
doesn't know which video matrix the video player will use to convert the video 
to RGB later in the processing chain. Of course VSFilter could guess, but it 
simply doesn't have the necessary information. E.g. if the video bitstream says 
the video is BT.709, VSFilter doesn't know about that.
(3) We want VSFilter (rendering in YCbCr) and XySubFilter (rendering in RGB) to 
produce identical colors in pretty much all situations.
(4) Subtitle authors want to create subtitles which are rendered correctly for 
the majority of users, and today still most people are using VSFilter. And 
VSFilter renders in YCbCr and not RGB.

In the end some people were in favor of simple straight RGB interpretation. I 
was originally one of them. But after a lot of arguing, we finally ended up 
with the "ycbcr matrix" ASS header field, which may require color correction.

However, please note that subtitle authors are allowed to use straight RGB by 
setting the yuv matrix to "none". So your favorite solution can already be used 
today. AFAIK Aegisub supports it, so does XySubFilter and madVR. However, in 
order to solve all of the above problems, we also support doing color 
correction, if it's needed. But then, the user is absolutely allowed to use the 
correct matrix. In that case the yuv matrix ASS header field will simply mark 
the matrix used during creation of the subtitle file, and no color correction 
will be necessary.

It's a bit late now to change the ASS standard to disallow YUV matrix color 
correction. The train left the station when we defined the new subtitle 
interface, which was many many months ago, and all the new software (latest 
Aegisub versions, XySubFilter, madVR etc) are already written based on what we 
had agreed on back then...

Practically, imagine the user switches madVR between different video matrixes 
during playback. With the current solution, doing so should still result in 
subtitles color matching video colors. This would not work if we used straight 
RGB interpretation. With straight RGB, subtitles would only be color matched to 
the video if Aegisub happened to use exactly the same matrix as the video 
renderer then uses during playback. In most cases the matrix should be the 
same, but it might not always be the case. So the current solution does have 
some practical advantages...

Original comment by mad...@gmail.com on 27 Jul 2013 at 11:08

[GoogleCodeExporter]

If you want it to look the same for everyone that's all the more reason to 
switch to RGB. Because let's face it, the majority of users use VLC with libass 
and AFAIK they do no color correction. Also, we have no plans to support this 
in the ISR in its current form.

My solution addresses all 4 points and I see no reason why we can't do it now 
just because it wasn't suggested months ago. It doesn't need any change in the 
functionality of the interface, this is purely about the ASS specification.

Original comment by Jules.B...@gmail.com on 27 Jul 2013 at 11:37

[GoogleCodeExporter]

I'm on my phone, so I have to keep it short, but you should really consider a 
post-VSFilter scenario. In the near future there will be no more good reason to 
use VSFilter and it will die out. In such a scenario convincing everyone to 
still do color correction by default for the few users that still use VSFilter 
is a bad idea.

Original comment by Jules.B...@gmail.com on 27 Jul 2013 at 11:59

[GoogleCodeExporter]

> it will always correct the color values unless explicitly
> specified that it doesn't have to

XySubFilter itself will currently only color correct TV.601 matrix scripts when 
the Consumer reports the video is using TV.709 matrix. This is the bare minimum 
required for compatibility with ASS scripts. madVR currently handles everything 
other possibility.

xy-VSFilter (VSFilter.dll) on the other hand uses YCbCr Matrix for all 
RGB->YCbCr output, which isn't color correction at all. madshi decided to 
support the full range of RGB color correction possibilities in madVR, so it 
would match the xy-VSFilter YCbCr output at all times. Other Subtitle Consumers 
are not necessarily required to go this far in matching xy-VSFilter output, 
IMHO.

> For this I propose a new version of the ASS standard

No, this is a bad idea. The ASS standard itself would need to be completely 
depreciated, and replaced with a new subtitle standard which is not backwards 
compatible with any existing ASS parsers. The current ASS standard is 
unfortunately based entirely around VSFilter behavior as the reference 
rasterizer, but it should remain that way until it dies. The only reasonable 
solution to the problem is to break compatibility with VSFilter with a new 
standard, and at that point, there is much more which is wrong, broken, and 
ill-planned about the ASS specification which would need to be fixed in any new 
subtitle format.

> I realize the importance of it, but subtitles in the ASS format should really 
just be 
> specified in RGB and we should encourage that by making it the default.

This encouragement could be easily done by just having Aegisub changing their 
default behavior for tagging YCbCr Matrix. Currently it forces BT.601 by 
default for VSFilter compatibility. If this option to for forcing BT.601 is 
disabled, scripts produced by Aegisub will always tag the real matrix of the 
video and be untouched RGB, without need of any color corrections. MPC-HC and 
MPC-BE are the main roadblocks here, since both projects continue to distribute 
BT.601-only legacy VSFilter.dll which do not support use of the YCbCr Matrix 
header for YCbCr video output like xy-VSFilter (VSFilter.dll) does. Until 
Aegisub is comfortable that xy-VSFilter has become the majority VSFilter.dll in 
use, or MPC-HC VSFilter supports the YCbCr Matrix header, I doubt they will be 
comfortable changing their default setting.

> Because let's face it, the majority of users use VLC with libass and AFAIK 
they do no color correction.

MPV (Mplayer/Mplayer2 fork) with Libass recently added support for our YCbCr 
Matrix header. Libass itself also supports reporting the YCbCr Matrix header. 
VLC has never been a favored media player to use with ASS scripts, and its 
video renderers have always been rather buggy on Windows, so overall it 
perfectly fine if they and other Libass implementations continue to do their 
own thing. Nothing will change for existing Libass implementations, since our 
header did not actually introduce any new types of scripts. Scripts now just 
have the benefit of being accurately tagged with the matrix they were authored 
with. The proper way to deal with this matrix information is common sense.

> Also, we have no plans to support this in the ISR in its current form.

My hope was that you would depreciate the ISR when you introduced your Subtitle 
Consumer for XySubFilter. There is nothing you need to support in your Subtitle 
Consumer other than reporting the video matrix used by the renderer to 
XySubFilter. As mentioned above, this will result in TV.601 scripts on TV.709 
being color corrected by XySubFilter, and every other combination else being 
untouched RGB. This is fine for more limited support IMHO, as it agrees with 
all scripts authored by Aegisub (except in cases of script portability). It's 
also the same limited color correction behavior I personally proposed to MPV 
and how their recent builds now behave by default with Libass.

> In the near future there will be no more good reason to use VSFilter and it 
will die out. 
> In such a scenario convincing everyone to still do color correction by 
default for the few 
> users that still use VSFilter is a bad idea.

As it stands today, the vast majority of scripts in circulation are all BT.601, 
without any simple way to fix them for RGB output except via YCbCr Matrix color 
correction support. If that ever changes in the future, and suddenly the 
majority of scripts become untouched RGB, we could easily adapt and push out an 
update to SubRenderIntf.h which changes our recommended behavior. The important 
part is our Subtitle Interface has the framework to support subtitle color 
correction if needed or desired. Outside of the subtitle interface, our YCbCr 
Matrix header is available for others to achieve the same thing.

Original comment by cyber.sp...@gmail.com on 28 Jul 2013 at 5:25

[GoogleCodeExporter]

We have no plans to replace the ISR. The current plan is to improve the 
performance and compliance of the ISR, likely with support of the matrix tag 
but I won't support a change that makes color correction default.  I was 
actually hoping you'd depreciate xy-VSFilter in favor of XySubFilter, because 
yours is the only VSFilter implementation still in wide usage. When XySubFilter 
is officially released there won't be any reason to use color correction and I 
believe fansubbers will quickly switch to RGB in which case I don't want them 
to have to set that YCbCr field to none just to get it to render normally.

So my suggestion is to release a new version where the behavior of using 
uncorrected values unless the YCbCr field is specified has been standardized. 
That's the only way I'll support making color correction default for old 
versions of the script files.

Original comment by Jules.B...@gmail.com on 28 Jul 2013 at 8:56

[GoogleCodeExporter]

> We have no plans to replace the ISR

Why stick to old code when XySubFilter is faster, has less bugs and an active 
developer? You just replaced the internal filters with LAV. IMHO that was a 
very good idea. In the same way it would make sense to replace the ISR with 
XySubFilter, especially if you're already in the process of making your 
renderers XySubFilter capable. Just my 2 cents, of course...

Or is there anything the ISR is better with compared to XySubFilter, once 
you've added support for the new subtitle interface to your internal renderers?

Original comment by mad...@gmail.com on 28 Jul 2013 at 9:14

[GoogleCodeExporter]

We've discussed it but eventually decided against it. Underground78 is working 
on performance improvements and I will improve compliance. We're committed to 
maintain our own subtitle renderer because we want te stay in control of the 
code and have expert knowledge on it within the team, but generally we want to 
do it just because it's fun. ;)

On top of that the ISR has support for the download subtitles dialog and drag 
and drop subtitle loading which are really important to us as they improve 
usability. We want to add support for that to XySubFilter, but I'll get back to 
that after I'm done implementing the current interface.

Original comment by Jules.B...@gmail.com on 28 Jul 2013 at 9:38

[GoogleCodeExporter]

Ok, I understand.

> When XySubFilter is officially released there won't
> be any reason to use color correction and I believe
> fansubbers will quickly switch to RGB in which case
> I don't want them to have to set that YCbCr field
> to none just to get it to render normally.

You only have to convince the Aegisub developer to disable compatability with 
old VSFilter releases (which means using BT.601 for HD files) by default. If 
you do that, both xy-VSFilter and XySubFilter will be able to render untouched 
native RGB in most cases, so will your ISR. The interface and the yuv matrix 
header field supports this. It's only a matter of Aegisub defaults.

Aegisub could either always write "None", or it could simply use the correct 
matrix and store that into the ASS file. In both cases xy-VSFilter and 
XySubFilter (and your ISR) could simply render straight RGB without any color 
correction, and subtitles would still be perfectly color matched.

There can be some weird situations where color correction might be necessary to 
achieve matched colors, but those are corner cases. madVR + XySubFilter will 
handle them correctly. Your ISR might not, if you only support the simple 
BT.601 correction for old ASS files. But those corner cases should be rare, so 
it should be ok for you to ignore those. We XySubFilter + madVR devs aim for 
perfection, though, so we worry about and bother with those corner cases, too, 
which is why the ASS yuv matrix field is still there...

Original comment by mad...@gmail.com on 28 Jul 2013 at 10:12

[GoogleCodeExporter]

> I was actually hoping you'd depreciate xy-VSFilter in favor of XySubFilter,
> because yours is the only VSFilter implementation still in wide usage.

We have no intentions of depreciating the functionality offered by xy-VSFilter 
(VSFilter.dll). 

XySubFilter is not a replacement for xy-VSFilter, they both serve different 
purposes.

> When XySubFilter is officially released there won't be any reason to use 
color correction

After the point xy-VSFilter added support for the YCbCr Matrix header, there 
was no longer any reason to use color correction, except for compatibility with 
other older versions of VSFilter like the one offered by MPC-HC. XySubFilter 
doesn't change anything in this regard. 

> So my suggestion is to release a new version where the behavior of using 
uncorrected 
> values unless the YCbCr field is specified has been standardized.

All scripts which are not tagged are legacy scripts, and the vast majority of 
these scripts were authored for the BT.601-only behavior of VSFilter. Ever 
since we added support for the YCbCr Matrix header, 100% of new scripts 
authored with Aegisub are all explicitly tagged.

xy-VSFilter & XySubFilter will continue to use BT.601 on legacy scripts 
indefinitely for compatibly reasons. Any script created today without the YCbCr 
Matrix header is no longer valid from our point of view.

> That's the only way I'll support making color correction default for old 
versions of the script files.

You are free to handle BT.601 legacy scripts without YCbCr Matrix tagged in 
your Consumer however you desire. XySubFilter can adapt and do the color 
correction on such scripts itself.

> We're committed to maintain our own subtitle renderer because we want te stay 
in control of the code 
> and have expert knowledge on it within the team, but generally we want to do 
it just because it's fun. ;)

Well it's up to you. Though I also believe that MPC-HC integrating XySubFilter 
+ subtitle interface and using it as a replacement for the core of the current 
ISR would have made much more sense from a duplication of labor perspective. 
There is no reason you couldn't retain control over the the code if your forked 
XySubFilter into your project.

> On top of that the ISR has support for the download subtitles dialog and drag 
and drop 
> subtitle loading which are really important to us as they improve usability.
> We want to add support for that to XySubFilter, but I'll get back to that 
after 
> I'm done implementing the current interface.

A few days ago MPC-BE added xy-VSFilter/XySubFilter support for these 
functions, integration with subtitle context menu and subtitle switching, as 
well as eliminated conflicts with both XySubFilter and the ISR being active at 
the same time. I hope some of these user friendly changes do eventually make 
there way into MPC-HC as well.

Original comment by cyber.sp...@gmail.com on 28 Jul 2013 at 10:13

[GoogleCodeExporter]

@cyber, what is the current Aegisub default? Does it write the *correct* matrix 
in the ASS file (e.g. BT.709 for HD files)? Or does it write BT.601 by default, 
to achieve compatability with non-xy VSFilter builds?

Original comment by mad...@gmail.com on 28 Jul 2013 at 10:16

[GoogleCodeExporter]

> Aegisub could either always write "None", or it could simply use 
> the correct matrix and store that into the ASS file.

The way we defined it, use of "YCbCr Matrix = None" was intended for RGB videos 
only. When you disable the "Force BT.601" option in Aegisub, it will tag the 
actual YCbCr Matrix of the video, and this will result in untouched RGB 
subtitles which do not require color correction (by default).

Original comment by cyber.sp...@gmail.com on 28 Jul 2013 at 10:22

[GoogleCodeExporter]

> @cyber, what is the current Aegisub default?

The current default behavior in Aegisub is to force BT.601 whenever a YCbCr 
video is loaded. It does this to achieve compatibility with non-xy-VSFilter 
builds.

> Does it write the *correct* matrix in the ASS file (e.g. BT.709 for HD files)?

Only if you disable the "Force BT.601" option does this occurs.

Original comment by cyber.sp...@gmail.com on 28 Jul 2013 at 10:25

[GoogleCodeExporter]

So once XySubFilter is out of beta and working fine, and supported by the major 
media players, maybe we could talk to the Aegisub developer to disable the 
"BT.601" compatability switch by default?

Users who still prefer VSFilter can switch to xy-VSFilter - problem solved. 
xy-VSFilter is better, anyway...

Original comment by mad...@gmail.com on 28 Jul 2013 at 10:59

[GoogleCodeExporter]

> xy-VSFilter & XySubFilter will continue to use BT.601 on legacy scripts 
indefinitely for compatibly reasons. Any script created today without the YCbCr 
Matrix header is no longer valid from our point of view.

Then why not make that distinction between new and old scripts official by 
defining a new version number to signify what scripts are "new"? This is just 
common sense in software development: you add new functionality, you increase 
the version number.

> After the point xy-VSFilter added support for the YCbCr Matrix header, there 
was no longer any reason to use color correction, except for compatibility with 
other older versions of VSFilter like the one offered by MPC-HC. XySubFilter 
doesn't change anything in this regard.

Ah it seems I misunderstood something, do you mean that ASS files tagged with 
YCbCr matrix contain untouched RGB which XySubFilter will display without 
correction? That makes the situation less grave, since I was under the 
impression the field specified how to convert the corrected values back to RGB.

Original comment by Jules.B...@gmail.com on 28 Jul 2013 at 10:59

[GoogleCodeExporter]

> Ah it seems I misunderstood something, do you mean
> that ASS files tagged with YCbCr matrix contain
> untouched RGB which XySubFilter will display without
> correction?

Yes, as long as the YCbCr Matrix stored in the ASS file matches the matrix used 
by the video renderer - which *should* be the case in all but a few corner 
cases.

However, the above only applies if the Aegisub "Force BT.601" compatability 
option is turned off.

Original comment by mad...@gmail.com on 28 Jul 2013 at 11:03

[GoogleCodeExporter]

> Then why not make that distinction between new and old scripts official
> by defining a new version number to signify what scripts are "new"?
> This is just common sense in software development: you add new functionality,
> you increase the version number.

Since it's really only an information field that doesn't effect functionality 
directly, it will likely be formally defined as a addendum to the current ASS 
specification. From what I remember, Aegisub was strongly against increasing 
the version number or adding anything additional to scripts for identification 
purposes. It was determined that the existence of "YCbCr Matrix" was enough to 
identify "new" from "old".

Original comment by cyber.sp...@gmail.com on 28 Jul 2013 at 11:39

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

For software standards a version number is more than an information field, it 
defines according to which standard the rest of the file is interpreted. I 
guess I'll take up the discussion with Aegisub then.

Original comment by Jules.B...@gmail.com on 28 Jul 2013 at 11:48

[GoogleCodeExporter]

I made a Pull Request for XySubFilter support in MPC-HC, feel free to test it 
out and provide feedback on the changes.

https://github.com/mpc-hc/mpc-hc/pull/114

Original comment by Jules.B...@gmail.com on 3 Oct 2013 at 1:20

[GoogleCodeExporter]

I'm going to finish that branch soon, but I'd also like to test a 64-bit build 
of XySubFilter and users will want such a build when support is added to EVR-CP 
and MVR9.

Also, could you remove the MPC-HC workaround when joining the filter graph? It 
hasn't been necessary ever since we added the exception for XySubFilter to the 
CleanGraph() function and it is just making things hard to debug at this point. 
Don't make me work around a workaround. ;)

Original comment by Jules.B...@gmail.com on 4 Nov 2013 at 8:32

[GoogleCodeExporter]

It is also time we finally decided on how to do auto-loading. There have been a 
couple of different suggestions and different test versions. And I hope that 
one of them can replace the current logic of having the video renderer load the 
subtitle renderer.

Original comment by mad...@gmail.com on 4 Nov 2013 at 8:44

[GoogleCodeExporter]

I would like it to be handled by DirectShow, but that means we will have to 
split the rendering code and the reading code. It also means every subtitle 
format needs to be supported by the media player. But without media player 
support the current method of auto-loading that madVR uses is the only way to 
do it. We could just have both methods: Players can either provide a subtitle 
pin or auto-load XySubFilter.

What we still need from the interface in that case are two things:

1. A flag to control the subtitle sources: If the media player provides 
subtitle pins for external subtitles then it should be possible to indicate 
that and prevent XySubFilter from loading them again creating duplicate entries.

2. Subtitle invalidation support: If the subtitle stream has changed or the 
subtitles have become invalid for any other reason the subtitle consumer needs 
to be notified of this so it can clear it's buffers.

Original comment by Jules.B...@gmail.com on 4 Nov 2013 at 8:58

[GoogleCodeExporter]

Correction: If we support both methods then the rendering code and the reading 
code doesn't need to be split. It only means every subtitle format needs to be 
supported by the media player and we need a way to control which subtitle 
sources XySubFilter will load.

Original comment by Jules.B...@gmail.com on 4 Nov 2013 at 9:01

[GoogleCodeExporter]

There were 2 different test builds with a different auto-loading method, which 
worked somewhat similar to the current solution, but without requiring the 
video renderer to do anything. But there were some advantages and disadvantages 
to every of those solutions. So I'm not sure what the latest word is...

Yes, we definitely need a way to invalidate/flush a subtitle connection.

I'm wondering if yuzhuohuang and cyber are still listening? Haven't heard from 
either one in several weeks now.

Original comment by mad...@gmail.com on 4 Nov 2013 at 9:06

[GoogleCodeExporter]

One thing I still fail to understand is whether the "yuvMatrix" field for the 
consumer should be set to the source or the target transfer matrix. Maybe you 
can shed some light on that, madshi?

Original comment by Jules.B...@gmail.com on 13 Nov 2013 at 10:20

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

The header file says:

// When rendering native YCbCr subtitles (DVD and Blu-Ray bitmap subtitles),
// the subtitle renderer should try to use the correct YCbCr -> RGB matrix.
// The subtitle renderer may either use the "yuvMatrix" information field
// which the subtitle consumer may optionally provide. Or alternatively the
// subtitle renderer can auto guess the correct matrix based on subtitle
// resolution.

In this situation (native YCbCr video and native YCbCr subtitles) both the 
video renderer and the subtitle renderer perform YCbCr -> RGB conversion. In 
order to get the best possible color matching, both renderers should use the 
same conversion matrix. The end user might switch the video renderer manually 
to a different matrix. E.g. if the video has SD resolution, but for some reason 
was encoded with BT.709 matrix (e.g. a custom downscale encode from a Blu-Ray), 
the user might switch the video renderer manually to BT.709. In this situation 
it would be nice if the subtitle renderer used BT.709, too, for the YCbCr -> 
RGB conversion of the YCbCr bitmap subtitles. So the video renderer should set 
the "yuvMatrix" field to the matrix used for video YCbCr -> RGB conversion, so 
that the subtitle renderer can use the same matrix.

Original comment by mad...@gmail.com on 13 Nov 2013 at 10:38

[GoogleCodeExporter]

But according to the following docs there are two kinds of these transfer 
matrices, the source and the target:

http://msdn.microsoft.com/en-us/library/windows/desktop/bb970322%28v=vs.85%29.as
px

"3. YUV to RGB conversion: Convert from Y'PbPr to non-linear R'G'B', using the 
source transfer matrix."

"7. RGB to YUV conversion: Convert R'G'B' to Y'PbPr, using the target transfer 
matrix."

Which one should be reported?

Original comment by Jules.B...@gmail.com on 13 Nov 2013 at 10:48

[GoogleCodeExporter]

Oh wait, I now see the direction, so it is the source transfer matrix.

Original comment by Jules.B...@gmail.com on 13 Nov 2013 at 10:56

[GoogleCodeExporter]

The YUV to RGB conversion is the one of interest here (named "source transfer 
matrix" by that web link).

Original comment by mad...@gmail.com on 13 Nov 2013 at 11:00

[GoogleCodeExporter]

Cross-post. Yes, you got it.

Original comment by mad...@gmail.com on 13 Nov 2013 at 11:00

[GoogleCodeExporter]

Okay, so that is not information stored in the media type of the renderer, 
that's the target if I understand correctly. Which means I need to travel 
upstream to the decoder and query the color information there to get the source 
transfer matrix.

Original comment by Jules.B...@gmail.com on 13 Nov 2013 at 11:03

[GoogleCodeExporter]

It actually could be in the media type of the renderer, depending on which 
decoder you're using and whether that decoder has your renderer listed as 
"supports matrix information in the media type". E.g. LAV Video Decoder does 
report the matrix to madVR via VIDEOINFOHEADER2.dwControlFlags. I'm not sure if 
LAV reports it to any other video renderer, though. IIRC there were some 
problems with other video renderers. You'd have to talk to nevcairiel about 
this.

However, the key question is: Which matrix do your video renderers actually 
use? Unlike madVR, most other renderers do not ask the decoder about this at 
all. They simply best guess, based on video resolution. I don't know how the 
MPC-HC video renderers do that. But my best guess is that they just guess the 
matrix like SD res = 601, HD res = 709. You'd have to check the source code to 
find out for sure. Or maybe your renderers leave this to Microsoft? Then 
probably Microsoft selects based on guessing. But I don't really know that. I 
don't even know how your renderers perform the color conversion? Do you let 
StretchRect do YCbCr -> RGB conversion for you? Or do you use DXVA for that? I 
don't know. madVR does this totally differently by using custom pixel shader 
code. So I have full control over the exact color conversion. When using DXVA, 
you can use flags to ask for a specific matrix to be used. But some GPUs ignore 
that (e.g. Intel). When using StretchRect, you have no way to select the 
matrix, the GPU driver will do that based on guessing.

Original comment by mad...@gmail.com on 13 Nov 2013 at 11:10

[GoogleCodeExporter]

Currently I'm using the mediatype in the renderer, which according to the 
following page describes the output:

http://msdn.microsoft.com/en-us/library/windows/desktop/ms697032%28v=vs.85%29.as
px

In the case of EVR-CP with default settings, this is BT.709 with PC levels. To 
get the source transfer matrix I need to retrieve the mediatype from LAV. Or 
are there more mediatypes known in the renderer that I am missing?

Original comment by Jules.B...@gmail.com on 13 Nov 2013 at 11:27

[GoogleCodeExporter]

I'm guessing I best try the mediatype received on pin connection?

Original comment by Jules.B...@gmail.com on 13 Nov 2013 at 11:37

[GoogleCodeExporter]

I don't know where EVR gathers the information for IMFVideoMediaType, but I 
would guess it's all based on the mediatype sent by the decoder, probably with 
missing information guessed by EVR. But I've never really looked into how EVR 
works, so I can only guess.

The key question for you is probably how to find out which matrix to report in 
"yuvMatrix". I don't know how to answer that. It probably depends on which 
renderer we're talking about (VMR7, VMR9, EVR, EVR-Custom etc), whether DXVA is 
used or not, which decoder we're talking about etc.

IMHO the best solution would be for the decoder to report the matrix (if the 
decoder knows it) via the decoder's output pin matrix, and then you could 
report this matrix as "yuvMatrix". However, this only makes sense if this 
matrix is then actually really used by the renderer! Which you may or may not 
have control over, when using the MS renderers. I don't know this in detail. 
Another possible solution would be to try to ask the MS renderers which matrix 
they're using. I don't know if they're telling you that. I'm probably the wrong 
guy to ask these questions, because I don't know a thing about how the MPC-HC 
video renderers work. Or the MS renderers, for that matter.

If you want to do this perfectly, you may have to research how your renderers 
are doing color conversion (DXVA, StretchRect, something else), and then to 
research which matrix is used in which circumstances. Unfortunately this may 
even differ for different GPU manufacturers or OSs...  :-(  Maybe you know 
someone who already knows all that? You could ask JanWillem, maybe he knows, 
considering that he's invested a lot of work into a custom video renderer...

BTW, I'm just thinking: You may have some "fun" ahead of you, trying to 
implement support for BT.2020 matrix, which Europe 4K broadcasting will 
probably be using. DXVA doesn't support it. So it might be tricky to make this 
work with EVR/VMR.

Original comment by mad...@gmail.com on 13 Nov 2013 at 11:51

[GoogleCodeExporter]

For those interested, here's a new version of the header:

http://madshi.net/SubRenderIntf.h

I've added a "Clear()" method which allows the subtitle provider to ask the 
consumer to clear its subtitle queue. This can be useful e.g. if the user 
switches to another subtitle track. This feature was requested by several 
developers, so I've added it now. Hope everybody is ok with this?

Original comment by mad...@gmail.com on 11 Mar 2014 at 2:56

[GoogleCodeExporter]

I will implement support for the new "Clear()" method in MPC-HC soon.

Original comment by Jules.B...@gmail.com on 2 Apr 2014 at 9:41

[GoogleCodeExporter]

Great, thanks. FWIW, madVR already supports it. But AFAIK there's no 
XySubFilter version available yet which actually uses it.

Original comment by mad...@gmail.com on 2 Apr 2014 at 9:44

[GoogleCodeExporter]

madshi, you also changed the auto-loading responsibility in version 1.0.7. Are 
you sure the provider can do this without ugly hacks? We were actually planning 
to implement an option dialog where you can select which subtitle provider you 
want to auto-load similar to the output renderers.

Original comment by Jules.B...@gmail.com on 3 Apr 2014 at 8:03

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

We found a way to auto-load XySubFilter without the help of the consumer. The 
"XySubFilterAutoLoader" does all the work. You can disable XySubFilter 
auto-loading in MPC-HC by blocking the "XySubFilterAutoLoader" filter. Overall 
this is an improvement because you have now full control over the auto-load 
behaviour in the media player. That wasn't possible with the old solution.

Having an option in the media player which controls which subtitle renderer is 
used (both for internal tracks and for external tracks = auto-loading) would be 
great, of course! It should be possible with the latest concept.

If you have wishes to change the behaviour somehow, we can discuss it. We 
removed the old consumer-based auto-loading due to feedback from several other 
developers who all disliked the old solution.

Original comment by mad...@gmail.com on 3 Apr 2014 at 8:16

[GoogleCodeExporter]

XySubFilterAutoLoader doesn't seem to work when adding it as an external filter 
rather than registering it on the system. How does this filter work exactly?

Original comment by Jules.B...@gmail.com on 3 Apr 2014 at 8:45

[GoogleCodeExporter]

IIRC it registers for all video formats with a high merit, so that it's loaded 
by DirectShow for any video file. However, it always refuses to actually accept 
a pin connection, because of which the graph builder automatically removes it 
again from the graph. But since it was loaded in the graph (for a short time) 
it had the chance to look for external subtitles and if some are found it 
manually adds XySubFilter to the graph.

At least that was the original concept, which I had suggested to replace the 
old auto-loading solution. I think yuzhuohuang made some changes. IIRC he might 
be registering for audio tracks now instead of video tracks. But I'm not sure...

I guess the filter needs to be registered to work. Maybe alternatively you 
could manually add it to the graph and then remove it again without connecting 
it to anything. That might have the same effect.

Original comment by mad...@gmail.com on 3 Apr 2014 at 8:56

[GoogleCodeExporter]

Support for "Clear()" has been merged in MPC-HC. The code was simple, but it 
was sitting on my development fork for a long time.

Original comment by Jules.B...@gmail.com on 10 May 2014 at 8:52

[GoogleCodeExporter]

Great!

So what are your thoughts about the new auto-loading behaviour? Is it alright 
with you?

Original comment by mad...@gmail.com on 11 May 2014 at 5:44

[GoogleCodeExporter]

I think it's alright, though I would prefer it connected only to video tracks 
instead of audio tracks. When playing audio files there is no video renderer to 
connect to, so it really shouldn't even try to.

I'm still considering what to do about the current situation with external 
subtitle providers. I'm considering allowing you to select the subtitle 
provider in the subtitle options just like for the video renderers. That way 
it's more clear to the user and we can do away with the subtitle provider 
blocking logic. (or generalize it anyway)

Though I'm not sure how we would detect the subtitle providers installed on the 
system. We'll probably have to keep an internal list of subtitle provider UUIDs.

Original comment by Jules.B...@gmail.com on 12 May 2014 at 10:10

[GoogleCodeExporter]

> I think it's alright, though I would prefer it connected only to video tracks
> instead of audio tracks.

It doesn't actually form a connection on video or audio tracks, it only waits 
for the DirectShow graph builder to attempt a connection on a non-video pin, as 
to not prevent VSFilter from being loaded. The autoloader is capable of 
removing VSFilter from the graph if it decides to load XySubFilter, so not 
blocking VSFilter outright is beneficial as a fallback.

Only when XySubFilter is set to "Always Load" does the autoloader load 
XySubFilter immediately, without waiting for a non-video pin.

> When playing audio files there is no video renderer to connect to, so it 
really shouldn't even try to.

FWIW, even VSFilter.dll is loaded into the graph on audio-only files. Both 
utilize a MEDIATYPE_NULL (Any type) pin to enter the graph. As mentioned above, 
the main difference is that the XySubFilter autoloader ignores video pins by 
default, while VSFilter autoloader does not. This allows both autoloaders to 
enter the graph, even though one has a much higher DirectShow Merit than the 
other.

We are not adverse to adjusting the behavior of the autoloader if someone comes 
up with a better idea, but we do not wish to give up graceful co-existence with 
legacy VSFilter.dll in the process, nor require explicit media player support 
for external subtitles to function at all.

Original comment by cyber.sp...@gmail.com on 12 May 2014 at 11:31

[GoogleCodeExporter]

I think one thing is important: If the media player devs add explicit support 
for external subtitle tracks via XySubFilter, that should be possible without 
interference from the auto-load helper. But I think this should already be 
possible right now, simply by blocking the auto-load helper, right?

@Jules, having a subtitle renderer option would be quite awesome, IMHO. 
Currently disabling/enabling the ISR can be difficult/confusing, when one wants 
to use XySubFilter. If MPC-HC would let the user choose and then automatically 
do all the necessary adjustments to let the user select between e.g. ISR, 
VSFilter, xy-VSFilter and XySubFilter, that would be wonderful. Of course 
MPC-HC should then also support all the extra features like drag & drop of 
external subtitle tracks, auto-loading etc for all supports subtitle renderers. 
This would greatly improve usability, IMHO. If there are changes in 
XySubFilter/xy-VSFilter necessary to make this work, I'm sure yuzhuohuang would 
be willing to implement them.

Original comment by mad...@gmail.com on 13 May 2014 at 6:15