Custom interface for rendering subtitles on an RGBA texture(s) | (XySubFilter for madVR) [Part 3]

[GoogleCodeExporter]

Part 1: http://code.google.com/p/xy-vsfilter/issues/detail?id=40

Part 2: http://code.google.com/p/xy-vsfilter/issues/detail?id=91

Topic to continue discussions about the subtitle interface we created along 
with madshi, which is now supported by madVR as well as xy-VSFilter in the form 
of XySubFilter.

Latest revision of the new subtitle interface:
http://madshi.net/SubRenderIntf.h

Original issue reported on code.google.com by cyber.sp...@gmail.com on 16 Jul 2013 at 6:50

[GoogleCodeExporter]

I don't like how the YUV Matrix is being used by XySubFilter. As I understand 
ot, it will always correct the color values unless explicitly specified that it 
doesn't have to. I realize the importance of it, but subtitles in the ASS 
format should really just be specified in RGB and we should encourage that by 
making it the default.

For this I propose a new version of the ASS standard where RGB values that 
compensate for the YUV matrix are deprecated and will be displayed without 
color correction unless the yuv matrix field is explicitly specified in the 
subtitle file. This new version would therefore also make the yuv matrix field 
an official part of the standard. Files that still specify that they're written 
in an older ASS version will still be color corrected by default as they are 
now.

This way we have a nice clean standard without the complicated color correction 
while we still maintain support for correctly displaying old subtitles with 
color correction.

What are your opinions on this?

Original comment by Jules.B...@gmail.com on 27 Jul 2013 at 9:24

[GoogleCodeExporter]

@Jules, we had discussed all this back in the day when we all tried to agree on 
how to solve the color matrix thing. Discussions were done between all 
interested parties, invitations to the MPC-HC team and other media player 
developers to join the dicussion were also issued, IIRC.

The key problems are that:

(1) 99.9% of all HD ASS files out there require color correction to look as 
intended by the subtitle author.
(2) If new subtitles were written in "simple RGB", that would make it harder 
for VSFilter style rendering to produce correct colors, because VSFilter 
renders the subtitles in YCbCr directly onto the video stream, and VSFilter 
doesn't know which video matrix the video player will use to convert the video 
to RGB later in the processing chain. Of course VSFilter could guess, but it 
simply doesn't have the necessary information. E.g. if the video bitstream says 
the video is BT.709, VSFilter doesn't know about that.
(3) We want VSFilter (rendering in YCbCr) and XySubFilter (rendering in RGB) to 
produce identical colors in pretty much all situations.
(4) Subtitle authors want to create subtitles which are rendered correctly for 
the majority of users, and today still most people are using VSFilter. And 
VSFilter renders in YCbCr and not RGB.

In the end some people were in favor of simple straight RGB interpretation. I 
was originally one of them. But after a lot of arguing, we finally ended up 
with the "ycbcr matrix" ASS header field, which may require color correction.

However, please note that subtitle authors are allowed to use straight RGB by 
setting the yuv matrix to "none". So your favorite solution can already be used 
today. AFAIK Aegisub supports it, so does XySubFilter and madVR. However, in 
order to solve all of the above problems, we also support doing color 
correction, if it's needed. But then, the user is absolutely allowed to use the 
correct matrix. In that case the yuv matrix ASS header field will simply mark 
the matrix used during creation of the subtitle file, and no color correction 
will be necessary.

It's a bit late now to change the ASS standard to disallow YUV matrix color 
correction. The train left the station when we defined the new subtitle 
interface, which was many many months ago, and all the new software (latest 
Aegisub versions, XySubFilter, madVR etc) are already written based on what we 
had agreed on back then...

Practically, imagine the user switches madVR between different video matrixes 
during playback. With the current solution, doing so should still result in 
subtitles color matching video colors. This would not work if we used straight 
RGB interpretation. With straight RGB, subtitles would only be color matched to 
the video if Aegisub happened to use exactly the same matrix as the video 
renderer then uses during playback. In most cases the matrix should be the 
same, but it might not always be the case. So the current solution does have 
some practical advantages...

Original comment by mad...@gmail.com on 27 Jul 2013 at 11:08

[GoogleCodeExporter]

If you want it to look the same for everyone that's all the more reason to 
switch to RGB. Because let's face it, the majority of users use VLC with libass 
and AFAIK they do no color correction. Also, we have no plans to support this 
in the ISR in its current form.

My solution addresses all 4 points and I see no reason why we can't do it now 
just because it wasn't suggested months ago. It doesn't need any change in the 
functionality of the interface, this is purely about the ASS specification.

Original comment by Jules.B...@gmail.com on 27 Jul 2013 at 11:37

[GoogleCodeExporter]

I'm on my phone, so I have to keep it short, but you should really consider a 
post-VSFilter scenario. In the near future there will be no more good reason to 
use VSFilter and it will die out. In such a scenario convincing everyone to 
still do color correction by default for the few users that still use VSFilter 
is a bad idea.

Original comment by Jules.B...@gmail.com on 27 Jul 2013 at 11:59

[GoogleCodeExporter]

> it will always correct the color values unless explicitly
> specified that it doesn't have to

XySubFilter itself will currently only color correct TV.601 matrix scripts when 
the Consumer reports the video is using TV.709 matrix. This is the bare minimum 
required for compatibility with ASS scripts. madVR currently handles everything 
other possibility.

xy-VSFilter (VSFilter.dll) on the other hand uses YCbCr Matrix for all 
RGB->YCbCr output, which isn't color correction at all. madshi decided to 
support the full range of RGB color correction possibilities in madVR, so it 
would match the xy-VSFilter YCbCr output at all times. Other Subtitle Consumers 
are not necessarily required to go this far in matching xy-VSFilter output, 
IMHO.

> For this I propose a new version of the ASS standard

No, this is a bad idea. The ASS standard itself would need to be completely 
depreciated, and replaced with a new subtitle standard which is not backwards 
compatible with any existing ASS parsers. The current ASS standard is 
unfortunately based entirely around VSFilter behavior as the reference 
rasterizer, but it should remain that way until it dies. The only reasonable 
solution to the problem is to break compatibility with VSFilter with a new 
standard, and at that point, there is much more which is wrong, broken, and 
ill-planned about the ASS specification which would need to be fixed in any new 
subtitle format.

> I realize the importance of it, but subtitles in the ASS format should really 
just be 
> specified in RGB and we should encourage that by making it the default.

This encouragement could be easily done by just having Aegisub changing their 
default behavior for tagging YCbCr Matrix. Currently it forces BT.601 by 
default for VSFilter compatibility. If this option to for forcing BT.601 is 
disabled, scripts produced by Aegisub will always tag the real matrix of the 
video and be untouched RGB, without need of any color corrections. MPC-HC and 
MPC-BE are the main roadblocks here, since both projects continue to distribute 
BT.601-only legacy VSFilter.dll which do not support use of the YCbCr Matrix 
header for YCbCr video output like xy-VSFilter (VSFilter.dll) does. Until 
Aegisub is comfortable that xy-VSFilter has become the majority VSFilter.dll in 
use, or MPC-HC VSFilter supports the YCbCr Matrix header, I doubt they will be 
comfortable changing their default setting.

> Because let's face it, the majority of users use VLC with libass and AFAIK 
they do no color correction.

MPV (Mplayer/Mplayer2 fork) with Libass recently added support for our YCbCr 
Matrix header. Libass itself also supports reporting the YCbCr Matrix header. 
VLC has never been a favored media player to use with ASS scripts, and its 
video renderers have always been rather buggy on Windows, so overall it 
perfectly fine if they and other Libass implementations continue to do their 
own thing. Nothing will change for existing Libass implementations, since our 
header did not actually introduce any new types of scripts. Scripts now just 
have the benefit of being accurately tagged with the matrix they were authored 
with. The proper way to deal with this matrix information is common sense.

> Also, we have no plans to support this in the ISR in its current form.

My hope was that you would depreciate the ISR when you introduced your Subtitle 
Consumer for XySubFilter. There is nothing you need to support in your Subtitle 
Consumer other than reporting the video matrix used by the renderer to 
XySubFilter. As mentioned above, this will result in TV.601 scripts on TV.709 
being color corrected by XySubFilter, and every other combination else being 
untouched RGB. This is fine for more limited support IMHO, as it agrees with 
all scripts authored by Aegisub (except in cases of script portability). It's 
also the same limited color correction behavior I personally proposed to MPV 
and how their recent builds now behave by default with Libass.

> In the near future there will be no more good reason to use VSFilter and it 
will die out. 
> In such a scenario convincing everyone to still do color correction by 
default for the few 
> users that still use VSFilter is a bad idea.

As it stands today, the vast majority of scripts in circulation are all BT.601, 
without any simple way to fix them for RGB output except via YCbCr Matrix color 
correction support. If that ever changes in the future, and suddenly the 
majority of scripts become untouched RGB, we could easily adapt and push out an 
update to SubRenderIntf.h which changes our recommended behavior. The important 
part is our Subtitle Interface has the framework to support subtitle color 
correction if needed or desired. Outside of the subtitle interface, our YCbCr 
Matrix header is available for others to achieve the same thing.

Original comment by cyber.sp...@gmail.com on 28 Jul 2013 at 5:25

[GoogleCodeExporter]

We have no plans to replace the ISR. The current plan is to improve the 
performance and compliance of the ISR, likely with support of the matrix tag 
but I won't support a change that makes color correction default.  I was 
actually hoping you'd depreciate xy-VSFilter in favor of XySubFilter, because 
yours is the only VSFilter implementation still in wide usage. When XySubFilter 
is officially released there won't be any reason to use color correction and I 
believe fansubbers will quickly switch to RGB in which case I don't want them 
to have to set that YCbCr field to none just to get it to render normally.

So my suggestion is to release a new version where the behavior of using 
uncorrected values unless the YCbCr field is specified has been standardized. 
That's the only way I'll support making color correction default for old 
versions of the script files.

Original comment by Jules.B...@gmail.com on 28 Jul 2013 at 8:56

[GoogleCodeExporter]

> We have no plans to replace the ISR

Why stick to old code when XySubFilter is faster, has less bugs and an active 
developer? You just replaced the internal filters with LAV. IMHO that was a 
very good idea. In the same way it would make sense to replace the ISR with 
XySubFilter, especially if you're already in the process of making your 
renderers XySubFilter capable. Just my 2 cents, of course...

Or is there anything the ISR is better with compared to XySubFilter, once 
you've added support for the new subtitle interface to your internal renderers?

Original comment by mad...@gmail.com on 28 Jul 2013 at 9:14

[GoogleCodeExporter]

We've discussed it but eventually decided against it. Underground78 is working 
on performance improvements and I will improve compliance. We're committed to 
maintain our own subtitle renderer because we want te stay in control of the 
code and have expert knowledge on it within the team, but generally we want to 
do it just because it's fun. ;)

On top of that the ISR has support for the download subtitles dialog and drag 
and drop subtitle loading which are really important to us as they improve 
usability. We want to add support for that to XySubFilter, but I'll get back to 
that after I'm done implementing the current interface.

Original comment by Jules.B...@gmail.com on 28 Jul 2013 at 9:38