Custom interface for rendering subtitles on an RGBA texture(s) | (XySubFilter for madVR) [Part 2]

[GoogleCodeExporter]

Original Issue: http://code.google.com/p/xy-vsfilter/issues/detail?id=40

Original issue reported on code.google.com by cyber.sp...@gmail.com on 30 Apr 2012 at 10:22

• Merged into: #152

[GoogleCodeExporter]

Issue 40 has been merged into this issue.

Original comment by cyber.sp...@gmail.com on 30 Apr 2012 at 10:25

[GoogleCodeExporter]

> I'm worried that xy-VSFilter will use a matrix
> which is different than that which was used to
> define the RGB colors, when outputting subtitles
> as YCbCr.

Can you help me out here? I'm not sure how Aegisub works inside. Does it also 
use the external VSFilter? If so, there are 2 potentially different matrixes 
involved during creation of an ASS file: The one used by VSFilter (RGB -> 
YCbCr) and the one used by Aegisub (YCbCr -> RGB). Correct? In this case by 
making sure that xy-vsfilter will use the same matrix that VSFilter used in 
Aegisub will only be a partial solution. The VSFilter matrix could have been 
wrong. The Aegisub matrix could have been wrong. But could have been wrong. The 
xy-vsfilter matrix could be wrong during playback. The video renderer's matrix 
could be wrong during playback. Basically we have *four* matrixes involved 
here, two during creation of the ASS file and two during final playback. Do I 
get this right? If so, then the only way to correct all possible problems would 
be for xy-vsfilter to get access to all other 3 matrixes to calculate the 
matrix is should use. I think this is what we named "triple correction" earlier.

Anyway. I think we should first discuss the following questions:

(1) Can we limit ourselves to only support new ASS files that were created with 
VSFilter and Aegisub both using the correct matrix? Or do you we have to 
support ASS files which were created with either VSFilter or Aegisub (or both) 
using a wrong matrix? (Of course those old hard coded BT.601 ASS files are a 
special case which we have to support, we all agree on that.)

(2) Can we limit ourselves to only support decoders and video renderers which 
are using the correct matrix during playback?

If we can limit both (1) and (2) then there's no need to store matrix 
information in the ASS file. Agreed? So cyber, it seems you want to support a 
wrong matrix somewhere? Or am I misunderstanding you?

Original comment by mad...@gmail.com on 30 Apr 2012 at 10:29

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

madshi, I don't exactly know, but I think in the original thread, it was hinted 
somewhere that AegiSub converts the video to RGB first and then uses VSFilter 
to overlay the subtitles onto the picture. So, while VSFilter would still be 
used, the RGB conversion would have been performed using the (sometimes 
guessed) correct matrix with the subtitles themselves being in pure RGB.

Original comment by TheDarkS...@googlemail.com on 30 Apr 2012 at 11:26

[GoogleCodeExporter]

Aegisub works almost exclusively with RGB video - the only time it actually 
sees YUV video data at all is when using the y4m reader, and the first thing 
that's done there is convert the video to RGB. As such, VSFilter is passed RGB 
video and does no colorspace conversions. libass is passed no video at all and 
Aegisub blends the ARGB subtitles into the video itself.

Perhaps one solution to the unsupported color matrix issue would be to make 
Aegisub never guess the color matrix, and always use the backwards compatible 
forced BT.601 mode if the video is not explicitly flagged with a recognized 
color matrix. Wouldn't this completely eliminate the case that cyberbeing is 
worried about?

Original comment by tgoyne on 30 Apr 2012 at 11:47

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

> Wouldn't this completely eliminate the case that cyberbeing is worried about?

Yes it would, at least for VSFilter.

The main problem would then be if Libass implementations refuse to implement 
this with YCbCr/RGB output. I was hopeful we could come to a solution which 
enabled VSFilter to usually use "correct" matrix without any changes required 
to Libass implementations. 

We would be stuck with BT.601 subtitles continued to being used on the majority 
of videos. It's unfortunate, but the majority of user encoded video never had 
the matrix specified when encoding.

Maybe it's about time we include the MPlayer/MPlayer2 and VLC developers in 
these discussions?

Original comment by cyber.sp...@gmail.com on 1 May 2012 at 12:37

[GoogleCodeExporter]

Doesn't libass already use the correct video matrix to do RGB->YCbCr 
conversion? (I don't know personally, but that's the impression I got from 
previous discussion.) If so, I don't see any compatibility problems with the 
new approach: both libraries would use the correct colorspace. Old subtitles 
that assume the conversion is always with BT.601 would be still be broken in 
libass, but that can't be helped.

Original comment by yuriks...@gmail.com on 1 May 2012 at 12:59

[GoogleCodeExporter]

Libass outputs untouched RGB, which currently means colors are incorrect for 
the majority of scripts which are tuned for VSFilter (BT.601) compatibility.

Original comment by cyber.sp...@gmail.com on 1 May 2012 at 1:05

[GoogleCodeExporter]

If libass outputs pure RGB, then what's the problem? Correct colors in the ASS 
will produce correct colors in the output. Broken (aka legacy-VSF compatible) 
will still be wrong just like it is now, but we can't do anything about that. 
(Short of adding the hack to libass with heuristics for detection, aka, the 
reverse of what is being proposed here, but that's a discussion that doesn't 
belong here IMO.)

Ultimately, it's a problem that will fix itself when correct-color ass files 
become the standard, without any change in libass.

Original comment by yuriks...@gmail.com on 1 May 2012 at 1:13

[GoogleCodeExporter]

> Ultimately, it's a problem that will fix itself when correct-color ass
> files become the standard, without any change in libass.

With what tgoyne just proposed, it will only be fixed for videos with matrix 
information contained in the bitstream.

Videos _without_ matrix info in the bitstream are the majority.

Videos _with_ matrix info in the bitstream are uncommon.

If everybody else agrees this is an acceptable solution, I'm okay with it. It 
just means we will match Libass less often than if we explicitly tagged the 
matrix used by Aegisub in the script. 

It's no worse off and slightly better than the current situation, but I was 
hopeful for a solution which bridges the gap between VSFilter and Libass more.

Original comment by cyber.sp...@gmail.com on 1 May 2012 at 1:24

[GoogleCodeExporter]

>Videos _without_ matrix info in the bitstream are the majority.
>Videos _with_ matrix info in the bitstream are uncommon.

Then let the presenter keep guessing. Then fetch that guess from it. If the 
presenter guessed incorrectly you'll have the incorrect guess and both wrong 
guesses will cancel themselves in the end. Or actually, make people damn tag 
their bitstreams correctly if they want correct video (not sub, *video*) colors 
from now on.

Original comment by yuriks...@gmail.com on 1 May 2012 at 1:31

[GoogleCodeExporter]

I apologize yuriks.br, but as much as I would like to explain everything to 
you, I'm going to ignore the first-half of Comment #12, since it's already been 
discussed and it will just get us side-tracked.

This debate has gone on too long, at this point everything that can be 
discussed has been discussed. I'm am going to begin deleting comments by those 
not involved unless they bring something new and useful to the discussion. 
Input is welcome, but I ask everybody not involved to keep focused on the 
current matrix implementation those involved are discussing (not other 
previously discussed possibilities), if they don't want their comments deleted.

Please keep in mind that our GoogleCode Issue tracker is primarily our work 
space, not a discussion forum. It's already crazy enough with myself, madshi, 
nevcairiel, and plorkyeran debating things. Once we have reached a final 
agreement on how to deal with this matrix issue, I'll allow everybody to 
comment freely again.

> Or actually, make people damn tag their bitstreams correctly 
> if they want correct video (not sub, *video*) colors from now on.

Yes, this is exactly what would need to happen, which is why it's an acceptable 
solution.

Original comment by cyber.sp...@gmail.com on 1 May 2012 at 2:28

[GoogleCodeExporter]

Ok. This will be my final comment here.

Just to clarify, I'm not trying to argue for yet another idea, but rather, was 
trying to help justify madshi's "no matrix in the script" proposal. Your 
counterpoints didn't seem to make sense to me, as in they were bringing up 
non-issues.

But I understand that you prefer to keep the discussion between implementors, 
and madshi is fully capable of defending his position anyway, so I'll remove 
myself now.

Original comment by yuriks...@gmail.com on 1 May 2012 at 2:52

[GoogleCodeExporter]

> Your counterpoints didn't seem to make sense to me, 
> as in they were bringing up non-issues.

My counterpoints haven't been complete non-issues, but so far people have been 
having trouble grasping what issues I'm actually talking about. They don't seem 
to make sense to a lot of people without a fully understanding how VSFilter, 
Aegisub, and Libass work, as well as the issues which VSFilter has caused over 
the years for the ASS spec. This is exactly why I'm now going to lock down 
comments which would sidetrack us, since we may be nearing a conclusion.

As an old ex-fansubber myself, I understand the problems which need to be 
solved, as do the Aegisub devs. Both madshi and nevcairiel have never 
experienced first-hand the issues which have plagued fansubbers for years, 
which is why some of the concepts I'm attempting to express are misunderstood 
as non-issues at first glance and need extra explanation.

tgoyne (aegisub dev) understands the potential issue I'm talking about, and 
that's a start. He also proposed a solution to that potential issue (that 
doesn't involve explicit tagging), which if madshi and nevcairiel agree, we're 
done.

This is the best compromise we've come to so far for a simplified solution. If 
madshi and nevcairiel reject it, we're back to the debate over explicit matrix 
tagging, which I still believe is the better solution because of Libass 
concerns. (It would also be nice to solve uncommon levels issues along the way, 
but that's insignificant in the big picture.)

The key point here, is a change like this to the ASS spec is going to be a 
one-time thing. It doesn't just need to work perfectly today, but also 10+ 
years into the future. If we're not careful, we run the risk of causing yet 
another matrix problem even worse than the current BT.601-only landscape caused 
by VSFilter.

That is why so far I've been promoting 'providing information only' tagging 
solutions which are flexible enough to solve any problem with embedded 
subtitles (aka MKV softsubs), yet at the same time doesn't force solving 
absurd/abusive problems on anybody. The only argument against this is the 
potential of unwanted complexity, which is indeed a valid concern. Unwanted 
complexity should be avoided, but not at the expense of a solution which could 
end up broken long-term.

I remain hopeful that madshi and nevcairiel will accept tgoyne's idea, as I 
have done. If tgoyne's idea is rejected, we're seriously running out of 
implementations which will be acceptable for everybody, especially if explicit 
matrix tagging remains a no-go.

Original comment by cyber.sp...@gmail.com on 1 May 2012 at 6:06

[GoogleCodeExporter]

@cyber, it would help a lot if you could formulate an example case where there 
would be problems with the "color repair needed: no" approach, while your 
approach would work ok. Can you find and completely spell out such a case? 
Which color matrix would the contained in the video (if any)? Which would be 
used by Aegisub? Which by the video renderer? Which by VSFilter with either 
"color repair needed: no" or your preferred approach? Hopefully, when I see 
such an example fully explained, so I can see why your approach improves things 
and mine does not, maybe we'll finally be able to get to the same page. Thanks.

Currently I'm wondering if we should change anything *at all*. Maybe we should 
just have Aegisub go back to the old ways and always use BT.601. We could even 
"officially announce" that ASS subtitles are specced to always be written for 
BT.601. The disadvantage would be that it's ugly and known to be "wrong", but 
at least all ASS files would stay the same, and everybody knew how ASS files 
are to be interpreted. If it's official that ASS is always (has always been and 
will always be) BT.601 then it should also be possible to have libass implement 
the necessary color correction.

Original comment by mad...@gmail.com on 1 May 2012 at 7:05

[GoogleCodeExporter]

> Perhaps one solution to the unsupported color matrix issue would be to make 
Aegisub never guess the color matrix, and always use the backwards compatible 
forced BT.601 mode if the video is not explicitly flagged with a recognized 
color matrix. Wouldn't this completely eliminate the case that cyberbeing is 
worried about?

I think that's not as good an idea as it may seem: AegiSub seems to guess the 
colourmatrix the same way a video renderer guesses the colourmatrix when the 
bitstream isn't flagged. This is probably the reason most streams aren't 
flagged correctly, since the video renderer guesses right anyway. And if the 
colourmatrix differs from the guess the video renderer is expected to make, it 
is (or at least should be) flagged to the bitstream anyway. This means that 
behaviour would actually be counter-productive, since, as I understood it, the 
goal was to display (at least the majority of) ASS subtitles in untouched RGB, 
while this behaviour would only promote the old workaround.
Also, I believe that scripts would need to be flagged either way, because I'm 
sure that there are at least some scripts out there that are not specifically 
created for VSFilter's BT.601 limitation, but actually use the correct colours 
(maybe the creator didn't know about the issue, or maybe it was written for 
libass, or maybe the creator simply didn't care). For those scripts, that are 
currently unflagged, the subtitle renderer wouldn't even be able to guess 
correctly because it would assume that all "unflagged" scripts need the BT.601 
workaround.
Personally, I think that, since the script has to be flagged either way, there 
should at least be an option to control the conversion of unflagged bitstreams.

Original comment by TheDarkS...@googlemail.com on 1 May 2012 at 8:45

[GoogleCodeExporter]

> it would help a lot if you could formulate an example case where there 
> would be problems with the "color repair needed: no" approach, while 
> your approach would work ok.

Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Guess matrix BT.709 by resolution + "color repair needed: no"
Result: BT.709 subtitles on non-BT.709 video

Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Never guess matrix and use BT.601 + "color repair needed: yes"
Result: BT.601 subtitles on non-BT.601/BT.709 video

Video Bitstream: Empty (PC-range YCbCr)
Aegisub: PC-Range RGB
VSFilter: TV-range YCbCr
Result: TV-range subtitles on PC-range video

Video Bitstream: Empty (TV-range RGB)
Aegisub: PC-Range RGB
VSFilter: PC-range RGB
Result: PC-range subtitles on TV-range video

Video Bitstream: Flagged (actual matrix != BT.601 or BT.709)
Aegisub: Actual matrix supported
VSFilter: Actual matrix supported (RGB output)
Video Renderer: Actual matrix NOT supported 

Video Bitstream: Doesn't matter
Subtitle Editor: Not Aegisub, and doesn't agree with how Aegisub chooses matrix
VSFilter: Only matches Aegisub, creating a matrix mismatch
Result: Incorrect colors

Video Bitstream: Doesn't matter
Subtitle Editor: Not Aegisub, and doesn't agree with how Aegisub chooses matrix
Libass: Untouched RGB
Video Renderer: Only matches Aegisub, creating a matrix mismatch
Result: Incorrect colors

Video Bitstream: flagged (actual matrix != BT.601 or BT.709)
Aegisub: Force BT.601 + "color repair needed: yes"
Libass: Untouched RGB
Video Renderer: Actual matrix
Result: Incorrect colors

Video Bitstream: flagged (actual matrix != BT.601 or BT.709)
Aegisub: Auto-guess BT.709 + "color repair needed: no"
Libass: Untouched RGB
Video Renderer: Actual matrix
Result: Incorrect colors

Video Bitstream in Aegisub: Full-range RGB
Aegisub: "color repair needed: no"
Final Muxed Video: Unflagged YCbCr
Subtitle Filter: Guesses incorrect matrix
Result: Incorrect colors

Video Bitstream in Aegisub: Flagged YCbCr (actual matrix != BT.601 or BT.709)
Aegisub: "color repair needed: no"
Final Muxed Video: Flagged YCbCr (BT.709)
Result: Unable to identify the problem by only looking at the script

If you look through my two week discussion with sneakerger and thedarkspaces, 
you can probably find more examples. Keep in mind that Libass implementations 
and hardware players are likely going to ignore anything we do here. Providing 
information only in a "use if needed/desired" way avoids drama. It offers a 
solution, but doesn't force the solution on anybody. There is also the fact 
that explicit tagging means less work for YuZhuoHuang and any other subtitle 
filters who plan to implement what we've discussed here. The sky is the limit, 
since any problem could potentially be solved (or at least identified before it 
becomes a problem) by explicitly tagging two matrixes.

Again, the key point here is while many of these things may not be a problem 
today, they could be in the future. The majority of problems would arise if 
BT.709 is ever replaced by some other matrix.

That said, tgoyne's idea which may cause "incorrect" BT.601 to be used more 
often, at least avoids any problems BT.709 being replaced. If you want 
extremely simplified solution, this is the only viable option. By using 
explicit matrix tagging, an "incorrect" matrix will never be used after this 
change, unless the user desires legacy VSFilter 2.39/2.40 compatibility.

__________
In many ways this make me re-think our use of RGB-only vs RGB+AYUV with the new 
interface. It's quickly becoming clear that dual-matrix correction of RGB may 
become a major pita. AYUV alpha-blending by the video renderer would only 
require a single-matrix correction to match the video colors.

madshi, can you refresh my memory of the advantage of using RGB vs AYUV in the 
new interface? I suspect it would require some major changes to the madVR 
workflow, but would it be worth it to further simplify these subtitle matrix 
issues?
__________

> Currently I'm wondering if we should change anything *at all*. 
> Maybe we should just have Aegisub go back to the old ways and 
> always use BT.601. We could even "officially announce" that ASS
> subtitles are specced to always be written for BT.601.

All this would do is cause drama with Libass who so far has refused to mimic 
VSFilter's BT.601 behavior because what they are doing is "correct" (and it 
would be if VSFilter didn't exist). In other words, nothing would improve. It 
would still be impossible to create a script which has correct colors in Libass 
and VSFilter.

Since with this new interface we are going to need to at least some of the time 
support BT.601 corrected RGB for VSFilter legacy compatibility, it's the 
perfect opportunity to give users the ability to solve this problem going 
forward if desired.

> This is probably the reason most streams aren't flagged correctly, 
> since the video renderer guesses right anyway.

Exactly, but what happens in the future when a third matrix enters the mix and 
replaces BT.709 as the logical option for auto-guessing?

This may never happen, but if it does we're screwed, as we've just created yet 
another matrix problem with the ASS spec which could have been avoided. How 
good is everybody's crystal ball? Not worth the risk repeating history, and 
making the same mistake Gabest did that got us into this mess.

tgoyne's solution of eliminating auto-guessing would avoid this potential 
problem, as long as nothing attempts to replace Aegisub with different matrix 
behavior. While it certainly wouldn't be my first choice, as it doesn't solve 
the underlying problems as much as I would like, it should work with minimal 
chance of breakage.

Original comment by cyber.sp...@gmail.com on 1 May 2012 at 11:14

[GoogleCodeExporter]

@cyber, thanks for those examples, but none of them is really complete. I just 
wanted one complete example, not a dozen incomplete ones. Let's take the first 
example you listed. You didn't spell out what VSFilter would do in that 
situation and what the video renderer would do (IYO). You also didn't spell out 
how your preferred solution would look like in that situation and how your 
solution would change VSFilter's and the video renderer's behaviour for the 
better exactly. It is still not obvious to me from the very first example in 
your list why your solution would work better. Could you please take one of 
those examples (preferably the one you think is the most important, and if 
possible one that deals with colors and not with levels) and make it clear how 
every software involved would behave with my vs. your preferred solution? 
Thanks.

Original comment by mad...@gmail.com on 2 May 2012 at 6:17

[GoogleCodeExporter]

For all of the examples I listed, I just stopped as soon as the matrix used 
became "incorrect". Everything after that point is irrelevant. With explicit 
matrix tagging, the matrix will never be incorrect as long as it's tagged 
correctly, and the matrix needed is supported up until the subtitles are 
alpha-blended on the video.

Below are a few variations of the first example which come to mind:
__________
"Color Repair needed: no"

Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Guess matrix BT.709 by resolution + "color repair needed: no"
VSFilter: Guess matrix BT.709 by resolution (if YCbCr, else untouched RGB)
madVR: Guess matrix BT.709 by resolution
Result: Subtitles correct, decoded matrix wrong

"Color Repair needed: no"

Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Guess matrix BT.709 by resolution + "color repair needed: no"
VSFilter: Guess matrix BT.709 by resolution (if YCbCr, else untouched RGB)
madVR: Guess or user-selected correct matrix
Result: Subtitles wrong, decoded matrix correct

"Color Repair needed: yes" [tgoyne's idea]

Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Force BT.601 + "color repair needed: yes"
VSFilter: BT.601 (if YCbCr, else double corrected RGB)
madVR: Guess matrix BT.709 by resolution
Result: Subtitles correct(?), decoded matrix wrong
(this one confuses me a bit, can someone confirm if the double-corrected RGB 
subtitles are correct in this case?)
__________

Explicit matrix tagging

Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: User specifies correct YCbCr->RGB video matrix (Aegisub tags matrix 
twice to script for untouched RGB)
VSFilter: Renders subtitles with correct matrix (if YCbCr, else untouched RGB)
madVR: Renders video with correct matrix when using new interface (untouched 
RGB subtitles)
Result: Subtitles correct, decoded matrix correct

Explicit matrix tagging

Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Guess matrix BT.709 by resolution (Aegisub tags BT.709 matrix twice to 
script for untouched RGB, but user edits in the correct Video Matrix so a 
correction will be performed)
VSFilter: Renders subtitles with correct matrix (if YCbCr, else 
double-corrected RGB)
madVR: Renders video with correct matrix when using new interface 
(double-corrected RGB subtitles)
Result: Subtitles correct, decoded matrix correct

Explicit matrix tagging

Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Guess matrix BT.709 by resolution (Aegisub tags BT.709 matrix twice to 
script for untouched RGB, but user edits in the correct Video Matrix so a 
correction will be performed)
VSFilter: Renders subtitles with correct matrix (if YCbCr, else 
double-corrected RGB)
EVR: Guess matrix BT.709 by resolution
Result: Subtitles correct, decoded matrix wrong
__________

Hopefully this is a good enough sampling of the variations possible for the 
first example. It's difficult to predict all possibilities with "Color Repair 
needed". The problem is that second variation. Someone will also need to 
clarify my inquiry about the example for tgoyne's idea. 

The "Explicit matrix tagging" options are extremely flexible, so if we go that 
route, there could be some discussion about the most practical way to achieve 
correct output with the information available.

madshi, after you process the above examples, could you comment on the below 
from my previous comment? AYUV would seem to have some advantages for picking 
the correct subtitle matrix, wouldn't it? In that way we wouldn't need to worry 
about the video matrix at all, since the subtitles would already be 
alpha-blended before YCbCr->RGB conversion is performed. We could then just 
continue using the Aegisub Dev solution which has been in place since January. 
From the xy-VSFilter standpoint, AYUV output isn't a problem since that's what 
we use internally with YCbCr output. How much of a roadblock would AYUV 
subtitles be to madVR?
__________
In many ways this make me re-think our use of RGB-only vs RGB+AYUV with the new 
interface. It's quickly becoming clear that dual-matrix correction of RGB may 
become a major pita. AYUV alpha-blending by the video renderer would only 
require a single-matrix correction to match the video colors.

madshi, can you refresh my memory of the advantage of using RGB vs AYUV in the 
new interface? I suspect it would require some major changes to the madVR 
workflow, but would it be worth it to further simplify these subtitle matrix 
issues?
__________

Original comment by cyber.sp...@gmail.com on 2 May 2012 at 8:25

[GoogleCodeExporter]

Before this starts in endless discussion again, i'm against AYUV for subtitles.
The interface is meant to be generic, and not match implementation issues of 
VSFilter.

The interface tells you which matrix the renderer uses for the RGB conversion, 
and with that information you can produce RGB that matches the renderers 
output. It doesn't actually solve any problems, it just avoids you having to do 
a RGB conversion with the matrix the renderer tells you to use. It only shifts 
the RGB conversion to the renderer, it doesn't change anything else.

The big advantage of RGBA for subtitles is that it can be blended onto the 
image as one of the last steps in the rendering process, giving the whole 
workflow much more time to prepare and render subtitles (ie. madVR can request 
the subtitle image as soon as it gets the image from the decoder, but only 
needs it for one of the very last rendering steps)

Not to mention generally bad GPU driver support for AYUV surfaces.

Original comment by h.lepp...@gmail.com on 2 May 2012 at 8:42

[GoogleCodeExporter]

(1) If we sum up all the matrixes supported by Microsoft DXVA, MediaFoundation, 
MPEG2, VC-1 and h264, we get these:

(a) BT.601 (used by NTSC and PAL)
(b) BT.709 (used by all HD content)
(c) SMPTE 240M
(d) YCgCo
(e) US National TV System 1953

Only BT.601 and BT.709 have any real meaning these days. The other 3 are not 
used in any digital consumer content, at the moment, AFAIK. Of course there's 
always a chance a different matrix might be used some day, but it doesn't look 
too likely to me right now.

I've also checked a draft of the upcoming h265/HEVC codec, rumored to be used 
by the future 4K Blu-Ray standard, and there are not any new matrixes 
introduced there, at this point in time.

(2) I'd like to point out that a video file which is neither matrixed with 
BT.601 nor BT.709 and on top of that doesn't have its real matrix stored in the 
video bitstream is IMHO a completely and totally broken encoding. I'm not 
really sure if it's worth it for us to worry about such a catastrophic 
encoding. Is that the most important case for why you want to have explicit 
matrix tagging?

(3) "Explicit matrix tagging
"madVR: Renders video with correct matrix when using new interface"

Why would madVR render video with the correct matrix in this situation? I don't 
see that happening. The new interface only supplies madVR with RGB subtitles, 
it doesn't change the matrix madVR uses to decode the video. So your example 
should read there: "madVR: Guess matrix BT.709 by resolution", same as EVR.

(4) "Explicit matrix tagging
EVR: Guess matrix BT.709 by resolution
Result: Subtitles correct, decoded matrix wrong"

How can the subtitles be correct if the decoded matrix is wrong - when using 
the VSFilter way of rendering subtitles in YCbCr? The wrong decode matrix will 
also damage the subtitles in that case, or am I wrong?

-------

Let's discuss AYUV later. I prefer discussing one thing at a time.

Original comment by mad...@gmail.com on 2 May 2012 at 9:36

[GoogleCodeExporter]

I don't see how using AYUV with the new interface would have anything to do 
with VSFilter (who's native format is RGB), but that's beside the point. Doing 
a simple RGB->AYUV conversion certainly seems more generic than 
"double-corrected RGB". AYUV would simplify achieving correct subtitle colors 
in relationship to a video by only requiring a single matrix (the matrix used 
by Aegisub) tagged in the script.

None the less, if AYUV is not seen as a viable option by madshi as well, I 
guess we're stuck with the complexity of "double corrected" RGB.

> The big advantage of RGBA for subtitles is that it can 
> be blended onto the image as one of the last steps in 
> the rendering process

Since we need to let madVR apply gamut/gamma correction to subtitles in the 
same way as the video (to avoid yet another subtitle color problem), I'm unsure 
this advantage exists. Even if it does, couldn't you theoretically achieve the 
same thing with AYUV considering RGB<->AYUV conversion can be lossless?

> Let's discuss AYUV later. I prefer discussing one thing at a time.

Okay. Posting this just because I already had it fully typed up before I 
noticed your reply. We can come back to it later if needed.
____________________
____________________

(1) I forgot about checking the h265/HEVC drafts. Since it doesn't list any new 
matrix, then that does likely means were safe from 4K video breaking things.

Uncertainty of what the future holds is a risk for us none the less. I'd still 
prefer not to take that risk, since I don't see any advantage which "color 
repair needed" offers over "explicit matrix tagging".

> Is that the most important case for why you want to have explicit matrix 
tagging?

It goes beyond that by potentially allowing more oddball mismatch cases to be 
corrected. The "color repair needed" idea only works as long as "Subtitle 
Editor" + "Subitle Filter" + "Video Renderer" are in perfect sync as far as 
matrix information is concerned. How do we ensure that this occurs when we are 
creating a generic interface?

I also find the "color repair needed" workaround way too VSFilter specific and 
restrictive. It will never be useful by anybody but those of us in this thread, 
so it doesn't belong in the ASS spec.

Could you please explain to me the advantage which "color repair needed" offers 
over even a just a single matrix (the one used by Aegisub) in the script? As 
you'll notice back in Issue #40 
(http://code.google.com/p/xy-vsfilter/issues/detail?id=40#c303) I listed them 
as being at identical levels of information. What can "color repair needed" 
accomplish which a single tagged matrix can't?

> The new interface only supplies madVR with RGB subtitles, 
> it doesn't change the matrix madVR uses to decode the video.

(3) Yet why shouldn't we make it do so? Being told with authority what the 
matrix is, would be much better than guessing. Even going back to your Issue 
#40 comment it seemed like something you had in mind to avoid needing to have 
madVR guess:
http://code.google.com/p/xy-vsfilter/issues/detail?id=40#c292

> How can the subtitles be correct if the decoded matrix is wrong 

(4) They were alpha-blended into the YCbCr video with the correct colors, so 
the subtitles are correct in relationship to the video.  This is an important 
concept to remember, since it's the entire reason why I brought up AYUV just 
now. Once the subtitles are part of the video, it matters little if the video 
renderer uses a wrong YCbCr->RGB matrix from the point of view of subtitle 
correctness.

Original comment by cyber.sp...@gmail.com on 2 May 2012 at 11:20

[GoogleCodeExporter]

Ok, I think I'm having a moment of clarity. Let me sum up the situation, as far 
as I understand it:

(1) The problem you're most worried about happens when Aegisub (during creation 
of the ASS) and VSFilter (during playback of the ASS) use a different color 
matrix. If that happens, the subtitle colors will not match the video, when 
using the VSFilter way of rendering subtitles in YCbCr.

(2) Which matrix the video renderer uses has no impact on whether subtitle and 
video colors match (when talking about VSFilter).

(3) If you would have to sort possible situations according to what you 
consider best (or least bad), you'd rate like this:
(a) every is perfect
(b) subtitle and video colors match, but both might be using (the same) wrong 
matrix
(c) video colors are correct, but subtitle colors are wrong -> color mismatch
(d) subtitle colors are correct, but video colors are wrong -> color mismatch
(e) subtitle and video colors mismatch, and both are wrong on top of that

(4) With your "Explicit matrix tagging", you would make sure that at least 
video and subtitle colors would always match.

(5) With my "color repair needed" option, it is possible that Aegisub and 
VSFilter could potentially use a different matrix, resulting in mismatching 
subtitle/video colors, when using YCbCr VSFilter rendering.

Is that a correct summary?

Original comment by mad...@gmail.com on 2 May 2012 at 12:22

[GoogleCodeExporter]

> If you would have to sort possible situations...

I'd just simplify it as follows.

Success:
(a) subtitle and video colors match using the correct matrix

Partial Success:
(b) subtitle and video colors match using the wrong matrix

Complete Failure:
(c) subtitle and video colors mismatch

> Is that a correct summary?

Overall, assuming you're talking about YCbCr output only.

After taking some time to sketch out the behavior of some of the discussed 
possibilities, something like the following, along with simple intuitive names, 
is beginning to grow on me. Enhance the new interface to have the provider send 
yuvMatrix to the consumer when rgbCorrection=False, and you could have a 
distinct advantage in terms of script portiblity.

yuvMatrix: TV|PC.Matrix
rgbCorrection: True|False

None the less, I continue to favor the idea of tagging both source and aegisub 
matrix, even if only for script identification purposes. The prospect of a 
subtitle editor having functionality to reverse a matrix mismatch in a script 
with reliability, would only be possible using this tagging method. Even for 
the new interface, I fail to see how having the potential to solve any problem 
on playback would be a bad thing. The hardest part would be deciding how much 
to actually support.

sourceMatrix: TV|PC.Matrix
yuvMatrix: TV|PC.Matrix

We could even go all out with three tags, but with the second matrix being 
optional. That could allow an optional identification benefit, but without any 
expectation to actually use it for anything ourselves. I'd even go so far as to 
say that Aegisub shouldn't even be allowed to tag sourceMatrix in such a case. 
We could just define what it represents, with the expectation it will be 
manually tagged.

sourceMatrix: TV|PC.Matrix <--- Optional
yuvMatrix: TV|PC.Matrix <--- Required
rgbCorrection: True|False <--- Required

Original comment by cyber.sp...@gmail.com on 3 May 2012 at 11:11

[GoogleCodeExporter]

>> Is that a correct summary?
> Overall, assuming you're talking about YCbCr output only.

Ok, so let me sum up what we might want to aim for, for a "good" ASS color 
matrix solution:

(1) Subtitle colors should always match video colors.
(2) Both VSFilter YCbCr rendering and the new interface RGB rendering should 
produce identical colors.
(3) Aegisub should prevent the user from accidentily screwing up the colors.
(4) Whatever we add to the ASS spec should be as simple and easy to understand 
as possible.

Everybody agree?

Well, I have a solution which I think would solve all of the above, plus it's 
very simple. But I fear you may still dislike it. Anyway, here goes:

We should simply go back to having old VSFilter be the ASS spec. In detail I 
suggest:

(1) We officially announce that ASS RGB colors are always to be interpreted as 
BT.601 YCbCr values.
(2) We revert VSFilter and xy-vsfilter back to always blindly using BT.601.
(3) We let Aegisub continue to auto-guess the correct matrix, maybe even allow 
users to select any matrix they want. *BUT* Aegisub has to follow the ASS spec 
which says that ASS RGB colors are always BT.601 YCbCr values. So whatever 
matrix Aegisub is using for video playback, it has to convert the colors back 
to BT.601 when saving the ASS file to disk.
(4) For the new RGB interface, we have to require that the final subtitle 
consumer (the one which blends the subtitles to video) has to perform the 
BT.601 -> whatever matrix correction.

Advantages:
- fulfills all listed requirements
- works with any current and future video matrix
- very easy, no new header ASS header fields needed at all

Disadvantages:
- kinda "ugly" because hard coded to BT.601
- Aegisub needs to be modified
- libass devs might refuse to support this

Honestly, I don't think libass is a problem at all. If e.g. Hendrik decides to 
implement a libass subtitle renderer, it would be dead easy for him to add the 
necessary corrections on top of libass - that is if he wants to support YCbCr 
rendering at all. If he only supports the new RGB interface, it would actually 
be correct for libass to not perform the correction - because that would be the 
duty of the final subtitle consumer (e.g. madVR), anyway.

Original comment by mad...@gmail.com on 3 May 2012 at 11:38

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

> If he only supports the new RGB interface, it would actually be correct for 
libass to not perform the correction - because that would be the duty of the 
final subtitle consumer (e.g. madVR), anyway.

Wait, what?
Who says we always want to send ASS using the new interface? There are other 
subtitle formats out there, you know...
That would require renderers of other formats to always do the inverse 
correction of the ASS BT.601 workaround.

Also, there already are libass implementations out there. Like mplayer2 and 
VLC. And I don't know how you want to "officially" announce anything. This is 
why I favor the route to go back to RGB (as soon as xy-vsfilter supports it and 
is ready to be used in CCCP). The workaround was vsfilter specific, so we only 
need to fix (xy-)vsfilter and make it aware of the colors, i.e. workaround used 
or not.
Also, I don't expect any future workarounds to appear, because the currently 
used implementation specific workaround was only feasible, because there was 
only a single usable implementation in the first place at that time. We 
currently have more than one implementation and with a hypothetical 
xy-vsfilter->RGB->madVR mode we get yet another one, so future typesetters 
won't be as easily tempted to use an implementation specific workaround instead 
of fixing their workflow/the players as they were back then.

Original comment by sneaker...@googlemail.com on 3 May 2012 at 5:27

[GoogleCodeExporter]

> Who says we always want to send ASS using
> the new interface?

Nobody.

> There are other subtitle formats out there, you know

Of course the color repair would only be performed for ASS subtitles. I thought 
that much was clear.

> Also, there already are libass implementations
> out there. Like mplayer2 and VLC.

Yes. What is your point? Whether we add a "workaround used" marker (your 
preferred solution) or whether we add explicit matrix information (cyber's 
preferred solution) or whether we use my latest suggestion, it won't change a 
thing about how the already existing libass implementations will treat the 
trillions of ASS files which are already out there.

Original comment by mad...@gmail.com on 3 May 2012 at 5:47

[GoogleCodeExporter]

> Nobody.
> Of course the color repair would only be performed for ASS subtitles. I 
thought that much was clear.

It wasn't clear, but it is now. Anyhow, I don't think this a job for the video 
renderer. If one wants to do it, it should be left to the subtitle filter and 
the renderer could communicate the matrix to it, as with the "yuvMatrix" 
DirectShow attribute proposed by cyberbeing. If other renderers would want to 
adopt this interface as well, they also would have to implement this 
correction, while it is originally a problem that is rather specific to a 
single implementation and format.

>Yes. What is your point? Whether we add a "workaround used" marker (your 
preferred solution) or whether we add explicit matrix information (cyber's 
preferred solution) or whether we use my latest suggestion, it won't change a 
thing about how the already existing libass implementations will treat the 
trillions of ASS files which are already out there.

Yes, these players (along with MPC-HC) are currently assuming RGB and just like 
among us participating in this discussion, there isn't any agreement around the 
world on how to interpret colors. Some devs will continue to interpret unmarked 
scripts as true RGB and we won't be able to convince them otherwise, as there 
is no way to "officially" announce anything in the ASS world. I can however see 
them willing to assume BT.601 workarounds (or in their eyes "incorrect" 
colors/a broken script) as long as the typesetter has explicitly marked his 
script as such. We only have influence on xy-vsfilter, madVR and Aegisub, which 
we should use to return to true RGB with marked scripts so future players (an 
already existing VLC, mplayer2, MPC-HC) will all display them correctly.

Some assumptions made during the turn of this discussions are not true:
- everyone agreeing on unmarked scripts being "BT.601 workaround" (libass devs, 
etc.)
- everyone wanting to match their subtitle colors to their video colors. The 
drawing of rectangles on a video for typesetting is only a very small part of 
ASS. Many people just want to have their dialogue  using a pretty font and the 
RGB color they chose.

Original comment by sneaker...@googlemail.com on 3 May 2012 at 6:25

[GoogleCodeExporter]

I'm against changing the spec to BT.601 YUV for the following (already 
mentioned) reasons:
(1) Our original goal was to create an interface that would transmit RGBA data 
to the renderer, and only in special, legacy VSFilter cases to perform a Matrix 
Compensation (any goal may change over time, though, and I admit that)
(2) There are already subtitle scripts out there that don't follow the BT.601 
legacy requirement for VSFilter. Possible reasons include, but are not limited 
to:
(2.1) the script creator being unaware of VSFilter's limitation to BT.601
(2.2) the script originally not being written for VSFilter
(2.2.1) but for libass
(2.2.2) but for xy-VSFilter with matrix guessing enabled
(2.3) the script creator simply not caring about the limitation, expecting it 
to be fixed sometime (soon)
(3) I assume a secondary goal was also at least some sort of interoperatibility 
between subtitle providers, and honestly, if ASS was designed to be rendered in 
YUV in the first place, it wouldn't use BGR colour codes, it would store the 
colours in YUV.

By the way, I'm not sure whether a simple field of whether Matrix Compensation 
has to be used or not will be actually useful for this interface, because if it 
is used, it indicates that xy-VSFilter has to render the subtitles in BT.601 
YUV (I assume 4:4:4 would be the most useful variant, but I don't really know 
much about YUV) directly (which it is capable of) and either send it to madVR, 
which then will proceed to convert it to RGB using the video's colour matrix, 
or get the colour matrix madVR is using via the interface and use it to convert 
the YUV image to RGB. The alpha channel is completely unaffected, which is why 
I conveniently left it out.
However, if explicit matrix tagging is used, xy-VSFilter will have to check 
whether the matrix the Script Matrix field and matrix used in the Video Matrix 
field match. If they match, it will treat the script colours as raw RGB and 
send them to madVR without any change. If they don't match, xy-VSFilter will 
proceed to render the subtitles directly in the YUV colourspace used as Script 
colourspace and either send the YUV image to madVR along with the information 
of what matrix madVR should use to convert the image back to RGB or convert the 
image to RGB using the flagged Video Matrix and send the RGB data to madVR.

I do, however, not see any reason to perform a "triple correction" of sorts for 
different sources: When madVR gets the data from the subtitle renderer, the 
colours already match (or will be converted to match, doesn't matter) those of 
the "source video" which was used to create the subtitles. As far as my 
understanding goes, it is completely irrelevant whether the colours come from 
the BT.601 DVD version of a movie, but will be used with the BT.709 HD version 
of said movie or whether they come directly from the HD BT.709 version, because 
the colour values of the BT.601 version have been converted into BT.709 values 
for the HD version. That means, both version (should) decode to the same 
colour, and therefore, the colours already match.

Original comment by TheDarkS...@googlemail.com on 3 May 2012 at 6:28

[GoogleCodeExporter]

@sneaker:

> there is no way to "officially" announce anything

Probably true, but I think adding something to the Aegisub documentation could 
help.

> We only have influence on xy-vsfilter, madVR and
> Aegisub, which we should use to return to true RGB
> with marked scripts so future players (an already
> existing VLC, mplayer2, MPC-HC) will all display
> them correctly.

If we follow my latest suggestion, it would easily be possible for everyone to 
achieve perfect colors with pretty much all current and future ASS files. Maybe 
libass/mplayer2/VLC devs will refuse to implement the necessary changes. But if 
they do, their users should complain. I don't think it's a good enough reason 
to change a spec just because some devs do not do their work properly.

> The drawing of rectangles on a video for
> typesetting is only a very small part of ASS

If we ignore this small part of ASS, then why are we discussing colors at all? 
No normal user will see the difference between BT.601 vs BT.709, unless we're 
talking about subtitle background colors being matched to video colors.

Original comment by mad...@gmail.com on 3 May 2012 at 7:02

[GoogleCodeExporter]

> Maybe libass/mplayer2/VLC devs will refuse to implement the necessary 
changes. But if they do, their users should complain. I don't think it's a good 
enough reason to change a spec just because some devs do not do their work 
properly.

As I view it, there is no real change to the spec. Some people have used true 
RGB values, some used the BT.601 workaround and we will probably see both kind 
of scripts for quite some time, though I hope that Aegisub pushes users back to 
true RGB in the medium to long term. Even now, without any changes, you can't 
be sure all users get correct colors because of this. The BT.601 workaround was 
a vsfilter specific workaround, so it seems logical so solve that problem in 
xy-vsfilter as an evolved vsfilter. The only thing we are changing is to add 
some optional information, which help xy-vsfilter solve that problem, while 
still being backwards-compatible to the old vsfilter (if wanted by the user).

> If we ignore this small part of ASS, then why are we discussing colors at all?

I'm not saying we should ignore it, but we should be aware that this is not 
always the author's will. Of course all this is to satisfy the 1% that do care 
(like madVR itself), and hopefully to make display as intended even for the 
remaining 99% as long as they use up-to-date filters.

But there will always be some problems and no matter how much I think about it, 
they are unsolvable. We can't force anyone to agree on the specs and currently 
I see vsfilter as being the problem, not the other programs, so going back to 
the original true RGB values seems appropriate. Scripts that have been marked 
can be correctly displayed as long as the devs care enough to implement the 
support (unless the script already matches their default behavior), but we will 
continue to have problems with programs whose devs don't. But we already have 
this situation right now, so it is no regression however you look at it.

Original comment by sneaker...@googlemail.com on 3 May 2012 at 7:41

[GoogleCodeExporter]

> I'm not saying we should ignore it, but we should be aware that this is not 
always the author's will.
I just thought of this earlier, but with matrix information in the ASS script, 
the color tab of xy-VSFilter's properities page would need to be reworked 
anyway. So, instead of simply adding a "follow BT.601 unless the script says 
otherwise" option, there could also be added an option to always enable / 
disable Matrix Compensation for scripts that aren't flagged at all.

Original comment by TheDarkS...@googlemail.com on 3 May 2012 at 7:49

[GoogleCodeExporter]

As to not get side tracked and ending up repeating the same arguments over and 
over again (#33), I will add a point to the requirements listed by madsi (#26):

> (1) Subtitle colors should always match video colors.
> (2) Both VSFilter YCbCr rendering and the new interface RGB rendering should 
produce identical colors.
> (3) Aegisub should prevent the user from accidentily screwing up the colors.
> (4) Whatever we add to the ASS spec should be as simple and easy to 
understand as possible.

(5) Scripts should be compatible with the majority of programs, regardless of 
whether their devs choose to implement any workarounds and additions or not

Original comment by sneaker...@googlemail.com on 3 May 2012 at 8:28

[GoogleCodeExporter]

Everything that isn't vsfilter would instantly benefit from compatibility with 
the new standard if pure RGB ends up being used. Furthermore, even if libass 
can be convinced to hop aboard, the more things differ from pure RGB, the more 
likely it is that someone else will implement it wrong or even not bother 
implementing it. Pure RGB seems to be the choice most likely to lead to an 
uniform handling of subs in the future, after vsfilter is fixed, IMO.

Original comment by yuriks...@gmail.com on 3 May 2012 at 8:46

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

cyber: Just as a clarification: (I'm not arguing, I'm trying to present what I 
understand as being real and asking for disagreements!)

It's important that H.264 have the matrix used flagged in the bitstream because 
H.264 video is stored in YCbCr space. And YCbCr data on it's own is meaningless 
from a display standpoint without knowing how to transform that data into RGB 
space.

ASS subs, on the other hand, are, in their 'bitstream' (that is, their 
plaintext disk representation) in RGB space already, and thus doesn't have any 
matrix instrinsically attached to them that is needed to decode them to RGB 
colors.

Any YCbCr conversion matrix that will get involved with the subs will come from 
the need to convert from that RGB space to or from some YCbCr color space. (For 
example, if you're burning them onto video you'll use the matrix specified by 
the video, since that's what the renderer will use to show it on-screen later. 
They need to match.)

Let's ignore the sub authoring stage for now and say we have an ASS script with 
100% certified correct RGB colors.

There are several possiblities during playback: (Correct in the results refers 
to correct compared to RGB original.)

1) Renderer is using the correct video matrix. Sub-renderer is doing untouched 
RGB.
 - Result: decoded video colors match real colors and match subtitle's RGB colors

2) Renderer is using *incorrect* video matrix. Sub-renderer is doing untouched 
RGB.
 - Result: decoded video colors are wrong. Subtitles are correct but don't match.

3) Renderer is using correct video matrix. Sub-renderer is burning subs onto 
video, using renderer's matrix.
 - Result: decoded video colors are correct. Due to the script colors also being correct and the correct matrix being used to burn subs, which will be reversed by the renderer's ->RGB conversion, sub colors match video.

4) Renderer is using incorrect video matrix. Sub-renderer is burning subs, 
using renderer's matrix.
 - Result: video colors wrong. Sub's RGB->YCbCr->RGB conversion is lossless due to sub-renderer matching the renderer's matrix. Sub colors are correct but don't match video.

5) Renderer is using correct video matrix. Sub-renderer is doing it's own thing 
and using incorrect matrix, which is different from the renderer's.
 - Result: video colors correct. Sub's RGB->YCbCr->RGB round trip will distort colors due to sub-renderer's and renderer's matrices not being matched. Sub colors are wrong and don't match video.

6) Renderer is using incorrect video matrix. Sub-renderer is doing it's own 
thing and guessed on the correct video matrix, which is different from 
renderer's.
 - Result: video colors wrong. Sub's RGB->YCbCr->RGB round trip distorts colors due to sub-renderer and renderer not agreeing on a matrix. Sub colors are incorrect but *do* match video.

7) Renderer and sub-renderer are both using the wrong matrix.
 - Result: video colors wrong. RGB->YCbCr->RGB will produce doubly wrong results. Sub colors incorrect and don't match video.

Summarizing: cases 1, 3 and 5 have correct video colors. Cases 1, 2, 3 and 4 
have the final sub colors being correct. Cases 1, 3 and 6 additionally have 
matching subtitle colors.

I think that covers all cases. 1 and 3 are how it's supposed to work when 
functioning correctly, I don't think they need any further comments.

2 and 4 are incorrect results caused by the video renderer's inability to use 
the correct video matrix. They produce wrong looking video colors. Even if the 
sub colors are correct, they don't match the video's, and thus aren't desirable.

5 is a good case for why the sub-renderer should listen to the video renderer. 
If it had, then this would be case 3 and all would be well.

6 supports cyber's proposal. If the renderer gets the video matrix wrong, the 
sub-renderer can at least step in and, hoping that the matrix specified in the 
script is correct for the current video (which it has no guarantee of being) 
salvage the situation by rendering subs with the wrong colors, but that 
nevertheless match the video.

On 7 no one has any idea of what's going on and there's not much that can be 
done about it.

Now at first expliciting the matrix may seem like an advantageous idea, as per 
6, however, that reasoning hinges on a premise that, at least for me is a bit 
hard to swallow: It assumes that Aegisub can and will do a better job at 
guessing the matrix used in the video than the very video renderer will. I'd 
expect the video renderer, whose sole purpose is to play back the formats it 
supports, to do a better job at it than aegisub. Why, if the renderer can't 
detect or use that format, then it's unsuitable for playing back that kind of 
content in the first place. Furthermore, the sub-renderer listening to the 
matrix specified in the script can just as easily lead to situation 5. If 
you're going to tag the script with the matrix used in the video bitstream, why 
not instead simply tag the video bitstream itself? If you tag the script you 
now have *two* tags that need to be kept in sync (or have the guessing 
algorithm be consistent) at all times.

If we assume that the video renderer is able to correctly decode the video with 
correct colors (and it better be) we are left with cases 1, 3 and 5. Of which 5 
only exists because the sub-renderer may insist on not following the renderer. 
That fallback ends up actually being harmful.

Of course, this whole post hinges on the ASS script being correct. But if it 
isn't, I'd have to place the blame on the author, or aegisub. Remember, garbage 
in, garbage out. (Perhaps instead of tagging the script aegisub can simply warn 
that streams with incorrect matrix color information are likely to result in 
broken output?)

I hope that was constructive enough.

(Addendum: Regarding YCbCr/RGB at the start, you could of course argue that RGB 
space is also only defined in the presence of color primaries, but in practice 
that tends to be a much lesser issue due in part to it being a standardized 
part of hardware, not a part of the algorithm, and so there's not much you can 
do about it. Furthermore, it globally affects all video output (color 
correction schemes notwithstanding) and so doesn't causes a discrepancy between 
subs and video, which means it doesn't really matter to us.)

Original comment by yuriks...@gmail.com on 4 May 2012 at 2:37

[GoogleCodeExporter]

@yuriks

> I hope that was constructive enough.

I'll leave it in case madshi has some of the same misunderstandings as you do 
in that post.

Original comment by cyber.sp...@gmail.com on 4 May 2012 at 5:06

[GoogleCodeExporter]

> It assumes that Aegisub can and will do a better job at guessing the matrix 
used in the video than the very video renderer will.

I think cyberbeing wants Aegisub to fall back to assuming BT.601 in the case it 
stumbles upon an unknown matrix, i.e. basically the workaround we are currently 
using. This can always ensure matching video and subtitles and in addition 
correct overall colors if the renderer is able to detect the video matrix 
correctly, even though Aegisub could not.
The downside is of course that it is not "true" RGB and will fail with other 
subtitle filters who do not implement the workaround and using it with the 
madVR interface will require additional correction.

To put it using one of his notation examples:
sourceMatrix: unknown
yuvMatrix: TV.BT.601
rgbCorrection: True

@cyber:
difference between (1) and (2)?
"(1)
yuvMatrix: TV|PC.Matrix
rgbCorrection: True|False

OR

(2)
yuvMatrix: TV|PC.Matrix
rgbCorrection: True|False"

Original comment by sneaker...@googlemail.com on 4 May 2012 at 5:59

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

> I think cyberbeing wants Aegisub to fall back to assuming BT.601 in the case 
> it stumbles upon an unknown matrix

No that's not what I want. What you state only makes sense without explicit 
tagging. 

In the case with both video and aegisub matrix tagged, it would be the user's 
job to correct the video matrix if aegisub guessed incorrectly.

> basically the workaround we are currently using

I'm against continuing the workaround we are currently doing indefinitly.

> difference between (1) and (2)?

Ack, don't know how that happened. I've fixed it below. I'll delete the other 
post.

_______________________________________________
_______________________________________________

> But I fear you may still dislike it. 

I do indeed.

As stated already in the comments that followed, one of the goals is to allow 
VSFilter to safely use the correct matrix, and not being stuck using 
BT.601-only forever. In other words, making it possible to match Libass 
implementations more often (as they've long since rejected mimicking VSFilter's 
BT.601 only behavior).

madshi, please individually address your problems to each and every one of the 
examples (1), (2), and (3) I list below. I want to hear what problems they pose 
for you personally, as the author of madVR. Over the last few comments you've 
been, intentionally or not, ignoring all my questions asking for clarification 
about what is preventing you from even considering anything with explicit 
tagging.
___
(1)
yuvMatrix: TV|PC.Matrix
rgbCorrection: True|False

OR

(2)
yuvMatrix: TV|PC.Matrix
sourceMatrix: TV|PC.Matrix

OR

(3)
sourceMatrix: TV|PC.Matrix <--- Optional
yuvMatrix: TV|PC.Matrix <--- Required
rgbCorrection: True|False <--- Required
___

(4)
Explicitly tagging "yuvMatrix" (matrix used by Aegisub) makes YCbCr output as 
simple and easy to understand as possible. 

Tagging "rgbCorrection true|false" makes RGB output as simple and easy to 
understand as possible. 

Explicitly tagging "sourceMatrix" + "yuvMatrix" allows solving uncommon 
problems with the other two in as simple and easy to understand way as possible.

(5)
As long as the new interface is RGB only, why do you even care what we do 
involving YCbCr output? Do you have some hidden agenda to leave VSFilter 
broken? Every time I suggest something that would allow xy-VSFilter to be less 
broken, you reject it. Why is that? The longer we continue this debate, the 
more it seems your disagreeing just for the sake for disagreeing. If I'm 
missing something, please clarify your personal stance.

(6)
I'll make this easier for you. 

Do you believe there is an advantage of having the correct matrix flagged in 
the video bitstream? 

If yes, how is this any different than a desire to define the correct matrix to 
use for YCbCr output with our RGB subtitle format?

The advantages of both are identical. 
The risks of not doing either are also identical.

If we were designing a new subtitle format from scratch, explicitly defining 
the correct YCbCr matrix to use would be near the top of my list of important 
items, for the exact same reasons the H.264 spec allows the correct YCbCr 
matrix to be flagged in the bitstream.

Original comment by cyber.sp...@gmail.com on 4 May 2012 at 6:15

[GoogleCodeExporter]

> In the case with both video and aegisub matrix tagged, it would be the user's 
job to correct the video matrix if aegisub guessed incorrectly.

Yes, but it's still the same as the existing workaround, isn't it? I.e. Aegisub 
and vsfilter using a known matrix instead of the actual video matrix. Granted, 
it doesn't have to be BT.601 but could also be e.g. BT.709. But how would a 
user tag a matrix unknown to any program, if no corresponding trivial matrix 
name has been agreed on?

Original comment by sneaker...@googlemail.com on 4 May 2012 at 6:52

[GoogleCodeExporter]

> As stated already in the comments that followed,
> one of the goals is to allow VSFilter to safely
> use the correct matrix, and not being stuck using
> BT.601-only forever. In other words, making it
> possible to match Libass implementations more
> often (as they've long since rejected mimicking
> VSFilter's BT.601 only behavior).

You've said yourself that VSFilter is the spec. So now you want to change the 
spec to accomodate developers who're too stubborn to follow the spec?

> (1)
> yuvMatrix: TV|PC.Matrix
> rgbCorrection: True|False

Why is there TV vs PC here? The topic was brought up a while ago and as far as 
I remember, *everybody* (except maybe you) agreed that we should leave levels 
as hard coded PC.

> Explicitly tagging "yuvMatrix" (matrix used
> by Aegisub) makes YCbCr output as simple and
> easy to understand as possible. 
> Tagging "rgbCorrection true|false" makes RGB
> output as simple and easy to understand as
> possible. 

The problem with all 3 examples is one and the same: You simply CANNOT separate 
YCbCr from RGB rendering. If you try to do that, users will get different 
colors depending on whether they use VSFilter YCbCr rendering or the new 
interface. That cannot be what we aim for. If we do try to make things clear 
then we need a way to unify YCbCr and RGB rendering so both look exactly the 
same. And that means if you add loads of funny YCbCr specific options, the RGB 
rendering path has to take them all into account, too, or else RGB rendering 
will look different to YCbCr. That's a major pita. From the view point of RGB 
rendering it's a painful to look at multiple new ASS header fields targetted at 
YCbCr rendering and trying to interpret them correctly, so that RGB rendering 
produces identical results.

We cannot use different options for YCbCr and RGB (unless they basically say 
exactly the same thing in just different ways). Whatever we decide, it has to 
be clear, simple and usable for both YCbCr and RGB.

> As long as the new interface is RGB only, why
> do you even care what we do involving YCbCr
> output?

See above.

> Do you have some hidden agenda to leave VSFilter
> broken?

Check my latest suggestion (to stay at hard coded BT.601) again. It makes it 
extremely easy for VSFilter and xy-vsfilter to work perfectly fine. Actually my 
suggestion makes it *much* easier for VSFilter and xy-vsfilter to render YCbCr 
with correct colors. No more worrying about which matrix to use. You'd just 
need to go back to how VSFilter always did things in the past and YCbCr 
rendering would automatically be perfect. So I don't see how you could suggest 
I had some hidden agenda or something.

> Do you believe there is an advantage of
> having the correct matrix flagged in the
> video bitstream?

Sure, for YCbCr video. But not for RGB video (e.g. FRAPS recordings).

> If yes, how is this any different than a
> desire to define the correct matrix to use
> for YCbCr output with our RGB subtitle format?

You've got to make up your mind whether ASS colors are to be interpreted as RGB 
colors or as YCbCr colors (just converted to RGB for storage). If the ASS 
colors are strictly RGB then there's zero reason to add color matrix 
information because the RGB values themselves are the law. If the ASS colors 
are to be interpreted as YCbCr colors then we can either add the necessary 
matrix information, or we can always hard code to BT.601 (which would be much 
simpler cause it's as it has always been treated by VSFilter, anyway). In 
either case both VSFilter with its YCbCr rendering and the new interface with 
its RGB rendering has to interprete the colors the same way.

If we decide to consider the ASS colors to be RGB then the new RGB interface is 
the one who can render natively with no conversions. In this case the way the 
RGB interface renders is the law and VSFilter has to adapt to achieve exactly 
the same output. The only way to do it would be to use the inverse of the video 
renderer matrix. That way RGB rendering and YCbCr rendering would be 100% 
identical.

If we decide to consider the ASS colors to be YCbCr (just stored as RGB) then 
VSFilter can render the colors natively with no conversions (oh well, one 
conversion would be necessary because the colors are stored as RGB). In this 
case what YCbCr rendering produces is the law and the RGB renderer has to do 
additional work to match the YCbCr output.

Pick one. Either one is fine with me. You can't mix them. If you're concerned 
about libass then the right way to handle things is probably to consider ASS 
colors as strictly RGB (according to my understand that's how the libass devs 
see it), and then VSFilter and xy-vsfilter need to be modified to do their best 
to mimick untouched RGB rendering. If you're most concerned with VSFilter YCbCr 
rendering producing correct colors then you can't get the libass devs into the 
same boat at the same time. I suppose libass renders in RGB, too? If so, I 
highly doubt they'd be willing to add a number of matrix corrections to make 
libass rendering produce the same results as VSFilter with your "explicit 
matrix" tagging.

Pick your poison. What should be view ASS subtitle colors as? RGB or YCbCr? 
Which rendering method (RGB or YCbCr) should be considered the law and which 
rendering method should have to do extra work to make sure it matches the other 
rendering method?

> If we were designing a new subtitle format
> from scratch, explicitly defining the correct
> YCbCr matrix to use would be near the top of
> my list of important items, for the exact same
> reasons the H.264 spec allows the correct YCbCr
> matrix to be flagged in the bitstream.

First of all you'd have to decide whether the colors in your new subtitle 
format are RGB or YCbCr. If they're YCbCr then your comparison to the h264 spec 
is valid. But then you should store the values directly as YCbCr values in the 
first place. Which would then also force libass devs to treat them as YCbCr 
values. But if you create a new subtitle format and decide to use RGB, then no 
matrix information needs to be stored, in the same way that e.g. PNG bitmap 
files don't contain matrix information.

Original comment by mad...@gmail.com on 4 May 2012 at 7:05

[GoogleCodeExporter]

E.g. let's take one of the examples you mentioned earlier:

Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Guess matrix BT.709 by resolution
VSFilter: Guess matrix BT.709 by resolution (if YCbCr, else untouched RGB)
madVR: user-selected correct matrix

If we use explicit matrix tagging then with VSFilter YCbCr rendering the 
subtitle colors would match the video colors regardless of whether the user 
selects the correct matrix in madVR or not. But with untouched RGB rendering 
(or libass rendering, I suppose) video and subtitle colors would not match if 
the user selects the correct matrix. This is bad. We need to make sure YCbCr 
and RGB rendering always produce identical results. It makes no sense to have 
them differ.

So, as I said in my previous post: We've got to finally make up our minds 
whether we want ASS colors to be considered native RGB or YCbCr colors. Either 
one is fine with me. But everything else depends on that decision. If we decide 
to consider them YCbCr then IMHO the best solution is to stick to BT.601 to 
keep things simple. Of course we could also use explicit matrix tagging in that 
case, but it would make things more complicated, it would require the RGB 
rendering path to do extra work and it would probably not be accepted by the 
libass devs. If we decide to consider ASS colors native RGB then untouched RGB 
rendering is the law and then it would be xy-vsfilter's job to make sure it 
always matches untouched RGB rendering. The one and only matrix needed for that 
would be the inverse of the video renderer's matrix.

Original comment by mad...@gmail.com on 4 May 2012 at 7:32

[GoogleCodeExporter]

> If we decide to consider ASS colors native RGB then untouched RGB rendering 
is the law and then it would be xy-vsfilter's job to make sure it always 
matches untouched RGB rendering. The one and only matrix needed for that would 
be the inverse of the video renderer's matrix.

What for?

Original comment by sneaker...@googlemail.com on 4 May 2012 at 7:46

[GoogleCodeExporter]

*If* we decide that RGB is the law and YCbCr rendering should mimick it, then 
VSFilter and xy-vsfilter need to use the inverse of the video renderer's matrix 
to convert the ASS RGB colors to YCbCr, because that's the only way to make 
sure that the YCbCr rendering method will produce the same results as untouched 
RGB rendering in all situations.

As I said, I'm (almost) ok with anything, but if we do change the spec then at 
least we should have a clear definition of what the ASS colors mean exactly in 
the new spec and then every rendering method (xy-vsfilter YCbCr, xy-vsfilter 
RGB, libass) has to make sure to follow the spec to the letter. It is not an 
option to have different rendering methods produce different colors with the 
new spec. It should be our absolute minimum requirement to clearly define what 
the ASS colors mean exactly and then to have every rendering method produce the 
same and correct output.

Hopefully everyone can agree with that?

Original comment by mad...@gmail.com on 4 May 2012 at 8:01

[GoogleCodeExporter]

Ah, I thought "untouched RGB rendering" meant using the new interface. You are 
right of course for the current way of vsfilter sitting between decoder and 
renderer.

Original comment by sneaker...@googlemail.com on 4 May 2012 at 8:08

[GoogleCodeExporter]

> So now you want to change the spec to accomodate developers 
> who're too stubborn to follow the spec?

I always did, so long as maintaining legacy VSFilter compatibility was 
possible. 

It goes a long way towards fixing a problem with the spec being based on 
VSFilter behavior. It also makes things easier on users who want to create 
scripts which work correctly outside of VSFilter.

> Why is there TV vs PC here? The topic was brought up a while ago 
> and as far as I remember, *everybody* (except maybe you) agreed
> that we should leave levels as hard coded PC.

Yes, we all agreed specifically that RGB output over the new interface should 
always be PC levels as I remember it.

Having levels for PC range YCbCr output support is still useful.

> You simply CANNOT separate YCbCr from RGB rendering. If you try to do that,
> users will get different colors depending on whether they use VSFilter YCbCr
> rendering or the new interface. That cannot be what we aim for. If we do try
> to make things clear then we need a way to unify YCbCr and RGB rendering so 
> both look exactly the same.

Now we're getting somewhere. This seems to be where we have a difference of 
opinion.

You seem to be saying you rather both YCbCr output and RGB output match each 
other but not the video, than YCbCr alone being correct because correct RGB 
output is impractical. 

I see why you'd want that, but that's absolutely crazy. If this is your main 
concern, you are just going to need to suck it up and accept either a complex 
explicit tagging solution so this never occurs, or just accept that RGB output 
will sometimes be incorrect.

I refuse to cripple YCbCr output when it could otherwise be avoided, just 
because getting correct RGB output is a major pita without extreme complexity.

That's why I say we should look into AYUV for the new interface once again, as 
it eliminates this problem.

> And that means if you add loads of funny YCbCr specific options, the RGB 
> rendering path has to take them all into account, too, or else RGB rendering
> will look different to YCbCr. That's a major pita. From the view point of
> RGB rendering it's a painful to look at multiple new ASS header fields
> targetted at YCbCr rendering and trying to interpret them correctly, 
> so that RGB rendering produces identical results.

Could you please give specific examples how the following causes trouble for 
RGB output?

yuvMatrix: TV|PC.Matrix
rgbCorrection: True|False

> In this case what YCbCr rendering produces is the law and the RGB renderer
> has to do additional work to match the YCbCr output.

This is exactly how it should be. In the majority of cases, this would result 
in untouched RGB output.

> Check my latest suggestion (to stay at hard coded BT.601) again. 

That equals leaving VSFilter broken. A requirement for us is allowing VSFilter 
to use the correct matrix which matches the video, without breaking legacy 
compatibility.

> Sure, for YCbCr video. But not for RGB video (e.g. FRAPS recordings).

Obviously. For native RGB video you would output RGB.

> You've got to make up your mind whether ASS colors are to be interpreted
> as RGB colors or as YCbCr colors (just converted to RGB for storage). 

Nothing to make up my mind about. I've consistently always stated the following:

With YCbCr video:
"YCbCr colors (just converted to RGB for storage)"

With native RGB video:
"interpreted as RGB colors"

> If you're most concerned with VSFilter YCbCr rendering producing correct 
> colors then you can't get the libass devs into the same boat at the same time.

That plan was to expect the Libass devs to do absolutely nothing. Explicit 
tagging allow use of YCbCr matrix & untouched RGB in the majority of cases and 
match Libass. In cases which we don't, we will be more correct than Libass and 
have an advantage. 

Correctness > Matching Libass > Not Matching Libass

Explicit tagging puts the power of matching Libass in the user's hands.

Original comment by cyber.sp...@gmail.com on 4 May 2012 at 8:33

[GoogleCodeExporter]

> matrix unknown to any program

Not a matrix unknown to any program. 

A known matrix which cannot be automatically guessed with reliability by any 
program.

Pretend you have an theoretical BT.800 matrix video, but Aegisub guesses BT.709 
and incorrectly tags the script as:

sourceMatrix: BT.709
yuvMatrix: BT.709

If the user knew their video was BT.800, they would fix the script as follows 
to be correct:

sourceMatrix: BT.800
yuvMatrix: BT.709

> if no corresponding trivial matrix name has been agreed on?

Why wait to define the trivial matrix names? A good start would be all the ones 
in the the H.264 specification. New ones would be added whenever they come into 
existence.

>> I think cyberbeing wants Aegisub to fall back to assuming BT.601 in the case 
>> it stumbles upon an unknown matrix

> No that's not what I want. What you state only makes sense without explicit 
tagging.

Actually, it may only make sense without explicit tagging for YCbCr output. I 
believe my earlier example from Comment #20 would be broken with RGB output.

It would seem that Forcing BT.601 breaks double-corrected RGB output whenever 
an unknown/unsupported matrix is encountered. So tgoyne's idea to workaround my 
concern with YCbCr output, would actually be harmful to the new interface which 
uses RGB.

Could someone confirm this?

> "Color Repair needed: yes" [tgoyne's idea]
>
> Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
> Aegisub: Force BT.601 + "color repair needed: yes"
> VSFilter: BT.601 (if YCbCr, else double corrected RGB)
> madVR: Guess matrix BT.709 by resolution
> Result: Subtitles correct(?), decoded matrix wrong
> (this one confuses me a bit, can someone confirm if the 
> double-corrected RGB subtitles are correct in this case?)

Original comment by cyber.sp...@gmail.com on 4 May 2012 at 8:35