Closed GoogleCodeExporter closed 8 years ago
Issue 40 has been merged into this issue.
Original comment by cyber.sp...@gmail.com
on 30 Apr 2012 at 10:25
> I'm worried that xy-VSFilter will use a matrix
> which is different than that which was used to
> define the RGB colors, when outputting subtitles
> as YCbCr.
Can you help me out here? I'm not sure how Aegisub works inside. Does it also
use the external VSFilter? If so, there are 2 potentially different matrixes
involved during creation of an ASS file: The one used by VSFilter (RGB ->
YCbCr) and the one used by Aegisub (YCbCr -> RGB). Correct? In this case by
making sure that xy-vsfilter will use the same matrix that VSFilter used in
Aegisub will only be a partial solution. The VSFilter matrix could have been
wrong. The Aegisub matrix could have been wrong. But could have been wrong. The
xy-vsfilter matrix could be wrong during playback. The video renderer's matrix
could be wrong during playback. Basically we have *four* matrixes involved
here, two during creation of the ASS file and two during final playback. Do I
get this right? If so, then the only way to correct all possible problems would
be for xy-vsfilter to get access to all other 3 matrixes to calculate the
matrix is should use. I think this is what we named "triple correction" earlier.
Anyway. I think we should first discuss the following questions:
(1) Can we limit ourselves to only support new ASS files that were created with
VSFilter and Aegisub both using the correct matrix? Or do you we have to
support ASS files which were created with either VSFilter or Aegisub (or both)
using a wrong matrix? (Of course those old hard coded BT.601 ASS files are a
special case which we have to support, we all agree on that.)
(2) Can we limit ourselves to only support decoders and video renderers which
are using the correct matrix during playback?
If we can limit both (1) and (2) then there's no need to store matrix
information in the ASS file. Agreed? So cyber, it seems you want to support a
wrong matrix somewhere? Or am I misunderstanding you?
Original comment by mad...@gmail.com
on 30 Apr 2012 at 10:29
[deleted comment]
madshi, I don't exactly know, but I think in the original thread, it was hinted
somewhere that AegiSub converts the video to RGB first and then uses VSFilter
to overlay the subtitles onto the picture. So, while VSFilter would still be
used, the RGB conversion would have been performed using the (sometimes
guessed) correct matrix with the subtitles themselves being in pure RGB.
Original comment by TheDarkS...@gmail.com
on 30 Apr 2012 at 11:26
Aegisub works almost exclusively with RGB video - the only time it actually
sees YUV video data at all is when using the y4m reader, and the first thing
that's done there is convert the video to RGB. As such, VSFilter is passed RGB
video and does no colorspace conversions. libass is passed no video at all and
Aegisub blends the ARGB subtitles into the video itself.
Perhaps one solution to the unsupported color matrix issue would be to make
Aegisub never guess the color matrix, and always use the backwards compatible
forced BT.601 mode if the video is not explicitly flagged with a recognized
color matrix. Wouldn't this completely eliminate the case that cyberbeing is
worried about?
Original comment by tgoyne
on 30 Apr 2012 at 11:47
[deleted comment]
> Wouldn't this completely eliminate the case that cyberbeing is worried about?
Yes it would, at least for VSFilter.
The main problem would then be if Libass implementations refuse to implement
this with YCbCr/RGB output. I was hopeful we could come to a solution which
enabled VSFilter to usually use "correct" matrix without any changes required
to Libass implementations.
We would be stuck with BT.601 subtitles continued to being used on the majority
of videos. It's unfortunate, but the majority of user encoded video never had
the matrix specified when encoding.
Maybe it's about time we include the MPlayer/MPlayer2 and VLC developers in
these discussions?
Original comment by cyber.sp...@gmail.com
on 1 May 2012 at 12:37
Doesn't libass already use the correct video matrix to do RGB->YCbCr
conversion? (I don't know personally, but that's the impression I got from
previous discussion.) If so, I don't see any compatibility problems with the
new approach: both libraries would use the correct colorspace. Old subtitles
that assume the conversion is always with BT.601 would be still be broken in
libass, but that can't be helped.
Original comment by yuriks...@gmail.com
on 1 May 2012 at 12:59
Libass outputs untouched RGB, which currently means colors are incorrect for
the majority of scripts which are tuned for VSFilter (BT.601) compatibility.
Original comment by cyber.sp...@gmail.com
on 1 May 2012 at 1:05
If libass outputs pure RGB, then what's the problem? Correct colors in the ASS
will produce correct colors in the output. Broken (aka legacy-VSF compatible)
will still be wrong just like it is now, but we can't do anything about that.
(Short of adding the hack to libass with heuristics for detection, aka, the
reverse of what is being proposed here, but that's a discussion that doesn't
belong here IMO.)
Ultimately, it's a problem that will fix itself when correct-color ass files
become the standard, without any change in libass.
Original comment by yuriks...@gmail.com
on 1 May 2012 at 1:13
> Ultimately, it's a problem that will fix itself when correct-color ass
> files become the standard, without any change in libass.
With what tgoyne just proposed, it will only be fixed for videos with matrix
information contained in the bitstream.
Videos _without_ matrix info in the bitstream are the majority.
Videos _with_ matrix info in the bitstream are uncommon.
If everybody else agrees this is an acceptable solution, I'm okay with it. It
just means we will match Libass less often than if we explicitly tagged the
matrix used by Aegisub in the script.
It's no worse off and slightly better than the current situation, but I was
hopeful for a solution which bridges the gap between VSFilter and Libass more.
Original comment by cyber.sp...@gmail.com
on 1 May 2012 at 1:24
>Videos _without_ matrix info in the bitstream are the majority.
>Videos _with_ matrix info in the bitstream are uncommon.
Then let the presenter keep guessing. Then fetch that guess from it. If the
presenter guessed incorrectly you'll have the incorrect guess and both wrong
guesses will cancel themselves in the end. Or actually, make people damn tag
their bitstreams correctly if they want correct video (not sub, *video*) colors
from now on.
Original comment by yuriks...@gmail.com
on 1 May 2012 at 1:31
I apologize yuriks.br, but as much as I would like to explain everything to
you, I'm going to ignore the first-half of Comment #12, since it's already been
discussed and it will just get us side-tracked.
This debate has gone on too long, at this point everything that can be
discussed has been discussed. I'm am going to begin deleting comments by those
not involved unless they bring something new and useful to the discussion.
Input is welcome, but I ask everybody not involved to keep focused on the
current matrix implementation those involved are discussing (not other
previously discussed possibilities), if they don't want their comments deleted.
Please keep in mind that our GoogleCode Issue tracker is primarily our work
space, not a discussion forum. It's already crazy enough with myself, madshi,
nevcairiel, and plorkyeran debating things. Once we have reached a final
agreement on how to deal with this matrix issue, I'll allow everybody to
comment freely again.
> Or actually, make people damn tag their bitstreams correctly
> if they want correct video (not sub, *video*) colors from now on.
Yes, this is exactly what would need to happen, which is why it's an acceptable
solution.
Original comment by cyber.sp...@gmail.com
on 1 May 2012 at 2:28
Ok. This will be my final comment here.
Just to clarify, I'm not trying to argue for yet another idea, but rather, was
trying to help justify madshi's "no matrix in the script" proposal. Your
counterpoints didn't seem to make sense to me, as in they were bringing up
non-issues.
But I understand that you prefer to keep the discussion between implementors,
and madshi is fully capable of defending his position anyway, so I'll remove
myself now.
Original comment by yuriks...@gmail.com
on 1 May 2012 at 2:52
> Your counterpoints didn't seem to make sense to me,
> as in they were bringing up non-issues.
My counterpoints haven't been complete non-issues, but so far people have been
having trouble grasping what issues I'm actually talking about. They don't seem
to make sense to a lot of people without a fully understanding how VSFilter,
Aegisub, and Libass work, as well as the issues which VSFilter has caused over
the years for the ASS spec. This is exactly why I'm now going to lock down
comments which would sidetrack us, since we may be nearing a conclusion.
As an old ex-fansubber myself, I understand the problems which need to be
solved, as do the Aegisub devs. Both madshi and nevcairiel have never
experienced first-hand the issues which have plagued fansubbers for years,
which is why some of the concepts I'm attempting to express are misunderstood
as non-issues at first glance and need extra explanation.
tgoyne (aegisub dev) understands the potential issue I'm talking about, and
that's a start. He also proposed a solution to that potential issue (that
doesn't involve explicit tagging), which if madshi and nevcairiel agree, we're
done.
This is the best compromise we've come to so far for a simplified solution. If
madshi and nevcairiel reject it, we're back to the debate over explicit matrix
tagging, which I still believe is the better solution because of Libass
concerns. (It would also be nice to solve uncommon levels issues along the way,
but that's insignificant in the big picture.)
The key point here, is a change like this to the ASS spec is going to be a
one-time thing. It doesn't just need to work perfectly today, but also 10+
years into the future. If we're not careful, we run the risk of causing yet
another matrix problem even worse than the current BT.601-only landscape caused
by VSFilter.
That is why so far I've been promoting 'providing information only' tagging
solutions which are flexible enough to solve any problem with embedded
subtitles (aka MKV softsubs), yet at the same time doesn't force solving
absurd/abusive problems on anybody. The only argument against this is the
potential of unwanted complexity, which is indeed a valid concern. Unwanted
complexity should be avoided, but not at the expense of a solution which could
end up broken long-term.
I remain hopeful that madshi and nevcairiel will accept tgoyne's idea, as I
have done. If tgoyne's idea is rejected, we're seriously running out of
implementations which will be acceptable for everybody, especially if explicit
matrix tagging remains a no-go.
Original comment by cyber.sp...@gmail.com
on 1 May 2012 at 6:06
@cyber, it would help a lot if you could formulate an example case where there
would be problems with the "color repair needed: no" approach, while your
approach would work ok. Can you find and completely spell out such a case?
Which color matrix would the contained in the video (if any)? Which would be
used by Aegisub? Which by the video renderer? Which by VSFilter with either
"color repair needed: no" or your preferred approach? Hopefully, when I see
such an example fully explained, so I can see why your approach improves things
and mine does not, maybe we'll finally be able to get to the same page. Thanks.
Currently I'm wondering if we should change anything *at all*. Maybe we should
just have Aegisub go back to the old ways and always use BT.601. We could even
"officially announce" that ASS subtitles are specced to always be written for
BT.601. The disadvantage would be that it's ugly and known to be "wrong", but
at least all ASS files would stay the same, and everybody knew how ASS files
are to be interpreted. If it's official that ASS is always (has always been and
will always be) BT.601 then it should also be possible to have libass implement
the necessary color correction.
Original comment by mad...@gmail.com
on 1 May 2012 at 7:05
> Perhaps one solution to the unsupported color matrix issue would be to make
Aegisub never guess the color matrix, and always use the backwards compatible
forced BT.601 mode if the video is not explicitly flagged with a recognized
color matrix. Wouldn't this completely eliminate the case that cyberbeing is
worried about?
I think that's not as good an idea as it may seem: AegiSub seems to guess the
colourmatrix the same way a video renderer guesses the colourmatrix when the
bitstream isn't flagged. This is probably the reason most streams aren't
flagged correctly, since the video renderer guesses right anyway. And if the
colourmatrix differs from the guess the video renderer is expected to make, it
is (or at least should be) flagged to the bitstream anyway. This means that
behaviour would actually be counter-productive, since, as I understood it, the
goal was to display (at least the majority of) ASS subtitles in untouched RGB,
while this behaviour would only promote the old workaround.
Also, I believe that scripts would need to be flagged either way, because I'm
sure that there are at least some scripts out there that are not specifically
created for VSFilter's BT.601 limitation, but actually use the correct colours
(maybe the creator didn't know about the issue, or maybe it was written for
libass, or maybe the creator simply didn't care). For those scripts, that are
currently unflagged, the subtitle renderer wouldn't even be able to guess
correctly because it would assume that all "unflagged" scripts need the BT.601
workaround.
Personally, I think that, since the script has to be flagged either way, there
should at least be an option to control the conversion of unflagged bitstreams.
Original comment by TheDarkS...@gmail.com
on 1 May 2012 at 8:45
> it would help a lot if you could formulate an example case where there
> would be problems with the "color repair needed: no" approach, while
> your approach would work ok.
Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Guess matrix BT.709 by resolution + "color repair needed: no"
Result: BT.709 subtitles on non-BT.709 video
Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Never guess matrix and use BT.601 + "color repair needed: yes"
Result: BT.601 subtitles on non-BT.601/BT.709 video
Video Bitstream: Empty (PC-range YCbCr)
Aegisub: PC-Range RGB
VSFilter: TV-range YCbCr
Result: TV-range subtitles on PC-range video
Video Bitstream: Empty (TV-range RGB)
Aegisub: PC-Range RGB
VSFilter: PC-range RGB
Result: PC-range subtitles on TV-range video
Video Bitstream: Flagged (actual matrix != BT.601 or BT.709)
Aegisub: Actual matrix supported
VSFilter: Actual matrix supported (RGB output)
Video Renderer: Actual matrix NOT supported
Video Bitstream: Doesn't matter
Subtitle Editor: Not Aegisub, and doesn't agree with how Aegisub chooses matrix
VSFilter: Only matches Aegisub, creating a matrix mismatch
Result: Incorrect colors
Video Bitstream: Doesn't matter
Subtitle Editor: Not Aegisub, and doesn't agree with how Aegisub chooses matrix
Libass: Untouched RGB
Video Renderer: Only matches Aegisub, creating a matrix mismatch
Result: Incorrect colors
Video Bitstream: flagged (actual matrix != BT.601 or BT.709)
Aegisub: Force BT.601 + "color repair needed: yes"
Libass: Untouched RGB
Video Renderer: Actual matrix
Result: Incorrect colors
Video Bitstream: flagged (actual matrix != BT.601 or BT.709)
Aegisub: Auto-guess BT.709 + "color repair needed: no"
Libass: Untouched RGB
Video Renderer: Actual matrix
Result: Incorrect colors
Video Bitstream in Aegisub: Full-range RGB
Aegisub: "color repair needed: no"
Final Muxed Video: Unflagged YCbCr
Subtitle Filter: Guesses incorrect matrix
Result: Incorrect colors
Video Bitstream in Aegisub: Flagged YCbCr (actual matrix != BT.601 or BT.709)
Aegisub: "color repair needed: no"
Final Muxed Video: Flagged YCbCr (BT.709)
Result: Unable to identify the problem by only looking at the script
If you look through my two week discussion with sneakerger and thedarkspaces,
you can probably find more examples. Keep in mind that Libass implementations
and hardware players are likely going to ignore anything we do here. Providing
information only in a "use if needed/desired" way avoids drama. It offers a
solution, but doesn't force the solution on anybody. There is also the fact
that explicit tagging means less work for YuZhuoHuang and any other subtitle
filters who plan to implement what we've discussed here. The sky is the limit,
since any problem could potentially be solved (or at least identified before it
becomes a problem) by explicitly tagging two matrixes.
Again, the key point here is while many of these things may not be a problem
today, they could be in the future. The majority of problems would arise if
BT.709 is ever replaced by some other matrix.
That said, tgoyne's idea which may cause "incorrect" BT.601 to be used more
often, at least avoids any problems BT.709 being replaced. If you want
extremely simplified solution, this is the only viable option. By using
explicit matrix tagging, an "incorrect" matrix will never be used after this
change, unless the user desires legacy VSFilter 2.39/2.40 compatibility.
__________
In many ways this make me re-think our use of RGB-only vs RGB+AYUV with the new
interface. It's quickly becoming clear that dual-matrix correction of RGB may
become a major pita. AYUV alpha-blending by the video renderer would only
require a single-matrix correction to match the video colors.
madshi, can you refresh my memory of the advantage of using RGB vs AYUV in the
new interface? I suspect it would require some major changes to the madVR
workflow, but would it be worth it to further simplify these subtitle matrix
issues?
__________
> Currently I'm wondering if we should change anything *at all*.
> Maybe we should just have Aegisub go back to the old ways and
> always use BT.601. We could even "officially announce" that ASS
> subtitles are specced to always be written for BT.601.
All this would do is cause drama with Libass who so far has refused to mimic
VSFilter's BT.601 behavior because what they are doing is "correct" (and it
would be if VSFilter didn't exist). In other words, nothing would improve. It
would still be impossible to create a script which has correct colors in Libass
and VSFilter.
Since with this new interface we are going to need to at least some of the time
support BT.601 corrected RGB for VSFilter legacy compatibility, it's the
perfect opportunity to give users the ability to solve this problem going
forward if desired.
> This is probably the reason most streams aren't flagged correctly,
> since the video renderer guesses right anyway.
Exactly, but what happens in the future when a third matrix enters the mix and
replaces BT.709 as the logical option for auto-guessing?
This may never happen, but if it does we're screwed, as we've just created yet
another matrix problem with the ASS spec which could have been avoided. How
good is everybody's crystal ball? Not worth the risk repeating history, and
making the same mistake Gabest did that got us into this mess.
tgoyne's solution of eliminating auto-guessing would avoid this potential
problem, as long as nothing attempts to replace Aegisub with different matrix
behavior. While it certainly wouldn't be my first choice, as it doesn't solve
the underlying problems as much as I would like, it should work with minimal
chance of breakage.
Original comment by cyber.sp...@gmail.com
on 1 May 2012 at 11:14
@cyber, thanks for those examples, but none of them is really complete. I just
wanted one complete example, not a dozen incomplete ones. Let's take the first
example you listed. You didn't spell out what VSFilter would do in that
situation and what the video renderer would do (IYO). You also didn't spell out
how your preferred solution would look like in that situation and how your
solution would change VSFilter's and the video renderer's behaviour for the
better exactly. It is still not obvious to me from the very first example in
your list why your solution would work better. Could you please take one of
those examples (preferably the one you think is the most important, and if
possible one that deals with colors and not with levels) and make it clear how
every software involved would behave with my vs. your preferred solution?
Thanks.
Original comment by mad...@gmail.com
on 2 May 2012 at 6:17
For all of the examples I listed, I just stopped as soon as the matrix used
became "incorrect". Everything after that point is irrelevant. With explicit
matrix tagging, the matrix will never be incorrect as long as it's tagged
correctly, and the matrix needed is supported up until the subtitles are
alpha-blended on the video.
Below are a few variations of the first example which come to mind:
__________
"Color Repair needed: no"
Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Guess matrix BT.709 by resolution + "color repair needed: no"
VSFilter: Guess matrix BT.709 by resolution (if YCbCr, else untouched RGB)
madVR: Guess matrix BT.709 by resolution
Result: Subtitles correct, decoded matrix wrong
"Color Repair needed: no"
Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Guess matrix BT.709 by resolution + "color repair needed: no"
VSFilter: Guess matrix BT.709 by resolution (if YCbCr, else untouched RGB)
madVR: Guess or user-selected correct matrix
Result: Subtitles wrong, decoded matrix correct
"Color Repair needed: yes" [tgoyne's idea]
Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Force BT.601 + "color repair needed: yes"
VSFilter: BT.601 (if YCbCr, else double corrected RGB)
madVR: Guess matrix BT.709 by resolution
Result: Subtitles correct(?), decoded matrix wrong
(this one confuses me a bit, can someone confirm if the double-corrected RGB
subtitles are correct in this case?)
__________
Explicit matrix tagging
Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: User specifies correct YCbCr->RGB video matrix (Aegisub tags matrix
twice to script for untouched RGB)
VSFilter: Renders subtitles with correct matrix (if YCbCr, else untouched RGB)
madVR: Renders video with correct matrix when using new interface (untouched
RGB subtitles)
Result: Subtitles correct, decoded matrix correct
Explicit matrix tagging
Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Guess matrix BT.709 by resolution (Aegisub tags BT.709 matrix twice to
script for untouched RGB, but user edits in the correct Video Matrix so a
correction will be performed)
VSFilter: Renders subtitles with correct matrix (if YCbCr, else
double-corrected RGB)
madVR: Renders video with correct matrix when using new interface
(double-corrected RGB subtitles)
Result: Subtitles correct, decoded matrix correct
Explicit matrix tagging
Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Guess matrix BT.709 by resolution (Aegisub tags BT.709 matrix twice to
script for untouched RGB, but user edits in the correct Video Matrix so a
correction will be performed)
VSFilter: Renders subtitles with correct matrix (if YCbCr, else
double-corrected RGB)
EVR: Guess matrix BT.709 by resolution
Result: Subtitles correct, decoded matrix wrong
__________
Hopefully this is a good enough sampling of the variations possible for the
first example. It's difficult to predict all possibilities with "Color Repair
needed". The problem is that second variation. Someone will also need to
clarify my inquiry about the example for tgoyne's idea.
The "Explicit matrix tagging" options are extremely flexible, so if we go that
route, there could be some discussion about the most practical way to achieve
correct output with the information available.
madshi, after you process the above examples, could you comment on the below
from my previous comment? AYUV would seem to have some advantages for picking
the correct subtitle matrix, wouldn't it? In that way we wouldn't need to worry
about the video matrix at all, since the subtitles would already be
alpha-blended before YCbCr->RGB conversion is performed. We could then just
continue using the Aegisub Dev solution which has been in place since January.
From the xy-VSFilter standpoint, AYUV output isn't a problem since that's what
we use internally with YCbCr output. How much of a roadblock would AYUV
subtitles be to madVR?
__________
In many ways this make me re-think our use of RGB-only vs RGB+AYUV with the new
interface. It's quickly becoming clear that dual-matrix correction of RGB may
become a major pita. AYUV alpha-blending by the video renderer would only
require a single-matrix correction to match the video colors.
madshi, can you refresh my memory of the advantage of using RGB vs AYUV in the
new interface? I suspect it would require some major changes to the madVR
workflow, but would it be worth it to further simplify these subtitle matrix
issues?
__________
Original comment by cyber.sp...@gmail.com
on 2 May 2012 at 8:25
Before this starts in endless discussion again, i'm against AYUV for subtitles.
The interface is meant to be generic, and not match implementation issues of
VSFilter.
The interface tells you which matrix the renderer uses for the RGB conversion,
and with that information you can produce RGB that matches the renderers
output. It doesn't actually solve any problems, it just avoids you having to do
a RGB conversion with the matrix the renderer tells you to use. It only shifts
the RGB conversion to the renderer, it doesn't change anything else.
The big advantage of RGBA for subtitles is that it can be blended onto the
image as one of the last steps in the rendering process, giving the whole
workflow much more time to prepare and render subtitles (ie. madVR can request
the subtitle image as soon as it gets the image from the decoder, but only
needs it for one of the very last rendering steps)
Not to mention generally bad GPU driver support for AYUV surfaces.
Original comment by h.lepp...@gmail.com
on 2 May 2012 at 8:42
(1) If we sum up all the matrixes supported by Microsoft DXVA, MediaFoundation,
MPEG2, VC-1 and h264, we get these:
(a) BT.601 (used by NTSC and PAL)
(b) BT.709 (used by all HD content)
(c) SMPTE 240M
(d) YCgCo
(e) US National TV System 1953
Only BT.601 and BT.709 have any real meaning these days. The other 3 are not
used in any digital consumer content, at the moment, AFAIK. Of course there's
always a chance a different matrix might be used some day, but it doesn't look
too likely to me right now.
I've also checked a draft of the upcoming h265/HEVC codec, rumored to be used
by the future 4K Blu-Ray standard, and there are not any new matrixes
introduced there, at this point in time.
(2) I'd like to point out that a video file which is neither matrixed with
BT.601 nor BT.709 and on top of that doesn't have its real matrix stored in the
video bitstream is IMHO a completely and totally broken encoding. I'm not
really sure if it's worth it for us to worry about such a catastrophic
encoding. Is that the most important case for why you want to have explicit
matrix tagging?
(3) "Explicit matrix tagging
"madVR: Renders video with correct matrix when using new interface"
Why would madVR render video with the correct matrix in this situation? I don't
see that happening. The new interface only supplies madVR with RGB subtitles,
it doesn't change the matrix madVR uses to decode the video. So your example
should read there: "madVR: Guess matrix BT.709 by resolution", same as EVR.
(4) "Explicit matrix tagging
EVR: Guess matrix BT.709 by resolution
Result: Subtitles correct, decoded matrix wrong"
How can the subtitles be correct if the decoded matrix is wrong - when using
the VSFilter way of rendering subtitles in YCbCr? The wrong decode matrix will
also damage the subtitles in that case, or am I wrong?
-------
Let's discuss AYUV later. I prefer discussing one thing at a time.
Original comment by mad...@gmail.com
on 2 May 2012 at 9:36
I don't see how using AYUV with the new interface would have anything to do
with VSFilter (who's native format is RGB), but that's beside the point. Doing
a simple RGB->AYUV conversion certainly seems more generic than
"double-corrected RGB". AYUV would simplify achieving correct subtitle colors
in relationship to a video by only requiring a single matrix (the matrix used
by Aegisub) tagged in the script.
None the less, if AYUV is not seen as a viable option by madshi as well, I
guess we're stuck with the complexity of "double corrected" RGB.
> The big advantage of RGBA for subtitles is that it can
> be blended onto the image as one of the last steps in
> the rendering process
Since we need to let madVR apply gamut/gamma correction to subtitles in the
same way as the video (to avoid yet another subtitle color problem), I'm unsure
this advantage exists. Even if it does, couldn't you theoretically achieve the
same thing with AYUV considering RGB<->AYUV conversion can be lossless?
> Let's discuss AYUV later. I prefer discussing one thing at a time.
Okay. Posting this just because I already had it fully typed up before I
noticed your reply. We can come back to it later if needed.
____________________
____________________
(1) I forgot about checking the h265/HEVC drafts. Since it doesn't list any new
matrix, then that does likely means were safe from 4K video breaking things.
Uncertainty of what the future holds is a risk for us none the less. I'd still
prefer not to take that risk, since I don't see any advantage which "color
repair needed" offers over "explicit matrix tagging".
> Is that the most important case for why you want to have explicit matrix
tagging?
It goes beyond that by potentially allowing more oddball mismatch cases to be
corrected. The "color repair needed" idea only works as long as "Subtitle
Editor" + "Subitle Filter" + "Video Renderer" are in perfect sync as far as
matrix information is concerned. How do we ensure that this occurs when we are
creating a generic interface?
I also find the "color repair needed" workaround way too VSFilter specific and
restrictive. It will never be useful by anybody but those of us in this thread,
so it doesn't belong in the ASS spec.
Could you please explain to me the advantage which "color repair needed" offers
over even a just a single matrix (the one used by Aegisub) in the script? As
you'll notice back in Issue #40
(http://code.google.com/p/xy-vsfilter/issues/detail?id=40#c303) I listed them
as being at identical levels of information. What can "color repair needed"
accomplish which a single tagged matrix can't?
> The new interface only supplies madVR with RGB subtitles,
> it doesn't change the matrix madVR uses to decode the video.
(3) Yet why shouldn't we make it do so? Being told with authority what the
matrix is, would be much better than guessing. Even going back to your Issue
#40 comment it seemed like something you had in mind to avoid needing to have
madVR guess:
http://code.google.com/p/xy-vsfilter/issues/detail?id=40#c292
> How can the subtitles be correct if the decoded matrix is wrong
(4) They were alpha-blended into the YCbCr video with the correct colors, so
the subtitles are correct in relationship to the video. This is an important
concept to remember, since it's the entire reason why I brought up AYUV just
now. Once the subtitles are part of the video, it matters little if the video
renderer uses a wrong YCbCr->RGB matrix from the point of view of subtitle
correctness.
Original comment by cyber.sp...@gmail.com
on 2 May 2012 at 11:20
Ok, I think I'm having a moment of clarity. Let me sum up the situation, as far
as I understand it:
(1) The problem you're most worried about happens when Aegisub (during creation
of the ASS) and VSFilter (during playback of the ASS) use a different color
matrix. If that happens, the subtitle colors will not match the video, when
using the VSFilter way of rendering subtitles in YCbCr.
(2) Which matrix the video renderer uses has no impact on whether subtitle and
video colors match (when talking about VSFilter).
(3) If you would have to sort possible situations according to what you
consider best (or least bad), you'd rate like this:
(a) every is perfect
(b) subtitle and video colors match, but both might be using (the same) wrong
matrix
(c) video colors are correct, but subtitle colors are wrong -> color mismatch
(d) subtitle colors are correct, but video colors are wrong -> color mismatch
(e) subtitle and video colors mismatch, and both are wrong on top of that
(4) With your "Explicit matrix tagging", you would make sure that at least
video and subtitle colors would always match.
(5) With my "color repair needed" option, it is possible that Aegisub and
VSFilter could potentially use a different matrix, resulting in mismatching
subtitle/video colors, when using YCbCr VSFilter rendering.
Is that a correct summary?
Original comment by mad...@gmail.com
on 2 May 2012 at 12:22
> If you would have to sort possible situations...
I'd just simplify it as follows.
Success:
(a) subtitle and video colors match using the correct matrix
Partial Success:
(b) subtitle and video colors match using the wrong matrix
Complete Failure:
(c) subtitle and video colors mismatch
> Is that a correct summary?
Overall, assuming you're talking about YCbCr output only.
After taking some time to sketch out the behavior of some of the discussed
possibilities, something like the following, along with simple intuitive names,
is beginning to grow on me. Enhance the new interface to have the provider send
yuvMatrix to the consumer when rgbCorrection=False, and you could have a
distinct advantage in terms of script portiblity.
yuvMatrix: TV|PC.Matrix
rgbCorrection: True|False
None the less, I continue to favor the idea of tagging both source and aegisub
matrix, even if only for script identification purposes. The prospect of a
subtitle editor having functionality to reverse a matrix mismatch in a script
with reliability, would only be possible using this tagging method. Even for
the new interface, I fail to see how having the potential to solve any problem
on playback would be a bad thing. The hardest part would be deciding how much
to actually support.
sourceMatrix: TV|PC.Matrix
yuvMatrix: TV|PC.Matrix
We could even go all out with three tags, but with the second matrix being
optional. That could allow an optional identification benefit, but without any
expectation to actually use it for anything ourselves. I'd even go so far as to
say that Aegisub shouldn't even be allowed to tag sourceMatrix in such a case.
We could just define what it represents, with the expectation it will be
manually tagged.
sourceMatrix: TV|PC.Matrix <--- Optional
yuvMatrix: TV|PC.Matrix <--- Required
rgbCorrection: True|False <--- Required
Original comment by cyber.sp...@gmail.com
on 3 May 2012 at 11:11
>> Is that a correct summary?
> Overall, assuming you're talking about YCbCr output only.
Ok, so let me sum up what we might want to aim for, for a "good" ASS color
matrix solution:
(1) Subtitle colors should always match video colors.
(2) Both VSFilter YCbCr rendering and the new interface RGB rendering should
produce identical colors.
(3) Aegisub should prevent the user from accidentily screwing up the colors.
(4) Whatever we add to the ASS spec should be as simple and easy to understand
as possible.
Everybody agree?
Well, I have a solution which I think would solve all of the above, plus it's
very simple. But I fear you may still dislike it. Anyway, here goes:
We should simply go back to having old VSFilter be the ASS spec. In detail I
suggest:
(1) We officially announce that ASS RGB colors are always to be interpreted as
BT.601 YCbCr values.
(2) We revert VSFilter and xy-vsfilter back to always blindly using BT.601.
(3) We let Aegisub continue to auto-guess the correct matrix, maybe even allow
users to select any matrix they want. *BUT* Aegisub has to follow the ASS spec
which says that ASS RGB colors are always BT.601 YCbCr values. So whatever
matrix Aegisub is using for video playback, it has to convert the colors back
to BT.601 when saving the ASS file to disk.
(4) For the new RGB interface, we have to require that the final subtitle
consumer (the one which blends the subtitles to video) has to perform the
BT.601 -> whatever matrix correction.
Advantages:
- fulfills all listed requirements
- works with any current and future video matrix
- very easy, no new header ASS header fields needed at all
Disadvantages:
- kinda "ugly" because hard coded to BT.601
- Aegisub needs to be modified
- libass devs might refuse to support this
Honestly, I don't think libass is a problem at all. If e.g. Hendrik decides to
implement a libass subtitle renderer, it would be dead easy for him to add the
necessary corrections on top of libass - that is if he wants to support YCbCr
rendering at all. If he only supports the new RGB interface, it would actually
be correct for libass to not perform the correction - because that would be the
duty of the final subtitle consumer (e.g. madVR), anyway.
Original comment by mad...@gmail.com
on 3 May 2012 at 11:38
[deleted comment]
> If he only supports the new RGB interface, it would actually be correct for
libass to not perform the correction - because that would be the duty of the
final subtitle consumer (e.g. madVR), anyway.
Wait, what?
Who says we always want to send ASS using the new interface? There are other
subtitle formats out there, you know...
That would require renderers of other formats to always do the inverse
correction of the ASS BT.601 workaround.
Also, there already are libass implementations out there. Like mplayer2 and
VLC. And I don't know how you want to "officially" announce anything. This is
why I favor the route to go back to RGB (as soon as xy-vsfilter supports it and
is ready to be used in CCCP). The workaround was vsfilter specific, so we only
need to fix (xy-)vsfilter and make it aware of the colors, i.e. workaround used
or not.
Also, I don't expect any future workarounds to appear, because the currently
used implementation specific workaround was only feasible, because there was
only a single usable implementation in the first place at that time. We
currently have more than one implementation and with a hypothetical
xy-vsfilter->RGB->madVR mode we get yet another one, so future typesetters
won't be as easily tempted to use an implementation specific workaround instead
of fixing their workflow/the players as they were back then.
Original comment by sneaker...@googlemail.com
on 3 May 2012 at 5:27
> Who says we always want to send ASS using
> the new interface?
Nobody.
> There are other subtitle formats out there, you know
Of course the color repair would only be performed for ASS subtitles. I thought
that much was clear.
> Also, there already are libass implementations
> out there. Like mplayer2 and VLC.
Yes. What is your point? Whether we add a "workaround used" marker (your
preferred solution) or whether we add explicit matrix information (cyber's
preferred solution) or whether we use my latest suggestion, it won't change a
thing about how the already existing libass implementations will treat the
trillions of ASS files which are already out there.
Original comment by mad...@gmail.com
on 3 May 2012 at 5:47
> Nobody.
> Of course the color repair would only be performed for ASS subtitles. I
thought that much was clear.
It wasn't clear, but it is now. Anyhow, I don't think this a job for the video
renderer. If one wants to do it, it should be left to the subtitle filter and
the renderer could communicate the matrix to it, as with the "yuvMatrix"
DirectShow attribute proposed by cyberbeing. If other renderers would want to
adopt this interface as well, they also would have to implement this
correction, while it is originally a problem that is rather specific to a
single implementation and format.
>Yes. What is your point? Whether we add a "workaround used" marker (your
preferred solution) or whether we add explicit matrix information (cyber's
preferred solution) or whether we use my latest suggestion, it won't change a
thing about how the already existing libass implementations will treat the
trillions of ASS files which are already out there.
Yes, these players (along with MPC-HC) are currently assuming RGB and just like
among us participating in this discussion, there isn't any agreement around the
world on how to interpret colors. Some devs will continue to interpret unmarked
scripts as true RGB and we won't be able to convince them otherwise, as there
is no way to "officially" announce anything in the ASS world. I can however see
them willing to assume BT.601 workarounds (or in their eyes "incorrect"
colors/a broken script) as long as the typesetter has explicitly marked his
script as such. We only have influence on xy-vsfilter, madVR and Aegisub, which
we should use to return to true RGB with marked scripts so future players (an
already existing VLC, mplayer2, MPC-HC) will all display them correctly.
Some assumptions made during the turn of this discussions are not true:
- everyone agreeing on unmarked scripts being "BT.601 workaround" (libass devs,
etc.)
- everyone wanting to match their subtitle colors to their video colors. The
drawing of rectangles on a video for typesetting is only a very small part of
ASS. Many people just want to have their dialogue using a pretty font and the
RGB color they chose.
Original comment by sneaker...@googlemail.com
on 3 May 2012 at 6:25
I'm against changing the spec to BT.601 YUV for the following (already
mentioned) reasons:
(1) Our original goal was to create an interface that would transmit RGBA data
to the renderer, and only in special, legacy VSFilter cases to perform a Matrix
Compensation (any goal may change over time, though, and I admit that)
(2) There are already subtitle scripts out there that don't follow the BT.601
legacy requirement for VSFilter. Possible reasons include, but are not limited
to:
(2.1) the script creator being unaware of VSFilter's limitation to BT.601
(2.2) the script originally not being written for VSFilter
(2.2.1) but for libass
(2.2.2) but for xy-VSFilter with matrix guessing enabled
(2.3) the script creator simply not caring about the limitation, expecting it
to be fixed sometime (soon)
(3) I assume a secondary goal was also at least some sort of interoperatibility
between subtitle providers, and honestly, if ASS was designed to be rendered in
YUV in the first place, it wouldn't use BGR colour codes, it would store the
colours in YUV.
By the way, I'm not sure whether a simple field of whether Matrix Compensation
has to be used or not will be actually useful for this interface, because if it
is used, it indicates that xy-VSFilter has to render the subtitles in BT.601
YUV (I assume 4:4:4 would be the most useful variant, but I don't really know
much about YUV) directly (which it is capable of) and either send it to madVR,
which then will proceed to convert it to RGB using the video's colour matrix,
or get the colour matrix madVR is using via the interface and use it to convert
the YUV image to RGB. The alpha channel is completely unaffected, which is why
I conveniently left it out.
However, if explicit matrix tagging is used, xy-VSFilter will have to check
whether the matrix the Script Matrix field and matrix used in the Video Matrix
field match. If they match, it will treat the script colours as raw RGB and
send them to madVR without any change. If they don't match, xy-VSFilter will
proceed to render the subtitles directly in the YUV colourspace used as Script
colourspace and either send the YUV image to madVR along with the information
of what matrix madVR should use to convert the image back to RGB or convert the
image to RGB using the flagged Video Matrix and send the RGB data to madVR.
I do, however, not see any reason to perform a "triple correction" of sorts for
different sources: When madVR gets the data from the subtitle renderer, the
colours already match (or will be converted to match, doesn't matter) those of
the "source video" which was used to create the subtitles. As far as my
understanding goes, it is completely irrelevant whether the colours come from
the BT.601 DVD version of a movie, but will be used with the BT.709 HD version
of said movie or whether they come directly from the HD BT.709 version, because
the colour values of the BT.601 version have been converted into BT.709 values
for the HD version. That means, both version (should) decode to the same
colour, and therefore, the colours already match.
Original comment by TheDarkS...@gmail.com
on 3 May 2012 at 6:28
@sneaker:
> there is no way to "officially" announce anything
Probably true, but I think adding something to the Aegisub documentation could
help.
> We only have influence on xy-vsfilter, madVR and
> Aegisub, which we should use to return to true RGB
> with marked scripts so future players (an already
> existing VLC, mplayer2, MPC-HC) will all display
> them correctly.
If we follow my latest suggestion, it would easily be possible for everyone to
achieve perfect colors with pretty much all current and future ASS files. Maybe
libass/mplayer2/VLC devs will refuse to implement the necessary changes. But if
they do, their users should complain. I don't think it's a good enough reason
to change a spec just because some devs do not do their work properly.
> The drawing of rectangles on a video for
> typesetting is only a very small part of ASS
If we ignore this small part of ASS, then why are we discussing colors at all?
No normal user will see the difference between BT.601 vs BT.709, unless we're
talking about subtitle background colors being matched to video colors.
Original comment by mad...@gmail.com
on 3 May 2012 at 7:02
> Maybe libass/mplayer2/VLC devs will refuse to implement the necessary
changes. But if they do, their users should complain. I don't think it's a good
enough reason to change a spec just because some devs do not do their work
properly.
As I view it, there is no real change to the spec. Some people have used true
RGB values, some used the BT.601 workaround and we will probably see both kind
of scripts for quite some time, though I hope that Aegisub pushes users back to
true RGB in the medium to long term. Even now, without any changes, you can't
be sure all users get correct colors because of this. The BT.601 workaround was
a vsfilter specific workaround, so it seems logical so solve that problem in
xy-vsfilter as an evolved vsfilter. The only thing we are changing is to add
some optional information, which help xy-vsfilter solve that problem, while
still being backwards-compatible to the old vsfilter (if wanted by the user).
> If we ignore this small part of ASS, then why are we discussing colors at all?
I'm not saying we should ignore it, but we should be aware that this is not
always the author's will. Of course all this is to satisfy the 1% that do care
(like madVR itself), and hopefully to make display as intended even for the
remaining 99% as long as they use up-to-date filters.
But there will always be some problems and no matter how much I think about it,
they are unsolvable. We can't force anyone to agree on the specs and currently
I see vsfilter as being the problem, not the other programs, so going back to
the original true RGB values seems appropriate. Scripts that have been marked
can be correctly displayed as long as the devs care enough to implement the
support (unless the script already matches their default behavior), but we will
continue to have problems with programs whose devs don't. But we already have
this situation right now, so it is no regression however you look at it.
Original comment by sneaker...@googlemail.com
on 3 May 2012 at 7:41
> I'm not saying we should ignore it, but we should be aware that this is not
always the author's will.
I just thought of this earlier, but with matrix information in the ASS script,
the color tab of xy-VSFilter's properities page would need to be reworked
anyway. So, instead of simply adding a "follow BT.601 unless the script says
otherwise" option, there could also be added an option to always enable /
disable Matrix Compensation for scripts that aren't flagged at all.
Original comment by TheDarkS...@gmail.com
on 3 May 2012 at 7:49
As to not get side tracked and ending up repeating the same arguments over and
over again (#33), I will add a point to the requirements listed by madsi (#26):
> (1) Subtitle colors should always match video colors.
> (2) Both VSFilter YCbCr rendering and the new interface RGB rendering should
produce identical colors.
> (3) Aegisub should prevent the user from accidentily screwing up the colors.
> (4) Whatever we add to the ASS spec should be as simple and easy to
understand as possible.
(5) Scripts should be compatible with the majority of programs, regardless of
whether their devs choose to implement any workarounds and additions or not
Original comment by sneaker...@googlemail.com
on 3 May 2012 at 8:28
Everything that isn't vsfilter would instantly benefit from compatibility with
the new standard if pure RGB ends up being used. Furthermore, even if libass
can be convinced to hop aboard, the more things differ from pure RGB, the more
likely it is that someone else will implement it wrong or even not bother
implementing it. Pure RGB seems to be the choice most likely to lead to an
uniform handling of subs in the future, after vsfilter is fixed, IMO.
Original comment by yuriks...@gmail.com
on 3 May 2012 at 8:46
[deleted comment]
cyber: Just as a clarification: (I'm not arguing, I'm trying to present what I
understand as being real and asking for disagreements!)
It's important that H.264 have the matrix used flagged in the bitstream because
H.264 video is stored in YCbCr space. And YCbCr data on it's own is meaningless
from a display standpoint without knowing how to transform that data into RGB
space.
ASS subs, on the other hand, are, in their 'bitstream' (that is, their
plaintext disk representation) in RGB space already, and thus doesn't have any
matrix instrinsically attached to them that is needed to decode them to RGB
colors.
Any YCbCr conversion matrix that will get involved with the subs will come from
the need to convert from that RGB space to or from some YCbCr color space. (For
example, if you're burning them onto video you'll use the matrix specified by
the video, since that's what the renderer will use to show it on-screen later.
They need to match.)
Let's ignore the sub authoring stage for now and say we have an ASS script with
100% certified correct RGB colors.
There are several possiblities during playback: (Correct in the results refers
to correct compared to RGB original.)
1) Renderer is using the correct video matrix. Sub-renderer is doing untouched
RGB.
- Result: decoded video colors match real colors and match subtitle's RGB colors
2) Renderer is using *incorrect* video matrix. Sub-renderer is doing untouched
RGB.
- Result: decoded video colors are wrong. Subtitles are correct but don't match.
3) Renderer is using correct video matrix. Sub-renderer is burning subs onto
video, using renderer's matrix.
- Result: decoded video colors are correct. Due to the script colors also being correct and the correct matrix being used to burn subs, which will be reversed by the renderer's ->RGB conversion, sub colors match video.
4) Renderer is using incorrect video matrix. Sub-renderer is burning subs,
using renderer's matrix.
- Result: video colors wrong. Sub's RGB->YCbCr->RGB conversion is lossless due to sub-renderer matching the renderer's matrix. Sub colors are correct but don't match video.
5) Renderer is using correct video matrix. Sub-renderer is doing it's own thing
and using incorrect matrix, which is different from the renderer's.
- Result: video colors correct. Sub's RGB->YCbCr->RGB round trip will distort colors due to sub-renderer's and renderer's matrices not being matched. Sub colors are wrong and don't match video.
6) Renderer is using incorrect video matrix. Sub-renderer is doing it's own
thing and guessed on the correct video matrix, which is different from
renderer's.
- Result: video colors wrong. Sub's RGB->YCbCr->RGB round trip distorts colors due to sub-renderer and renderer not agreeing on a matrix. Sub colors are incorrect but *do* match video.
7) Renderer and sub-renderer are both using the wrong matrix.
- Result: video colors wrong. RGB->YCbCr->RGB will produce doubly wrong results. Sub colors incorrect and don't match video.
Summarizing: cases 1, 3 and 5 have correct video colors. Cases 1, 2, 3 and 4
have the final sub colors being correct. Cases 1, 3 and 6 additionally have
matching subtitle colors.
I think that covers all cases. 1 and 3 are how it's supposed to work when
functioning correctly, I don't think they need any further comments.
2 and 4 are incorrect results caused by the video renderer's inability to use
the correct video matrix. They produce wrong looking video colors. Even if the
sub colors are correct, they don't match the video's, and thus aren't desirable.
5 is a good case for why the sub-renderer should listen to the video renderer.
If it had, then this would be case 3 and all would be well.
6 supports cyber's proposal. If the renderer gets the video matrix wrong, the
sub-renderer can at least step in and, hoping that the matrix specified in the
script is correct for the current video (which it has no guarantee of being)
salvage the situation by rendering subs with the wrong colors, but that
nevertheless match the video.
On 7 no one has any idea of what's going on and there's not much that can be
done about it.
Now at first expliciting the matrix may seem like an advantageous idea, as per
6, however, that reasoning hinges on a premise that, at least for me is a bit
hard to swallow: It assumes that Aegisub can and will do a better job at
guessing the matrix used in the video than the very video renderer will. I'd
expect the video renderer, whose sole purpose is to play back the formats it
supports, to do a better job at it than aegisub. Why, if the renderer can't
detect or use that format, then it's unsuitable for playing back that kind of
content in the first place. Furthermore, the sub-renderer listening to the
matrix specified in the script can just as easily lead to situation 5. If
you're going to tag the script with the matrix used in the video bitstream, why
not instead simply tag the video bitstream itself? If you tag the script you
now have *two* tags that need to be kept in sync (or have the guessing
algorithm be consistent) at all times.
If we assume that the video renderer is able to correctly decode the video with
correct colors (and it better be) we are left with cases 1, 3 and 5. Of which 5
only exists because the sub-renderer may insist on not following the renderer.
That fallback ends up actually being harmful.
Of course, this whole post hinges on the ASS script being correct. But if it
isn't, I'd have to place the blame on the author, or aegisub. Remember, garbage
in, garbage out. (Perhaps instead of tagging the script aegisub can simply warn
that streams with incorrect matrix color information are likely to result in
broken output?)
I hope that was constructive enough.
(Addendum: Regarding YCbCr/RGB at the start, you could of course argue that RGB
space is also only defined in the presence of color primaries, but in practice
that tends to be a much lesser issue due in part to it being a standardized
part of hardware, not a part of the algorithm, and so there's not much you can
do about it. Furthermore, it globally affects all video output (color
correction schemes notwithstanding) and so doesn't causes a discrepancy between
subs and video, which means it doesn't really matter to us.)
Original comment by yuriks...@gmail.com
on 4 May 2012 at 2:37
@yuriks
> I hope that was constructive enough.
I'll leave it in case madshi has some of the same misunderstandings as you do
in that post.
Original comment by cyber.sp...@gmail.com
on 4 May 2012 at 5:06
> It assumes that Aegisub can and will do a better job at guessing the matrix
used in the video than the very video renderer will.
I think cyberbeing wants Aegisub to fall back to assuming BT.601 in the case it
stumbles upon an unknown matrix, i.e. basically the workaround we are currently
using. This can always ensure matching video and subtitles and in addition
correct overall colors if the renderer is able to detect the video matrix
correctly, even though Aegisub could not.
The downside is of course that it is not "true" RGB and will fail with other
subtitle filters who do not implement the workaround and using it with the
madVR interface will require additional correction.
To put it using one of his notation examples:
sourceMatrix: unknown
yuvMatrix: TV.BT.601
rgbCorrection: True
@cyber:
difference between (1) and (2)?
"(1)
yuvMatrix: TV|PC.Matrix
rgbCorrection: True|False
OR
(2)
yuvMatrix: TV|PC.Matrix
rgbCorrection: True|False"
Original comment by sneaker...@googlemail.com
on 4 May 2012 at 5:59
[deleted comment]
> I think cyberbeing wants Aegisub to fall back to assuming BT.601 in the case
> it stumbles upon an unknown matrix
No that's not what I want. What you state only makes sense without explicit
tagging.
In the case with both video and aegisub matrix tagged, it would be the user's
job to correct the video matrix if aegisub guessed incorrectly.
> basically the workaround we are currently using
I'm against continuing the workaround we are currently doing indefinitly.
> difference between (1) and (2)?
Ack, don't know how that happened. I've fixed it below. I'll delete the other
post.
_______________________________________________
_______________________________________________
> But I fear you may still dislike it.
I do indeed.
As stated already in the comments that followed, one of the goals is to allow
VSFilter to safely use the correct matrix, and not being stuck using
BT.601-only forever. In other words, making it possible to match Libass
implementations more often (as they've long since rejected mimicking VSFilter's
BT.601 only behavior).
madshi, please individually address your problems to each and every one of the
examples (1), (2), and (3) I list below. I want to hear what problems they pose
for you personally, as the author of madVR. Over the last few comments you've
been, intentionally or not, ignoring all my questions asking for clarification
about what is preventing you from even considering anything with explicit
tagging.
___
(1)
yuvMatrix: TV|PC.Matrix
rgbCorrection: True|False
OR
(2)
yuvMatrix: TV|PC.Matrix
sourceMatrix: TV|PC.Matrix
OR
(3)
sourceMatrix: TV|PC.Matrix <--- Optional
yuvMatrix: TV|PC.Matrix <--- Required
rgbCorrection: True|False <--- Required
___
(4)
Explicitly tagging "yuvMatrix" (matrix used by Aegisub) makes YCbCr output as
simple and easy to understand as possible.
Tagging "rgbCorrection true|false" makes RGB output as simple and easy to
understand as possible.
Explicitly tagging "sourceMatrix" + "yuvMatrix" allows solving uncommon
problems with the other two in as simple and easy to understand way as possible.
(5)
As long as the new interface is RGB only, why do you even care what we do
involving YCbCr output? Do you have some hidden agenda to leave VSFilter
broken? Every time I suggest something that would allow xy-VSFilter to be less
broken, you reject it. Why is that? The longer we continue this debate, the
more it seems your disagreeing just for the sake for disagreeing. If I'm
missing something, please clarify your personal stance.
(6)
I'll make this easier for you.
Do you believe there is an advantage of having the correct matrix flagged in
the video bitstream?
If yes, how is this any different than a desire to define the correct matrix to
use for YCbCr output with our RGB subtitle format?
The advantages of both are identical.
The risks of not doing either are also identical.
If we were designing a new subtitle format from scratch, explicitly defining
the correct YCbCr matrix to use would be near the top of my list of important
items, for the exact same reasons the H.264 spec allows the correct YCbCr
matrix to be flagged in the bitstream.
Original comment by cyber.sp...@gmail.com
on 4 May 2012 at 6:15
> In the case with both video and aegisub matrix tagged, it would be the user's
job to correct the video matrix if aegisub guessed incorrectly.
Yes, but it's still the same as the existing workaround, isn't it? I.e. Aegisub
and vsfilter using a known matrix instead of the actual video matrix. Granted,
it doesn't have to be BT.601 but could also be e.g. BT.709. But how would a
user tag a matrix unknown to any program, if no corresponding trivial matrix
name has been agreed on?
Original comment by sneaker...@googlemail.com
on 4 May 2012 at 6:52
> As stated already in the comments that followed,
> one of the goals is to allow VSFilter to safely
> use the correct matrix, and not being stuck using
> BT.601-only forever. In other words, making it
> possible to match Libass implementations more
> often (as they've long since rejected mimicking
> VSFilter's BT.601 only behavior).
You've said yourself that VSFilter is the spec. So now you want to change the
spec to accomodate developers who're too stubborn to follow the spec?
> (1)
> yuvMatrix: TV|PC.Matrix
> rgbCorrection: True|False
Why is there TV vs PC here? The topic was brought up a while ago and as far as
I remember, *everybody* (except maybe you) agreed that we should leave levels
as hard coded PC.
> Explicitly tagging "yuvMatrix" (matrix used
> by Aegisub) makes YCbCr output as simple and
> easy to understand as possible.
> Tagging "rgbCorrection true|false" makes RGB
> output as simple and easy to understand as
> possible.
The problem with all 3 examples is one and the same: You simply CANNOT separate
YCbCr from RGB rendering. If you try to do that, users will get different
colors depending on whether they use VSFilter YCbCr rendering or the new
interface. That cannot be what we aim for. If we do try to make things clear
then we need a way to unify YCbCr and RGB rendering so both look exactly the
same. And that means if you add loads of funny YCbCr specific options, the RGB
rendering path has to take them all into account, too, or else RGB rendering
will look different to YCbCr. That's a major pita. From the view point of RGB
rendering it's a painful to look at multiple new ASS header fields targetted at
YCbCr rendering and trying to interpret them correctly, so that RGB rendering
produces identical results.
We cannot use different options for YCbCr and RGB (unless they basically say
exactly the same thing in just different ways). Whatever we decide, it has to
be clear, simple and usable for both YCbCr and RGB.
> As long as the new interface is RGB only, why
> do you even care what we do involving YCbCr
> output?
See above.
> Do you have some hidden agenda to leave VSFilter
> broken?
Check my latest suggestion (to stay at hard coded BT.601) again. It makes it
extremely easy for VSFilter and xy-vsfilter to work perfectly fine. Actually my
suggestion makes it *much* easier for VSFilter and xy-vsfilter to render YCbCr
with correct colors. No more worrying about which matrix to use. You'd just
need to go back to how VSFilter always did things in the past and YCbCr
rendering would automatically be perfect. So I don't see how you could suggest
I had some hidden agenda or something.
> Do you believe there is an advantage of
> having the correct matrix flagged in the
> video bitstream?
Sure, for YCbCr video. But not for RGB video (e.g. FRAPS recordings).
> If yes, how is this any different than a
> desire to define the correct matrix to use
> for YCbCr output with our RGB subtitle format?
You've got to make up your mind whether ASS colors are to be interpreted as RGB
colors or as YCbCr colors (just converted to RGB for storage). If the ASS
colors are strictly RGB then there's zero reason to add color matrix
information because the RGB values themselves are the law. If the ASS colors
are to be interpreted as YCbCr colors then we can either add the necessary
matrix information, or we can always hard code to BT.601 (which would be much
simpler cause it's as it has always been treated by VSFilter, anyway). In
either case both VSFilter with its YCbCr rendering and the new interface with
its RGB rendering has to interprete the colors the same way.
If we decide to consider the ASS colors to be RGB then the new RGB interface is
the one who can render natively with no conversions. In this case the way the
RGB interface renders is the law and VSFilter has to adapt to achieve exactly
the same output. The only way to do it would be to use the inverse of the video
renderer matrix. That way RGB rendering and YCbCr rendering would be 100%
identical.
If we decide to consider the ASS colors to be YCbCr (just stored as RGB) then
VSFilter can render the colors natively with no conversions (oh well, one
conversion would be necessary because the colors are stored as RGB). In this
case what YCbCr rendering produces is the law and the RGB renderer has to do
additional work to match the YCbCr output.
Pick one. Either one is fine with me. You can't mix them. If you're concerned
about libass then the right way to handle things is probably to consider ASS
colors as strictly RGB (according to my understand that's how the libass devs
see it), and then VSFilter and xy-vsfilter need to be modified to do their best
to mimick untouched RGB rendering. If you're most concerned with VSFilter YCbCr
rendering producing correct colors then you can't get the libass devs into the
same boat at the same time. I suppose libass renders in RGB, too? If so, I
highly doubt they'd be willing to add a number of matrix corrections to make
libass rendering produce the same results as VSFilter with your "explicit
matrix" tagging.
Pick your poison. What should be view ASS subtitle colors as? RGB or YCbCr?
Which rendering method (RGB or YCbCr) should be considered the law and which
rendering method should have to do extra work to make sure it matches the other
rendering method?
> If we were designing a new subtitle format
> from scratch, explicitly defining the correct
> YCbCr matrix to use would be near the top of
> my list of important items, for the exact same
> reasons the H.264 spec allows the correct YCbCr
> matrix to be flagged in the bitstream.
First of all you'd have to decide whether the colors in your new subtitle
format are RGB or YCbCr. If they're YCbCr then your comparison to the h264 spec
is valid. But then you should store the values directly as YCbCr values in the
first place. Which would then also force libass devs to treat them as YCbCr
values. But if you create a new subtitle format and decide to use RGB, then no
matrix information needs to be stored, in the same way that e.g. PNG bitmap
files don't contain matrix information.
Original comment by mad...@gmail.com
on 4 May 2012 at 7:05
E.g. let's take one of the examples you mentioned earlier:
Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
Aegisub: Guess matrix BT.709 by resolution
VSFilter: Guess matrix BT.709 by resolution (if YCbCr, else untouched RGB)
madVR: user-selected correct matrix
If we use explicit matrix tagging then with VSFilter YCbCr rendering the
subtitle colors would match the video colors regardless of whether the user
selects the correct matrix in madVR or not. But with untouched RGB rendering
(or libass rendering, I suppose) video and subtitle colors would not match if
the user selects the correct matrix. This is bad. We need to make sure YCbCr
and RGB rendering always produce identical results. It makes no sense to have
them differ.
So, as I said in my previous post: We've got to finally make up our minds
whether we want ASS colors to be considered native RGB or YCbCr colors. Either
one is fine with me. But everything else depends on that decision. If we decide
to consider them YCbCr then IMHO the best solution is to stick to BT.601 to
keep things simple. Of course we could also use explicit matrix tagging in that
case, but it would make things more complicated, it would require the RGB
rendering path to do extra work and it would probably not be accepted by the
libass devs. If we decide to consider ASS colors native RGB then untouched RGB
rendering is the law and then it would be xy-vsfilter's job to make sure it
always matches untouched RGB rendering. The one and only matrix needed for that
would be the inverse of the video renderer's matrix.
Original comment by mad...@gmail.com
on 4 May 2012 at 7:32
> If we decide to consider ASS colors native RGB then untouched RGB rendering
is the law and then it would be xy-vsfilter's job to make sure it always
matches untouched RGB rendering. The one and only matrix needed for that would
be the inverse of the video renderer's matrix.
What for?
Original comment by sneaker...@googlemail.com
on 4 May 2012 at 7:46
*If* we decide that RGB is the law and YCbCr rendering should mimick it, then
VSFilter and xy-vsfilter need to use the inverse of the video renderer's matrix
to convert the ASS RGB colors to YCbCr, because that's the only way to make
sure that the YCbCr rendering method will produce the same results as untouched
RGB rendering in all situations.
As I said, I'm (almost) ok with anything, but if we do change the spec then at
least we should have a clear definition of what the ASS colors mean exactly in
the new spec and then every rendering method (xy-vsfilter YCbCr, xy-vsfilter
RGB, libass) has to make sure to follow the spec to the letter. It is not an
option to have different rendering methods produce different colors with the
new spec. It should be our absolute minimum requirement to clearly define what
the ASS colors mean exactly and then to have every rendering method produce the
same and correct output.
Hopefully everyone can agree with that?
Original comment by mad...@gmail.com
on 4 May 2012 at 8:01
Ah, I thought "untouched RGB rendering" meant using the new interface. You are
right of course for the current way of vsfilter sitting between decoder and
renderer.
Original comment by sneaker...@googlemail.com
on 4 May 2012 at 8:08
> So now you want to change the spec to accomodate developers
> who're too stubborn to follow the spec?
I always did, so long as maintaining legacy VSFilter compatibility was
possible.
It goes a long way towards fixing a problem with the spec being based on
VSFilter behavior. It also makes things easier on users who want to create
scripts which work correctly outside of VSFilter.
> Why is there TV vs PC here? The topic was brought up a while ago
> and as far as I remember, *everybody* (except maybe you) agreed
> that we should leave levels as hard coded PC.
Yes, we all agreed specifically that RGB output over the new interface should
always be PC levels as I remember it.
Having levels for PC range YCbCr output support is still useful.
> You simply CANNOT separate YCbCr from RGB rendering. If you try to do that,
> users will get different colors depending on whether they use VSFilter YCbCr
> rendering or the new interface. That cannot be what we aim for. If we do try
> to make things clear then we need a way to unify YCbCr and RGB rendering so
> both look exactly the same.
Now we're getting somewhere. This seems to be where we have a difference of
opinion.
You seem to be saying you rather both YCbCr output and RGB output match each
other but not the video, than YCbCr alone being correct because correct RGB
output is impractical.
I see why you'd want that, but that's absolutely crazy. If this is your main
concern, you are just going to need to suck it up and accept either a complex
explicit tagging solution so this never occurs, or just accept that RGB output
will sometimes be incorrect.
I refuse to cripple YCbCr output when it could otherwise be avoided, just
because getting correct RGB output is a major pita without extreme complexity.
That's why I say we should look into AYUV for the new interface once again, as
it eliminates this problem.
> And that means if you add loads of funny YCbCr specific options, the RGB
> rendering path has to take them all into account, too, or else RGB rendering
> will look different to YCbCr. That's a major pita. From the view point of
> RGB rendering it's a painful to look at multiple new ASS header fields
> targetted at YCbCr rendering and trying to interpret them correctly,
> so that RGB rendering produces identical results.
Could you please give specific examples how the following causes trouble for
RGB output?
yuvMatrix: TV|PC.Matrix
rgbCorrection: True|False
> In this case what YCbCr rendering produces is the law and the RGB renderer
> has to do additional work to match the YCbCr output.
This is exactly how it should be. In the majority of cases, this would result
in untouched RGB output.
> Check my latest suggestion (to stay at hard coded BT.601) again.
That equals leaving VSFilter broken. A requirement for us is allowing VSFilter
to use the correct matrix which matches the video, without breaking legacy
compatibility.
> Sure, for YCbCr video. But not for RGB video (e.g. FRAPS recordings).
Obviously. For native RGB video you would output RGB.
> You've got to make up your mind whether ASS colors are to be interpreted
> as RGB colors or as YCbCr colors (just converted to RGB for storage).
Nothing to make up my mind about. I've consistently always stated the following:
With YCbCr video:
"YCbCr colors (just converted to RGB for storage)"
With native RGB video:
"interpreted as RGB colors"
> If you're most concerned with VSFilter YCbCr rendering producing correct
> colors then you can't get the libass devs into the same boat at the same time.
That plan was to expect the Libass devs to do absolutely nothing. Explicit
tagging allow use of YCbCr matrix & untouched RGB in the majority of cases and
match Libass. In cases which we don't, we will be more correct than Libass and
have an advantage.
Correctness > Matching Libass > Not Matching Libass
Explicit tagging puts the power of matching Libass in the user's hands.
Original comment by cyber.sp...@gmail.com
on 4 May 2012 at 8:33
> matrix unknown to any program
Not a matrix unknown to any program.
A known matrix which cannot be automatically guessed with reliability by any
program.
Pretend you have an theoretical BT.800 matrix video, but Aegisub guesses BT.709
and incorrectly tags the script as:
sourceMatrix: BT.709
yuvMatrix: BT.709
If the user knew their video was BT.800, they would fix the script as follows
to be correct:
sourceMatrix: BT.800
yuvMatrix: BT.709
> if no corresponding trivial matrix name has been agreed on?
Why wait to define the trivial matrix names? A good start would be all the ones
in the the H.264 specification. New ones would be added whenever they come into
existence.
>> I think cyberbeing wants Aegisub to fall back to assuming BT.601 in the case
>> it stumbles upon an unknown matrix
> No that's not what I want. What you state only makes sense without explicit
tagging.
Actually, it may only make sense without explicit tagging for YCbCr output. I
believe my earlier example from Comment #20 would be broken with RGB output.
It would seem that Forcing BT.601 breaks double-corrected RGB output whenever
an unknown/unsupported matrix is encountered. So tgoyne's idea to workaround my
concern with YCbCr output, would actually be harmful to the new interface which
uses RGB.
Could someone confirm this?
> "Color Repair needed: yes" [tgoyne's idea]
>
> Video Bitstream: Empty (actual matrix != BT.601 or BT.709)
> Aegisub: Force BT.601 + "color repair needed: yes"
> VSFilter: BT.601 (if YCbCr, else double corrected RGB)
> madVR: Guess matrix BT.709 by resolution
> Result: Subtitles correct(?), decoded matrix wrong
> (this one confuses me a bit, can someone confirm if the
> double-corrected RGB subtitles are correct in this case?)
Original comment by cyber.sp...@gmail.com
on 4 May 2012 at 8:35
Original issue reported on code.google.com by
cyber.sp...@gmail.com
on 30 Apr 2012 at 10:22