kasper93
commented
8 months ago Seems to have been introduced in https://github.com/mpv-player/mpv/commit/704afb89681bc6d7fb87cce0a389444de3fac2a0, though I didn't confirm that myself yet and the issue is more nuanced than that
I think this is good tl;dr of this whole situation and why I didn't really have much motivation to go through the mud of it. It is late I will not go through everything you said, but just few remarks.
Before commit 704afb89681bc6d7fb87cce0a389444de3fac2a0 every subtitle were rendered in the target color space, not video one, your display one. I wasn't even interested in ASS in this commit, I was fixing PGS subtitle rendering and copied the sRGB logic from vo_gpu, which in lack of authoritative documentation was the closest thing that "should" work correctly. Because apparently it has been tested for years.
To be honest to not dwell to much into it, current status quo is that everything is bt.709, both video and your display. That's why it silently works for seemingly everyone.
The main issue with "just use video colorspace duh" is that it works now, because like said above everything is bt.709, but I assure you no one wants to create HDR subtitles. But since there is no HDR anime, the problem didn't arise yet. Frankly it is shortsighted from libass devs that the subtitle colorspace is still not defined. I don't want to make this thread about what should be and fixing the world. I will just patch mpv to whatever you guys want and leave others with the unsolved problem.
Note that "just use video colorspace" also makes subtitles tied to specific video, one cannot use SDR subtitles on HDR video (or even on WCG), because... they are not tagged. And I don't know how you roll in typesetting community, but having to redo all subtitles is not the best approach. Sure there will be color difference and if you need perfect accuracy make a new subs, but if slight offset is acceptable just use existing ones. (also good luck creating subtitles for DV P5 and alike)
There are currently two main issues:
- libplacebo uses different gamut mapping and tone mapping algorithms for overlays. Mostly to make OSC look better, because unlike video we want slightly different behavior, at least historically, because now it actually works fine with default mapping. Especially with ICC profile or manually adjusting target parameters, it can produce vastly different overlay look currently which includes subtitles. I have patch for that for months, but didn't find time to finish the last 5% of it. There are nuisances like peak detection in hdr content, gamut mapping... well nevermind, need to just upload MR.
- mpv hardcode sRGB, which well can be fixed instantly, but without the above fix in libplacebo it will not work correctly anyway. I guess it fixes most dumb case of bt.1886 mismatch.
Note that up until recently libplacebo didn't even apply ICC profile to subtitles, so everything looked also different.
Traneptora
ghost
haasn
arch1t3cht
witchymary
mightyhuhn
cubicibo
First of all, I tagged this issue as
gpu-next:since right now it mainly comes up ongpu-next, the core issue is a deeper one and also affects some configurations ofgpu(see below).Bug report formalities
Important Information
Provide following Information:
[AMD/ATI] Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] (rev c1)withxf86-video-amdgpuReproduction steps
The attached zip file contains a video with a rectangle for every possible (limited) 8-bit brightness value (plus a few black frames at the start so it's easy to see that the subtitle file is indeed loaded and visible), and a subtitle file that draws a smaller square into each of those rectangles. The subtitle file was generated in Aegisub in such a way that the square in each rectangle matches the color of that rectangle. When viewing the subtitles on that video file inside of Aegisub, the colors of the squares match the colors of the rectangles.
Steps to reproduce: Open the given video (on an SDR display) with the given subtitles in mpv with
--vo=gpu-nextor a different--transfer-trcand seek past the initial black frames.Expected behavior
Just like in Aegisub, the colors of the subtitle squares match the video's colors, so that the squares are not actually distinguishable on screen.
Actual behavior
Screenshot (done with mpv's screenshot command; screenshots of the actual window look similar) of
mpv --no-config brightness_test.mkv --sub-file=brightness_test.ass --screenshot-format=png --start=2 --pause --vo=gpu-nextOne can see that, in the dark areas (especially the top right corner), the squares are visible, i.e. their colors do not exactly match the video.
Compare this to the same command on
--vo=gpu:mpv --no-config brightness_test.mkv --sub-file=brightness_test.ass --screenshot-format=png --start=2 --pause --vo=gpuHere, the squares are almost completely invisible, except for some very subtle differences due to dithering.
However,
gpu-nextis not the only place where colors differ: For example, the following happens withmpv --no-config brightness_test.mkv --sub-file=brightness_test.ass --screenshot-format=png --start=2 --pause --vo=gpu --target-trc=bt.1886.More generally, any value of
target-trcother thanautocauses a mismatch.Note also that with
--vo=gpu-next, this behavior is independent of the value of--blend-subtitles, even though the documentation states that--blend-subtitles=yes"makes the subtitles be treated as being in the video's color space".Log file
output_gpu.txt output_gpu_next.txt
Sample files
brightness_test.zip
Discussion
This is a fairly complex topic so I'll detail everything that I believe to be relevant for this below. There were also some past discussions about similar questions, in particular this issue at libass.
But why is gpu-next wrong here?
mpv assumes that subtitles are sRGB both in
gpuand ingpu-next:https://github.com/mpv-player/mpv/blob/57d6826249dee3f9ea26937fb58ba2b8e2d3eae8/video/out/gpu/video.c#L2901-L2911
https://github.com/mpv-player/mpv/blob/57d6826249dee3f9ea26937fb58ba2b8e2d3eae8/video/out/vo_gpu_next.c#L316-L319
The only reason why the colors match in
gpuis thatgpudoesn't do conversions between SDR transfers whentarget-trcis not set:https://github.com/mpv-player/mpv/blob/57d6826249dee3f9ea26937fb58ba2b8e2d3eae8/video/out/gpu/video.c#L2621-L2627
(This is probably intended to some degree, but I'd argue that it is quite surprising behavior that setting
target-trcfromautotobt.1886on a normal SDR display affects subtitle colors but not video colors)wm4 also said that "Subtitles are sRGB" in a related issue at libass, and this can be assumed to be the case for subtitle formats like SRT. But for ASS subtitles it's more complicated.
To my knowledge, the color space of ASS subtitles is not specified anywhere (just like almost everything else about ASS subtitles). The only thing that's specified is the
YCbCr Matrixwrangling, but that only concerns R'G'B' -> Y'CbCr -> R'G'B' conversion (though it exists for a reason related to this issue).So, what color space should ASS subtitles have? Well, in the absence of an authoritative specification the reference for ASS rendering is VSFilter. Classic VSFilter drew its subtitles straight onto the Y'CbCr video stream. The Y'CbCr colors it used were computed from the given RGB triples via the BT.601 matrix (and the fact that that matrix was originally hardcoded led to the
YCbCr Matrixheader being created later). Equivalently, one can say that the subtitles were blended directly onto the video's R'G'B' colors (after converting the video from Y'CbCr to R'G'B' via its color matrix) without doing any gamma correction to either the subtitles or the video. Newer renderers, including authoring tools like Aegisub, then matched this behavior, and needless to say many existing subtitle files rely on this behavior (and note that wanting to exactly match the video's colors is a common use case for ASS subtitles, since subtitle authors want to create translations for on-screen text that blend in seamlessly).So, with this justification, one can say that ASS subtitles should not be sRGB, and instead be in whatever color space the video is, i.e. that no color correction should be applied to ASS subtitles. (I also brought up this question on the libass IRC before opening this issue, where this was confirmed by the maintainers.)
But it gets more complicated
Unfortunately, it's clear that while the philosophy of "ASS subtitles should be in whatever color space the video is in" works for ordinary SDR BT.601/BT.709 video (possibly also for BT.2020 though things may already get tricky there), this can clearly cause problems for HDR. While sometimes subtitle authors want to exactly match the video's colors, often they also just want to create some simple color - the most common case being white dialogue subtitles. Taking an average subtitle file with 255,255,255 white subtitles and applying them onto an HDR video in the video's color space would put that white at 10,000 nits, which is clearly not desirable by anybody. It's also not possible to say "Well, people that want to author subtitles for HDR content should account for that and use darker colors for their white" since libass is also used to render various other subtitle formats like SRT via ffmpeg auto-conversion - whether on the fly or not. Also, this philosphy would lock subtitle files to a single video (in a case where there exist SDR and HDR versions of the same video) even when the file does not actually contain any complex typesetting that needs to match video colors. Finally, it would also make subtitle style overrides harder, since different video color spaces would need different style overrides.
And what about primaries?
More or less the same issues can be raised about differing primaries, i.e. on how to play ASS subtitles on content with BT.2020 or DCI-P3 content, although they're less critical here. Still, a subtitle author that wants to color their subtitles in red may be surprised when that red suddenly shows up as much deeper color because the subtitles were suddenly assumed to have BT.2020 primaries by the player, but typesetters will still want to be able to match the video's BT.2020 colors (so in particular it needs to be possible to draw BT.2020 colors at all).
So, what should we do?
The above was a more or less objective description of all the factors involved in deciding how to interpret ASS colors. The following will be my suggestions on how to fix (or at least improve) the current logic, but naturally it's a bit more opinionated.
First of all, the issues with HDR notwithstanding, one can still safely say that for SDR content and primaries like BT.709 and BT.601, ASS subtitles should be taken to be in the video's color space. This would cover the vast majority of files that are broken with the current handling, and it's what I'd recommend to do as an immediate "fix" for this issue.
For HDR files or bigger gamuts, it's clear that there is no simple solution covering all use cases. For now this is not a big issue since (to my knowledge) next to no subtitle files exist that aim to do typesetting on HDR/WCG content, but this might change in the future. There, a solution is needed that allows for easily matching video colors without the problems mentioned above.
Spelling it out more explicitly, there are two cases that need to be covered:
In pretty much all cases I can think of from the perspective of a subtitle author, this distinction between typesetting and dialogue matches the distinctions made in two other subjects:
--vo=gpu --blend-subtitles=videoin the video's resolution. When discussing whether--blend-subtitles=videocould also be supported forgpu-nextit was discussed that instead libass should provide a reliable way to separate typesetting from dialogue, so that typesetting could be blended at video resolution while dialogue subtitles are blended at window resolution. See e.g. https://github.com/libass/libass/pull/570 for discussion about how to implement such a distinction in ASS files.The question of what color space to blend subtitles in aligns very closely with these two questions. Thus, a possible future solution to these three problems could be to implement such a typesetting/dialogue split and to then split subtitle rendering into two phases:
This would be a comparatively simple (as opposed to something like allowing arbitrary color spaces to be specified per style or per event) solution that still covers the vast majority of use cases. (Though it does make some things harder or impossible: It would no longer be possible to draw something above the dialogue subtitles in the video's colors (think of a timing meme where dialogue is supposed to disappear behind an object in the scene). This would already be harder with a rendering resolution split, but a color space split would make it impossible to work correctly in all cases. But this feels like a necessary trade-off.)
This solution would sidestep the issue of tonemapping and any dependencies on whether the viewer has an HDR or SDR display, and it'd require fairly little cooperation from subtitle authors or authoring tools: Authoring tools do not strictly need to apply any gamma correction and could just pretend that all video is BT.709 or sRGB (of course it would be nice if they properly supported HDR and bigger color spaces, but they wouldn't need to to output working files), and subtitle authors do not need to manually map colors as long as they correctly tag their subtitles as dialogue or typesetting. The only problem is that this first requires cooperation from the subtitle renderer, so this can only be implemented once support for a typesetting/dialogue distinction is made possible in libass.
So, for now this remains a hypothetical solution. Still, I wanted to lay out all my thoughts here because they may help with decisions on how the more short-term fixes should look. Again, my main recommendation for now is to blend subtitles in the video's color space when the video is BT.709/BT.601. (For HDR, assuming them to be sRGB is probably fine for now.)
Finally, a note about
--blend-subtitles: I'm aware that--blend-subtitles=yesalready blends subtitles in the video's color space ongpu(but it doesn't ongpu-nextand I believe that this is a bug). I guess with that in mind the issue is that--blend-subtitles=noshould not be the default for ASS subtitles on SDR video. Of course it does not make sense to change the default value in such a conditional way. (I guess an additional--blend=subtitles=autovalue could be added and made the default, but that has its own issues.) I don't really know of a simple way to make the effect on--blend-subtitleson this behavior make more sense, and I don't know enough how color spaces are/should be handled for non-ASS subtitles, but I wanted to bring up the fact that the behavior of this option is also involved here.