ASS subs are not rendered correctly in gpu-next

Kapppa commented 3 months ago

mpv Information

mpv v0.38.0-525-g265056fa Copyright © 2000-2024 mpv/MPlayer/mplayer2 projects
 built on Jun 23 2024 00:07:06
libplacebo version: v7.349.0 (v7.349.0-rc1-3-g1fd3c7b-dirty)
FFmpeg version: N-115975-g0c0e7ec81
FFmpeg library versions:
   libavcodec      61.8.100
   libavdevice     61.2.100
   libavfilter     10.2.102
   libavformat     61.3.104
   libavutil       59.25.100
   libswresample   5.2.100
   libswscale      8.2.100

Other Information

Windows version: Microsoft Windows [Version 10.0.19045.4291]
GPU model, driver and version: RTX 3080, 551.86
Source of mpv: shinchiro latest windows
Introduced in version: Unknown

Reproduction Steps

Load provided video with gpu-next. --no-config --vo=gpu-next

Expected Behavior

Subs are displayed like in normal rending. expected

Actual Behavior

Subs are displayed incorrectly, looks like some frames are dropped or the layers are displayed in the wrong order. Instead of being white with a purple border they are just purple. actual

Log File

Expected Actual

Sample Files

issue.zip

I carefully read all instruction and confirm that I did the following:

[X] I tested with the latest mpv version to validate that the issue is not already fixed.
[X] I provided all required information including system and mpv version.
[X] I produced the log file with the exact same set of files, parameters, and conditions used in "Reproduction Steps", with the addition of --log-file=output.txt.
[X] I produced the log file while the behaviors described in "Actual Behavior" were actively observed.
[X] I attached the full, untruncated log file.
[X] I attached the backtrace in the case of a crash.

llyyr commented 3 months ago

It's not that the color is incorrect, the subtitles aren't rendered at all. Only the purple drop shadow is rendered. You can get it to render at arbitrary zoom and panscan though, which is odd

edit:

There's a mask(?) to hide this part of the original video, when enough of the mask is off screen, the subtitle is rendered correctly.

Kapppa commented 3 months ago

It's not that the color is incorrect, the subtitles aren't rendered at all. Only the purple drop shadow is rendered. You can get it to render at arbitrary zoom and panscan though, which is odd

edit:

There's a mask(?) to hide this part of the original video, when enough of the mask is off screen, the subtitle is rendered correctly.

Yeah I shouldn't said frames I meant layers. I assume the GPU rendering is dropping certain layers in the ass file.

na-na-hi commented 2 weeks ago

when enough of the mask is off screen, the subtitle is rendered correctly.

The problem is that even the mask itself isn't rendered correctly if the video is scaled. In the screenshots provided, the masked text can still be seen faintly, but it should not be visible at all with correct rendering.

Due to the way subtitles are rendered in mpv, only blend-subtitles=video (which is how VSFilter renders subtitles but isn't supported in gpu-next) can render this correctly. See the following screenshot (vo=gpu blend-subtitles=video, video scaled to 1600x900)

mask

kasper93 commented 1 week ago

Due to the way subtitles are rendered in mpv, only blend-subtitles=video (which is how VSFilter renders subtitles but isn't supported in gpu-next) can render this correctly.

It is supported in a way that it would be rendered relative to the video, but blended after scaling, which is in theory fine, if the scaling is done correctly. That said, it is likely fixable, even without blending at video size. Thoughts?

na-na-hi commented 1 week ago

It is supported in a way that it would be rendered relative to the video, but blended after scaling, which is in theory fine, if the scaling is done correctly. That said, it is likely fixable, even without blending at video size. Thoughts?

This is correct. But the crucial thing is to render all subtitle parts onto one buffer at the subtitle's resolution (which is usually the same as the video resolution for "good" files), and then scale the whole buffer to output size. See the commit message of https://github.com/mpv-player/mpv/commit/dec73f503f948d61e30f868eea4348e26a3ea804 for more details.

Unfortunately, this means that one extra copy is needed for subtitle contents. blend-subtitles=video, as noted above, isn't 100% correct because it fails when video and subtitle resolutions don't match, but it doesn't involve an extra copy.

kasper93 commented 1 week ago

isn't 100% correct because it fails when video and subtitle resolutions don't match

This is the case where it should render subtitles to proper rect. It doesn't mean we have to blend before, it means that we shouldn't use video rect directly.

na-na-hi commented 1 week ago

It doesn't mean we have to blend before

This is almost true, but due to the boundary condition issue mentioned, scaling the video and the "combined" subtitle plane separately and then blend them later gives different results from blending before scaling.

Additionally, doing it before blending has an advantage that the scale shaders only need to be applied once to the blended image. Currently, it seems that all image subtitles and overlays are always rendered with naive bilinear scaling regardless of the scaler used for video scaling, which is low quality, unless blend-subtitles=video is used. It's possible to achieve almost the same effect by also scaling subtitle plane with shaders, but it's more performance intensive and the boundary condition issue remains.

Not suggesting that blend-subtitles=video have to be implemented for gpu_next, just listing some of the trade offs it needs to make to avoid that.

mpv-player / mpv