\fax/y influenced by \fscx/y

[GoogleCodeExporter]

With libass, a line's scale affects its shear, causing very broken looking 
lines at high scales.

In the attached file, the two lines are rendered identically by vsfilter, but 
very differently by libass.

(Also, it looks like the scale is slightly off since libass renders one larger 
than the other, whereas vsfilter renders both of them as the same size).

Original issue reported on code.google.com by lulzbot3...@gmail.com on 27 Jan 2012 at 12:21

Attachments:

• shear.ass

[GoogleCodeExporter]

Here's a simple patch that should fix this.

It removes the code that INTENTIONALLY scaled the shearing according to the 
scale.

Original comment by nfxjfg@googlemail.com on 24 Feb 2012 at 8:23

Attachments:

• libass.patch

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Attached the wrong files in the first comment, sorry about that.
Thanks, the shearing is gone but \fay\fax is still broken.
Attached subs and screenshots.

Original comment by ri...@linuxmail.org on 26 Feb 2012 at 8:16

Attachments:

• subs.ass
• screen.jpg

[GoogleCodeExporter]

There was another line that used fay scaled. Looks like fax/fay should never be 
scaled.

Original comment by nfxjfg@googlemail.com on 26 Feb 2012 at 11:00

Attachments:

• patch2.patch

[GoogleCodeExporter]

Thanks for the patches, some lines are still being displayed wrong.

Here are some screenshots: libass on mouse over.
http://screenshotcomparison.com/comparison/110607/picture:0
http://screenshotcomparison.com/comparison/110607/picture:1
http://screenshotcomparison.com/comparison/110607/picture:2

Attached subs.

Original comment by ri...@linuxmail.org on 2 Mar 2012 at 9:02

Attachments:

• subs.ass

[GoogleCodeExporter]

The first issue ("squirrel test" intersecting with "penguin test") is easily 
solved. Here's what went wrong:
- VSFilter creates the font with the current font size, then applies fax/fay 
and rotations, then scales according to fscx/fscy
- libass scales to font size and fscx/fscy, then fax/fay and rotations
So you have to pass info->fax/info->scale_y and info->fay/info->scale_x instead 
of fax and fay to transform_3d to undo the scaling. Maybe the same when 
shearing is applied to cluster_advance.y.

No patch, because I have no idea what's wrong with the other samples, and I 
can't test with VSFilter.

Original comment by nfxjfg@googlemail.com on 3 Mar 2012 at 2:28

[GoogleCodeExporter]

I'm looking at this. Thanks for the nice test cases.

Original comment by g...@chown.ath.cx on 11 Mar 2012 at 4:38

[GoogleCodeExporter]

What's the status on this? It's one of the most annoying unfixed bugs.

By the way, I think some of the differences (or maybe all) are because freetype 
"snaps" the font size to the pixel grid (or something like this), even if 
hinting is disabled. So if the script requests a specific font size, that font 
size will be "pixel fitted" by freetype. Then libass will scale the outline 
according to the fscx and fscy tags. This is less than ideal.

In particular, this means libass won't scale the output evenly if the window is 
resized. You can test this with mplayer and the first test case posted here (be 
sure to use vdpau or gl video output). Make the window smaller, and the 
rendering output completely changes. This _sucks_.

It's not a real bug, just a consequence how freetype works and how libass uses 
it. Normally you'd want this behavior for optimal font rendering, but it 
doesn't make much sense here.

This could probably be solved by always requesting a fixed font size, and then 
scaling the outline according to script requested or screen-scaled font size. 
Alternatively, you could try to determine the final font size, and request the 
resulting size from freetype. This would require intrusive changes all over the 
rendering code.

I've attached my patch from comment 7 in case someone wants to use it. The test 
script from comment 6 doesn't render completely correctly. (It doesn't try to 
fix the issues I mentioned in this post.)

Original comment by nfxjfg@googlemail.com on 11 May 2012 at 11:49

Attachments:

• issue_46.patch

[GoogleCodeExporter]

I attempted to fix the issue I mentioned in my previous comment. Now the font 
is always scaled according to font_size * font_scale * scale_y. 
ass_font_set_transform() is only used to "stretch" the resulting glyph 
according to scale_x. Freetype can do its grid fitting, changing the window 
size will not exhibit the strange behavior I described, and subtitle rendering 
matches VSFilter more closely (especially for high scale values).

One particularly evil bug was that ensure_font_scale clipped the "final" font 
size against 1, so a script containing \fs1 and large fsc values would always 
end up incorrectly if the mplayer window was made smaller than the video 
(because font_scale is below 1.0 in this case). The patch moves this call and 
clips the ACTUAL final font size.

The patch for issue 42 (not originally by me) is included in this patch as well 
in order to correct the scaling for "fay".

All test cases seem to render fine with the patch applied. There are some size 
differences, but I guess you can't expect pixel-perfect compatibility with 
VSFilter, and you probably don't even want that.

Original comment by nfxjfg@googlemail.com on 24 May 2012 at 2:39

Attachments:

• issue_46_2.patch

[GoogleCodeExporter]

PS: I didn't test ASS drawings and text with BorderStyle=3. I don't know if 
these render correctly with issue_46_2.patch applied.

Original comment by nfxjfg@googlemail.com on 24 May 2012 at 2:54

[GoogleCodeExporter]

there is a new issue with issue_46_2.patch applied. text which is scaled and 
moved is 'pulsating' now. kind of hard to explain. it wasn't the case before 
with patch2.patch, though with patch2 the scaling and position was wrong.

i attached an example file showing the issue.

Original comment by Franz.We...@gmail.com on 28 May 2012 at 4:55

Attachments:

• issue.ass

[GoogleCodeExporter]

The subtitle script in comment 12 tries to animate the text by changing fscx 
and fscy on every subtitle event.

I suspect my patch works as intended, but freetype's grid fitting (and maybe 
hinting) ruins the animation by making it "steppy". I'm surprised it's so 
visible. (Maybe I'm actually overlooking something...)

There are basically three issues due to freetype grid fitting, depending what 
you do and what you pass to freetype:

1. Size of rendered text is slightly different compared to VSFilter. With 
default libass, this is particularly extreme: it normally does grid fitting at 
the given font size (\fs tag or the style field), and then scales according to 
\fsc tags. Low font size and high scale value means you get a big deviation 
compared to VSFilter.
2. "Steppy" or "pulsating" text scaling when changing the video player window.
3. Same as 2., but when animating \fsc or similar tags (at same window size).

issue_46_2.patch doesn't have problem 2, but it has problem 3.

I'm posting issue_46_2_badfs.patch, which has problem 2, but not problem 3. 
(This is a slightly fixed version of issue_46.patch. I made some mistake there, 
due to trying to fix this issue with try & error every other week.)

I'm also posting issue_46_4.patch, which shouldn't have any of these problems. 
It always passes font size 200 to freetype, and then scales down the resulting 
outline. It's a big hack, but it seems to work. Unfortunately, it completely 
kills hinting in every case. (Though I could have broken other things, I didn't 
test it very well.)

greg, what do you say?

By the way, whoever was posting the test case that included \fs2: it's a really 
bad idea to do this. Why not set the font size to something close to the final 
font size? In general, I'd say values passed to \fsc tags should be small.

Original comment by nfxjfg@googlemail.com on 29 May 2012 at 8:35

Attachments:

• issue_46_2_badfs.patch
• issue_46_4.patch
• subs_more.ass

[GoogleCodeExporter]

(Sorry, blogging some more...)

Summary:
a) issue_46_4.patch should work in all cases (minus broken hinting, and 
possibly some random regressions)
b) issue_46_2.patch should give best text rendering, but is broken if animated 
(pulsating text)
c) issue_46_2_badfs.patch is the minimal fix. But with \fsc, the font size is 
off, and the rendering is not truly independent from screen size (actually, 
it's extremely bad)

Maybe as a compromise, libass could use b/c by default, but switch to a) if 
\fsc, \fa, or similar tags are used. (Even if freetype's rounding of font sizes 
to pixel boundaries can be disabled, there's a fundamental conflict between 
nice text rendering and smooth font size scaling.)

Original comment by nfxjfg@googlemail.com on 4 Jun 2012 at 4:12

[GoogleCodeExporter]

>By the way, whoever was posting the test case that included \fs2: it's a 
really bad idea to do this.
>Why not set the font size to something close to the final font size? In 
general, I'd say values
>passed to \fsc tags should be small.

There were two reasons for doing this: the first being that it makes the 
problem significantly more obvious than having \fs40 and \fscx/y110 or similar. 
Given that the script was intended to be a demonstration of how the rendering 
was broken, I feel the example was suitable.

The second reason is that, believe it or not, people actually do this in actual 
subtitles. This is how this issue was first brought to my attention. It's 
unfortunate, but it's a "workaround" for some annoying behavior on vsfilter's 
part: it does not read noninteger scale values from \fscx/y override tags. 
Using a very small font size and a very large scale significantly mitigates the 
effects of "rounding" (or more accurately a lack thereof) errors, e.g. the 
difference between 100.01 and 100.99 is nearly 1%, but at 1000.01 and 1000.99, 
the error is an order of magnitude smaller.

And believe me, it does make a noticeably significant difference on playback. 
The typical cause for this sort of typesetting is motion tracked signs, and 
it's usually quite obvious if they aren't being rendered correctly.

Original comment by lulzbot3...@gmail.com on 9 Jun 2012 at 12:32

[GoogleCodeExporter]

My apologies, I seem to have misunderstood who that question was directed 
toward.

Feel free to entirely ignore the first paragraph, though I believe the rest of 
the information is a suitable response. The file in comment 12 seems to be 
exactly the same sort of thing I've seen myself, and I'm sure it was created 
the same way.

Original comment by lulzbot3...@gmail.com on 9 Jun 2012 at 12:47

[GoogleCodeExporter]

i didn't make the script myself. i got that typesetting from a group known for 
heavily doing sign tracking. i believe they did it because of the reasons 
mentioned in the comments above mine.

Original comment by Franz.We...@gmail.com on 10 Jun 2012 at 12:39

[GoogleCodeExporter]

Thank you, these are some good explanations. I wonder how xy-vsfilter handles 
this. Do they not have problems with hinting/rounding, or do they scale the 
font outlines instead of adjusting the font size when rendering subtitles in 
screen resolution?

Original comment by nfxjfg@googlemail.com on 11 Jun 2012 at 9:09

[GoogleCodeExporter]

I hope this is (mostly) fixed now, will test soon.

Original comment by g...@chown.ath.cx on 1 Oct 2012 at 2:41

• Changed state: Fixed