Aspect ratio affects the horizontal aliasing/blur

[loicvdb]

Describe the bug Using a very low width/height ratio results in a blurry image. And a very high ratio leads to aliasing.

I have a strong suspicion that PixelScale isn't set to the right value in the vertex shader of most cameras.

To Reproduce

1. Select an example, 2D or 3D
2. Give the preview a small width
3. A horizontal blur appears

Expected behavior I would expect it not to blur/create aliasing.

Screenshots Here's a 512x512 3D render, cropped to 512x64 And here's the exact same scene rendered at 512x64, a blur is visible. Here's the same thing but 2D:

Desktop (please complete the following information):

• OS: Win10
• Gfx card: nVidia GTX 980
• Driver version: nVidia 461.92

[loicvdb]

After a little bit of investigating, it seems like it's indeed an issue with the vertex shader. When setting aaScale for 2D cameras and PixelScale for 3D, the width of the screen is taken into account by pixeSize, but it probably already is by gl_ProjectionMatrix. Changing pixelSize to pixelSize.yy seems to fix it, but I'm unsure how to check if it works correctly for now.

Here's my quick fix for 2D: aaScale = vec2(gl_ProjectionMatrix[0][0], gl_ProjectionMatrix[1][1])*pixelSize.yy*AntiAliasScale/Zoom; And for 3D: PixelScale = vec2(gl_ProjectionMatrix[0][0], gl_ProjectionMatrix[1][1])*pixelSize.yy;

[loicvdb]

I've done more testing and finally understood how the vertex shader works. The fix of replacing by pixelSize.yy works in practice, but also in theory, the idea behind it is actually just that the view coordinates are scaled by the height only (coord.y goes from -1 to 1, coord.x's bounds vary to keep the pixels square). I applied more changes to my own camera to make the code less redundant, but they wouldn't be compatible with FragM's raytracers, so it would be best to keep just the pixelSize.yy fix for FragM's cameras.