search

qbism / q2tools-220

Quake 2 map compiler tools with v220 map support, automatic phong, enhancements, and fixes.
GNU General Public License v2.0
61 stars 20 forks source link

x87 / SSE floating point mismatch with surface extents #26

Closed Paril closed 1 year ago

Paril commented 1 year ago

This is a bit of a difficult issue to explain, but the short story is that the calculation that handles texture coordinates in vanilla Quake 2 and qbsp3 relied on x87 floating point calculation, which uses 80-bit precision intermediate values. Tools and engines compiled with SSE (either x86 or x64), however, will only use 32 or 64-bit floating point values, resulting in a mismatch for what the engines compiled with x87 instructions expect. The line in question is this one - https://github.com/qbism/q2tools-220/blob/master/4rad/lightmap.c#L135 (and the same line below it for QBSP) - creating a version of DotProduct that does the calculation via long double casts should be enough to fix it; it will then agree with vanilla engines on the surface extents.

I'm also working with engines to fix the disagreement as well. The same fix was applied across the board in the Q1 community for the bug in question, so I'm hoping we can match up with them. You can see a much more detailed description of the problem here: https://github.com/skullernet/q2pro/issues/261#issuecomment-1229587645

qbism commented 1 year ago

I don't see any downsides. Made the change in beta release https://github.com/qbism/q2tools-220/releases/tag/08-30-22

qbism commented 1 year ago

Visually appears improved. Please post any bug as an issue.