XMVectorFloatToHalf doesn't properly handle denormals

microsoft / DirectXMath

DirectXMath is an all inline SIMD C++ linear algebra library for use in games and graphics apps

https://walbourn.github.io/introducing-directxmath/

MIT License

1.54k stars 238 forks source link

XMVectorFloatToHalf doesn't properly handle denormals #110

Closed walbourn closed 4 years ago

walbourn commented 4 years ago

The implementation of XMVectorFloatToHalf for C is not producing the correct denormals compared to the CVT16/F16C version. Ideally these should match.

walbourn commented 4 years ago

Checking all 32-bit values, the current implementation scores:

Mismatch (NaN F32): 16744446
Mismatch (e16): 0
Mismatch (e-14): 8190
Mismatch (denorm): 12288
Mismatch (flush): 603967488
Percentage (NaN F32): 0.195318
Percentage (e16): 100.000000
Percentage (e-14): 99.998375
Percentage (denorm): 99.993340
Percentage (flush): 64.706604
Percentage: 85.547447

walbourn commented 4 years ago

Now 100% match for CVT/F16C. See this PR