fix denormalized value encoding for fp16 and 3PK.

microsoft / DirectXMath

DirectXMath is an all inline SIMD C++ linear algebra library for use in games and graphics apps

https://walbourn.github.io/introducing-directxmath/

MIT License

1.52k stars 234 forks source link

fix denormalized value encoding for fp16 and 3PK. #104

Closed yosshin4004 closed 4 years ago

yosshin4004 commented 4 years ago

If the argument is close to zero, XMConvertFloatToHalf (non intrinsic version) will try to generate a denormalized value and the result will be invalid. This is due to a bit shift counter greater than 32.

XMStoreFloat3PK implementation has similar code. It also generates invalid denormalized value.

Please fix it.

walbourn commented 4 years ago

Fixed fp16 behavior in this commit

walbourn commented 4 years ago

Fixed the problem with float11/float10 conversions in this commit by adding a test for values that are too small for a denorm. This avoids all the cases where Shift was greater than 32.