IEEE half precision uses incorrect range

aopp-pred / rpe

An emulator for reduced floating-point precision written in Fortran.

http://rpe.readthedocs.io/

Apache License 2.0

9 stars 8 forks source link

IEEE half precision uses incorrect range #15

Closed ajdawson closed 8 years ago

ajdawson commented 8 years ago

I believe the special case treatment for IEEE half precision uses an incorrect range value for detecting overflow. The correct value should be (2 - 2**(-10)) * 2**15 = 65504.0 but we are using 2**15 = 32768.0. This means we are being too conservative, so code that works in this mode should still work in real half precision. Ping @dueben.

dueben commented 8 years ago

I implemented this number range since I was not sure whether all hardware configurations would allow the use of the number range beyond 2**15. However, I guess this is not justified. -> You are right.