chummersone / chummersone.github.io

MIT License
12 stars 1 forks source link

Value of 1 with Q1.63 #6

Closed hinxx closed 1 month ago

hinxx commented 1 month ago

If I understand correctly with Q1.63 signed conversion values are in range 0.99999999 .. -1 and trying to convert 1 should raise an Out of Range error (like it happens when doing Q1.31 signed conversion).

Screenshot_2024-09-20_10-01-28

Screenshot_2024-09-20_10-01-50

chummersone commented 1 month ago

The precision of 2^-63 is much smaller than the machine epsilon of JavaScript (which is double precision floating point, in which case the machine epsilon is something like 2^-53). So this is a very reasonable rounding error. Not much we can do about that in JavaScript.