dankamongmen / panelreels

high-level objects built atop ncurses
https://nick-black.com/dankwiki/index.php/Outcurses
Apache License 2.0
8 stars 1 forks source link

genprefix() incorrect for some very large values #22

Closed dankamongmen closed 4 years ago

dankamongmen commented 4 years ago

While adding examples to the README, I noted that

genprefix(UINTMAX_MAX, 1, out, 1, 1024, 'i'); ---> "15.99E"

but it is not true :/. This call actually returns 15.10. Similarly, we seem to think that 7.10 * 250 == 263-1 (INTMAX_MAX).

dankamongmen commented 4 years ago

Weirdly, if we pass omitdec==1, it calculates INTMAX_MAX to be 8Ei properly. With omitdec==0, we get 7.100E.

For that matter, since when do we print three sigfigs of mantissa?!

dankamongmen commented 4 years ago

ohhhhhhhhhhhhhhhhh that represents a "mantissa" of 1.00, doesn't it? and thus 7.00 + 1.00 == 8.00. yeah that's what's going on.