search

basvandijk / scientific

Arbitrary-precision floating-point numbers represented using scientific notation
BSD 3-Clause "New" or "Revised" License
73 stars 40 forks source link

Flooring a big number returns 0? #39

Closed nh2 closed 7 years ago

nh2 commented 8 years ago 
> import Data.Scientific
> read "1e100" :: Scientific
1.0e100
> floor (read "1e100" :: Scientific) :: Int
0

What's going on?

The comment at https://hackage.haskell.org/package/scientific-0.3.4.4/docs/src/Data-Scientific.html#line-208 says

-- | @'floor' s@ returns the greatest integer not greater than @s@

That doesn't seem to be true.

nh2 commented 8 years ago

1e100 mod 2^64 is 0, so the result makes sense, but the comment doesn't.

basvandijk commented 8 years ago

What is happening internally is:

fromInteger (10^100 :: Integer) :: Int
0

Maybe I should make the documentation more specific like:

-- | @'floor' s@ returns 'fromInteger' applied to the greatest Integer not greater than @s@

What do you think?

basvandijk commented 7 years ago

Since this isn't user facing documentation it's not that important to be fully formal.