Make Arbitrary instance of Integer able to generate large numbers out of the box

[arybczak]

There is a slight problem with Arbitrary instance of Integer, namely that it won't generate large numbers (by which I mean numbers that don't fit into Int64) by default, which makes it impossible to test all the code paths out of the box as it is not uncommon for the code that deals with Integers distinguish between small and large numbers (which makes sense even given its constructors: https://hackage.haskell.org/package/integer-gmp-1.0.1.0/docs/src/GHC.Integer.Type.html#Integer).

Can current instance be replaced with something along the lines of

instance Arbitrary Integer where
    arbitrary = sized $ \sz -> do
        n <- choose (1, sz)
        sign <- arbitrary
        (if sign then negate else id) . foldr f 0
            <$> replicateM n arbitrary
      where
        f :: Word8 -> Integer -> Integer
        f w acc = (acc `shiftL` 8) + fromIntegral w

?

As of now defining a newtype LargeInteger = LargeInteger Integer with appropriate Arbitrary instance is somewhat of a workaround, but there are two problems with that: 1) You need to remember to do that. 2) Sometimes it's impossible/inconvenient to use it, e.g. when we're testing a data type with fields of type Integer (or better yet, where Integer is nested somewhere deep within tested data type).

[nick8325]

Well, the choice to generate smaller integers by default is deliberate. It's not appropriate for numeric code, but for code which uses integers as (e.g.) loop counters and indexes into lists, it's a good choice.

There is the Large modifier for generating large integers, but unfortunately it only works on bounded integral types. It would make a lot of sense to have a LargeUnbounded modifier which did the same thing for unbounded integral types. I'll have a think about how this can be generalised to more types than Integer (it would be nice if it worked on Natural too).

If a data type has an Integer field, you can use the newtype/modifier in the generator, like so:

instance Arbitrary Whatever where
   arbitrary = do
      LargeInteger n <- arbitrary
      -- now you can use n in your generator

[andrewthad]

Well, the choice to generate smaller integers by default is deliberate. It's not appropriate for numeric code, but for code which uses integers as (e.g.) loop counters and indexes into lists, it's a good choice.

Using Integer as a loop counter or an index into a data structure is not idiomatic Haskell. I've never seen a high-performance Haskell library that does this. By the time you reach an index over 2^63, the structure you are indexing into will have already consumed the entire heap. I would rather have an Arbitrary instance that produces values outside the range of machine integers. This would be more useful in things that I work on that use Integer.

[nomeata]

A modifier that produces truely large Integer and Natural values would be good; I am testing an encoding library right now. I ended up using this code, inspired by @arybczak above:

newtype LargeInteger = LargeInteger Integer deriving (Show, Eq, Ord, Num)
newtype LargeNatural = LargeNatural Natural deriving (Show, Eq, Ord, Num)

instance Arbitrary LargeNatural where
    arbitrary = LargeNatural . fromWords <$> arbitrary
      where
        fromWords :: [Word8] -> Natural
        fromWords = foldl' go 0
        go :: Natural -> Word8 -> Natural
        go acc w = (acc `shiftL` 8) + fromIntegral w

instance Arbitrary LargeInteger where
    arbitrary = do
       sign <- arbitrary
       LargeNatural n <- arbitrary
       return $ LargeInteger $ (if sign then negate else id) $ fromIntegral n