Configurable length of the command sequence for rc::state::check

[mfherbst]

I would find it pretty useful if it was possible to configure the length of the command sequence rc::state::check generates and tests. Ideally I would like to have the possibility to have a different maximum sequence length for different tests / invocations of rc::state::check. (Perhaps there already exists a way and I did not manage to find it.)

Some background: I am currently writing tests for a piece of linear algebra software and I make use of rc::state::check in order to check a random sequence of certain matrix operations. If this sequence gets too long I loose too much numerical precision and the tests make no sense, so I'd like to check a large number of short command sequences instead.

[emil-e]

The length of the command sequence that rc::state::check uses is based on the size (currently a random number between 0 and size) so simply scaling the size would give you shorter sequence. To be able to adjust this, you will need to use the rc::state::gen::commands generator manually (that is used internally by rc::state::check) and then run the commands using rc::state::runAll.

To do the resizing, you can use one of the various combinators that change the size of a generator passed to it. This unfortunately has the disadvantage of also changing the size of the individual commands which is likely not what you want. You can probably get around this by resizing the generator returned by the GenerationFunc parameter back to the original size using some clever combination of resize and withSize.

But I could definitely see how being able to change the length of the sequence itself could be a useful thing but I'm not sure what the API should be. gen::container has a version that takes the length of the sequence as a parameter that generates a fixed length container but I'm not sure that this would be a good idea for this use case really. I think you do want the length to always be random but in cases like yours, controlled.

Please let me know your thoughts.

[mfherbst]

First of all: Cheers for your quick reply.

I managed to achieve the shortening of the sequences by a similar procedure to what you described above. I used rc::gen::scale to change the size to a fraction of the usual value and undid that effect with another rc::gen::scale enwrapped in a lambda which has the interface of state::gen::execOneOf in order to obtain the normal size in the inner commands again. I have not properly checked the resulting values and whether they follow the "usual" patterns, but on first look it seems to be ok.

Regarding the interface: Indeed, generally I would like to see random sequence lengths, just in cases like these I would like to have more control. Actually the length of the sequence may grow with the size, too, for the current cases I have in mind. But perhaps a fixed size might become useful as well, so I would suggest trying to cover all three cases:

• totally random length
• maximum length
• fixed size length

I would tackle this by having three command generators:

• state::gen::commands
• state::gen::maxCommands
• state::gen::numberOfCommands

These then internally deal with the proper scaling or resizing of the generator returned by the GenerationFunc or whatever else is necessary to get the appropriate sequence.

I guess no other changes need to be made. Whoever wants the current functionality runs rc::state::check and who wants more control needs to generate the command list using the aforementioned generators and then call runAll on it.

[emil-e]

The disadvantage of using scale to undo the downscaling is that you lose precision since the size is integral. But that probably does not matter a lot in this case.

I'll try to add something like what you describe here once I get the time for it.

[mfherbst]

Is it possible to cache the size somehow? If there is then I think caching that value and resetting it later on the state::gen::execOneOf is indeed better for the means of keeping the precision. I have noticed that I get a lot of zero entries in my matrices with the method I mentioned above, which makes a lot of sense now, that I know that size is integral ;).

By the way: Is there a clear relationship between the number of commands in the sequence and size? Is that something one can rely on or is it implementation-specific?

[emil-e]

The hacky way to cache the size would be to have a generator that simply generates the current size but I won't show you how to because it's not something I recommend.

The slightly less hacky way is to use withSize, perhaps like this:

MyModel initialState;
const auto commands = *gen::withSize([=](int size) {
  return gen::scale(0.5, gen::commands<MyCmd>(initialState, [=](const MyModel &state) {
    return gen::resize(size, state::gen::execOneOf<Cmd1, Cmd2, Cmd3>(state));
  }));
});

There is a clear relationship between size and the number of commands in the sequence right now, as I wrote before. The size defines the maximum number of commands that will be generated, i.e. (randomNumber % (size + 1)) + 1. But you shouldn't bet your life on that specific implementation being the same, although I don't have any plans to change it.

In general, you should probably keep in mind that there may very well be API breakages when upgrading to newer versions of RapidCheck since I don't want to freeze the API right now, I want more real-world usage to get some more feedback before I do.

[emil-e]

Another idea, if the are other commands that can still be applied once you've reached the limits of numeric precision, you can include the number of precision breaking command in the model and assert a limit as a precondition for that command.

[mfherbst]

Oh I just realised that I understood the gen::withSize generator wrong: I thought it would only allow me to use an inner generator with a ever-fixed size. I agree now that your proposed solution is better. I will use that instead.

For my current use case pretty much all operations could kill numeric precision in some way or another. When I have more time to revise my testing I might use more fine-grained generators for the matrix elements, where I can make sure that some operations are less problematic for precision. In that case your second comment is certainly an option as well. I'll keep it in mind, thanks.

[emil-e]

No, fixing the size is done with gen::resize.

It should perhaps be added that the floating point generator probably leaves some to be desired. There is really no sound theory behind it, it's mostly ad-hoc. If you have suggestions on how to generate arbitrary floating point numbers that scale with size, that would be welcome. Right now I'm considering simply doing reinterpret_cast to set the bits of the mantissa and exponent manually.

[mfherbst]

Hmm I still have problems there as well. I experimented with a few things there. My first thought was that in order to get sensible results after a few computations, I want the generated values to stay within a certain range. So at first I used a similar approach to generate an arbitrary mantissa and --- separately --- an exponent which is between -20 and 20, say. Plainly using gen::arbitrary<int> and then scaling the value into the correct range (using std::numeric_limits<int>::min() and std::numeric_limits<int>::max() did not work so nice, since it generated too extreme values too quickly. So right now I run the fraction *gen::arbitrary<int>/std::numeric_limits<int>::max() through a polynomial $x^5$ first before using it in order to damp the growth a little. I guess using a gen::scale generator instead of the polynomial could do the trick as well, but I haven't tried that yet.

Other than that I thought about changing the distribution of positive and negative numbers such that slightly more positive numbers are produced (to avoid the error when similar sized numbers are subtracted), but so far this was not yet necessary.

[emil-e]

Floating point numbers are tricky, it's not as obvious what the "right thing" to do is since it depends (I guess) on the calculations you're doing.