Closed sgrif closed 3 years ago
@sgrif I think the original idea here is that the size
method on quickcheck::Gen
would control the magnitude of the integers generated, but perhaps that is a misuse. With #116 merged, I think there is now good precedent for biasing towards known problematic values.
It is somewhat reversed #99: numbers being biased too much towards -100...100
.
I'm going to prepare a reformed number generation analogous to char (but simpler) which should cover entire range, yet preferring some particular numbers.
There needs to be a new api for requesting a quickcheck-controlled size value. Something that's appropriate for selecting the size of a datastructure to generate. I think I mostly use usize's Arbitrary impl for that at the moment.
Experienced a problem with some tests: https://github.com/stratis-storage/stratisd/pull/217/commits/0f7838d536619861899cb8d49106b750d59c787d, all u64 values were less than minimum required, so all testt were discarded.
With current behaviour that is expected @mulkieran and you simply need a wrapper type that defines the distribution of values that you need.
@bluss Can you give me an example?
if anyone who understand this problem can give me some help, I'ld appreciate it.
Something like this to make a wrapper type that puts a value inside using the regular Rand
impl (not Arbitrary
)
extern crate quickcheck;
use quickcheck::Arbitrary;
use quickcheck::Gen;
extern crate rand;
use rand::{Rand, Rng};
#[derive(Copy, Clone, Debug)]
pub struct Random<T>(pub T);
impl<T> Arbitrary for Random<T>
where T: Rand + Clone + Send + 'static
{
fn arbitrary<G: Gen>(g: &mut G) -> Self {
Random(g.gen())
}
}
Maybe it's time to make a change like #99, but for numbers?
That could make "sane default" for multiple varied uses. Obviously one could override the distribution to task-specific.
I would always ask myself what Hypothesis (https://github.com/HypothesisWorks/hypothesis-python) does. But it might not be such a useful question for this problem in this situation.
The Haskell implementation seems to first decide maximum bits the generated number has. Then generate a number within the range (-2 ^ max_bits, 2 ^ max_bits)
. Doing something similar is possible but doing it without regressing performance would be tricky.
Current workaround would be setting environment variable QUICKCHECK_GENERATOR_SIZE=18446744073709551615
on 64-bit platform. Not sure how feasible it is to change the default gen.size()
from 100 to usize::MAX
. It seems it does not hurt performance at least.
@BurntSushi I'd like to hear your thoughts about following propositions addressing this issue.
1) Ensure that every valid value* for each integer type has some positive probability of being returned by Arbitrary
2) Completely ignore the size
of Gen
(for integer types)
3) Have a greater chance to pick problematic values
*valid values for an integer type T
means every value in
T::min_value()..=T::max_value()
@maxbla Thanks for writing that out! I think that all seems pretty reasonable to me.
@maxbla Have you seen https://github.com/BurntSushi/quickcheck/pull/221 ? I closed it after ~6 months because there was apparently no interest in merging it, but maybe there is still something useful in there for you, if you want to revive the topic with a new PR, since it implements something very similar to what you propose (the size
parameter is not ignored but acts as a weighted bias towards a preferred range).
I closed it after ~6 months because there was apparently no interest in merging it
There is interest. But I maintain dozens of projects, so it can take me a very long time to merge PRs and/or it's very easy for me to lose track of PRs. Editing comments to include status updates probably doesn't help, as was done in #221. I don't get emails when edits are made.
There is interest. But I maintain dozens of projects, so it can take me a very long time to merge PRs and/or it's very easy for me to lose track of PRs. Editing comments to include status updates probably doesn't help, as was done in #221. I don't get emails when edits are made.
No worries and I did not mean to criticize, of course. It's just a matter of my personal PR hygiene that I close unmerged PRs on my own after a while when I've also moved on to other things and no longer intend to keep them updated / mergeable.
@romanb yep, I saw #221, and thanks, it was helpful in creating #240.
I have a type which wraps an
i32
, and wrote anArbitrary
impl like so:When I actually output what it's running with, the values are only ranging from
100
to-100
, instead of 100 completely random numbers as I'd expect. Additionally,shrink
appears to be flawed on these types. My understanding was that quickcheck would test for known problematic values for a given type. Fori32
I'd expect that to at minimum be1
,0
,-1
,i32::MAX
andi32::MIN
, but when I add theshrink
like so:I just see the same range of -100 to 100. This caused a critical bug that occurs for any values less than
-2451545
to go unnoticed.