search

leanovate / gopter

GOlang Property TestER
MIT License
598 stars 40 forks source link

properties.Run doesn't stop #63

Closed anisjonischkeit closed 4 years ago

anisjonischkeit commented 4 years ago

I'm running some tests but for some reason gopter.Run doesn't stop. The tests make calls against a db that isn't very fast and after around 2000 tests I just kill the gopter.Run process because it takes too long. Is there any way to limit the amount of tests that get run before gopter says that the tests have passed?

Also I don't know why but other tests don't need nearly as many attempts.

anisjonischkeit commented 4 years ago

Okay, looks like lowering the shrink count has fixed it. it was doing 1000+ shrinks which was taking a very long time

untoldwind commented 4 years ago

Do you think it would make sense to lower the default settings?

anisjonischkeit commented 4 years ago

I haven't looked into how shrinkers work at all but If they do a binary search kind of pattern I can't imagine many situations that would actually use 1000 shrinks....

anisjonischkeit commented 4 years ago

Also it seemed like it was running the test function more than 1000 times with the DefaultTestParameters so I'm not sure if there's a bug there somewhere (or maybe I just didn't wait for the 1100th try.

anisjonischkeit commented 4 years ago

I guess nested structures might want that amount of shrinkage.

untoldwind commented 4 years ago

I fear that MaxShringCount is indeed per level and not overall. I suppose that could need some improvement.

anisjonischkeit commented 4 years ago

That makes sense, was going to ask about it but thought surely that's not the case. Well if it's per level (I assume you mean it's actually per generator so if there are 5 string generators in a struct I assume it does 5* the maxShrinks), and shrinkers are implemented in a binary search kind of way, I think reducing maxshrinks is probably not going to do any harm. Remember you only need 21 comparisons in a binary search to find any item in a list of 1 million items so with 100-200 shrinks per generator you are actually covering a very improbable area.