search

flyingmutant / rapid

Rapid is a modern Go property-based testing library
https://pkg.go.dev/pgregory.net/rapid
Mozilla Public License 2.0
579 stars 25 forks source link

Explicit control of settings per test #56

Closed prateek closed 1 year ago

prateek commented 1 year ago

I stumbled upon your library completely by accident earlier this evening, and wanted to first off say - it's quite great. I've been using gopter since before generics were in Go. I ported a few simple tests to see how the UX feels during writing tests - gopter v rapid. (I love it!)

Wanted to callout a small gap in the library as it stands: the lack of explicit control of number of tests run/seed/etc. I've relied on gopter.TestParameters to override that stuff via code and find it quite useful. I think the -rapid.args is incredibly useful for testing/development but I'd love the ability to override them via code to capture the required knobs for CI and so on.

How do you feel about exposing cmdline in some shape to library users?

flyingmutant commented 1 year ago

Glad you are liking rapid!

FYI rapid automatically lowers the number of checks/steps when standard go test -short flag is specified, and it also tries to not run over the standard global -timeout flag.

Can you please tell what are the knobs you want to override, and in what scenarios?

prateek commented 1 year ago

Hey @flyingmutant! I don't need the knobs for my personal dev necessarily. For my day job - I write code in a few different monorepos (https://github.com/m3db/m3 is the only public one, there's another couple internal to my company - one using standard go tooling, and another using Bazel).

Across these monorepos, I have a bunch of different tests using gopter. Some of the properties being checked are extremely quick, so I want those tests to run for X min runs; and some of the properties are quite slow, so I set a max runs of like 10 or something. The CI environments for these repos vary greatly (some are using cheap AWS instances, some internal VMs/Containers/etc) as does the tooling (e.g. there's a go<>bazel plugin which doesn't respect all the go test -<args>).

All that to say, my primary usecase for the knobs is to encode explicitly the number of iterations each test checks.

flyingmutant commented 1 year ago

Thanks for the detailed information! I'll close this issue in favor of #38. Customizing number of checks to run is a common request and I hope to implement this in 1.1 release.