pschanely / CrossHair

An analysis tool for Python that blurs the line between testing and type systems.
Other
996 stars 47 forks source link

Can I use a seed to start exploration when using cover command? #255

Closed 9iang22 closed 6 months ago

9iang22 commented 6 months ago

I want to expand the test suite (generate more testcases for example), so I wonder is there any easy way to set a seed instead of using a specify value when crosshair cover start exploring the code? It seems crosshair will always start with some "empty" values such as "" or 0.

pschanely commented 6 months ago

Thanks for trying CrossHair and asking about this!

Unlike other test generation tools, CrossHair isn't using randomized input generation. Instead, it delegates the work to a constraint solver which produces an arbitrary (but often zero/empty) input that explores a code path. So there is no "seed" in that sense of the word. And, unfortunately, the internals of the solver are fairly opaque.

However, if you haven't noticed already, there are a few command line options that will make crosshair generate additional tests:

crosshair cover --coverage_type=path --max_uninteresting_iterations=20

(and/or try increasing the --max_uninteresting_iterations=value as needed to generate even more)

And then you can manually filter or tweak the results. However, there is no guarantee that the additional tests will be interesting or useful - e.g. they may likely just start trying integers going up from zero etc.

I have a hope that someday someone will build a more fully featured test generation tool on top of crosshair that would try to make tests that are more "realistic." (there is an the undocumented crosshair search command which could potentially be helpful for this) However, there is so much to do with the core of the system that it's unlikely I'd ever be able to work on such a tool. 😢

I'll mark this closed for now, but I'd still love to hear about how it goes!