Open killercup opened 6 years ago
Some more assorted notes of the top of my head:
When we generating temporary directories, there are a few aspects one might want to control:
It seems to me that providing a bunch of Strategy
combinators could satisfy this wealth of needed configuration. Generation of file contents also could use the standard regex based string generation or if a more rich grammar is needed, you could use CFGs (see: https://github.com/AltSysrq/proptest/issues/61).
Some CLI programs are interactive; for those kinds of programs, you not only have input/output, but the requested inputs and requested outputs can depend on previous input/output pairs. Sometimes multiple inputs are also requested based on previous inputs. This can probably be modeled with State Machines (see: https://github.com/AltSysrq/proptest/issues/28).
In my BSc thesis (see: http://publications.lib.chalmers.se/records/fulltext/251311/251311.pdf, repo: https://github.com/DATX02-17-26/DATX02-17-26), we discuss a Haskell DSL that is related to this (see the testing bit, the other parts are irrelevant for CLI apps).
For testing protocols, which is tangentially related to CLI apps, there's the MSc thesis "Bidirectional Testing of Communicating Systems" which provides the tool called SessionCheck
(see: http://publications.lib.chalmers.se/records/fulltext/254897/254897.pdf, repo: https://github.com/MaximilianAlgehed/SessionCheck)
cc @MaximilianAlgehed on the feasibility of reusing ideas from SessionCheck here.
Love the idea and would like to help with whatever we come up with but also want to bring up some concerns.
Also, in case people aren't aware, we have both assert_cmd and assert_fs that we can leverage.
Some challenges
cargo-generate
needs some structure (passing to stdin newline delimited strings where each would have its own strategy).btw for interactivity, we might want to explore how we can reuse or be inspired by rexpect
@Centril I'm no rust expert, but I don't think it matters here. I'm confident the ideas from SessionCheck carry over at least somewhat to CLI applications. The big problem with SessionCheck is that it is essentially message-oriented, meaning that the I/O of the CLI applications needs to be discretized somehow. For applications, for example REPL-style applications this is of course no problem what so ever.
But I'm more curious why you need the bidirectionality which SessionCheck offers, have you got two programs interacting using the CLI?
Best, Max
If y'all need examples, then xsv
has been using quickcheck for this purpose. To address performance concerns, tests are compiled with opt-level = 3
.
@Centril and I talked a bit about writing property tests for CLI apps. The general idea is to write input generation and assertions on CLI app invocations without much overhead (e.g. without having to deal with manually formatting types as CLI arguments).
The most trivial implementation that already works is probably this:
It generates random strings and asserts that echos gives us back the same string it was fed with.
We found that aside from the prop-testing specific challenges, there are also CLI-specific helpers needed, e.g. to easily generate temp dirs with (also generated) files, and that OS-specific or environment specific settings need to be able to be passed as arguments (or to be overwritten to reduce/eliminate dependence on global state).
In general, we think that we can use assert_cli to write these helpers. What we need to find is a way to express relations between arguments (inputs) and the asserts (like "stdout contains").
For the record but not meant as a concrete suggestions as something to implement today, I proposed a macro/DLS like:
that passes two arguments to a binary called
brighten
and asserts that, given valid arguments, the tool writes a string to stdout that can be parsed asColor
. Obviously missing here is a way to setup the environment. Side-effect of this syntax is that generating invalid (or omitting) arguments and asserting that the program fails is easy.