Access failed command sequence as data

[dmarjenburgh]

Hello, when a test fails, the errors and the seed are printed and the following data is returned:

{:test 1, :pass 0, :fail 1, :error 0, :type :summary}

Is there a way to get access to the commands and system responses as data? It would be really useful to have this for debugging the system when a test fails. Accessing the generated commands so you can execute them against the system manually is really useful. Currently the only way to do this is by copy-pasting them.

[czan]

Hmm. Where do you get the return value of {:test 1, :pass 0, :fail 1, :error 0, :type :summary} from? That isn't returned by stateful-check, as far as I can tell. That looks more like the result of your test runner?

There isn't currently a way to get the trace information as a data structure. Enough pieces exist to be able to write your own functions: you could use stateful-check.core/run-specification, then if the test fails (ie. throws an exception) you can use failure-exception-data to get the trace of the execution as tuples under :sequential (a seq of tuples) and :parallel (a seq of seqs of tuples). This isn't a public interface, so I don't want to guarantee its stability, but I haven't changed it in a long time.

I did at one point toy with the idea of trying to generate Clojure code from the trace of a failed test run to include as part of the printed information (or even instead of the #<1> = ... output). I didn't really pursue it because I wanted it to generate code that was independent of the command definitions, which didn't seem to be possible.

Can you give me an idea of what you'd expect the data from this to look like, and how you would expect to be able to use it? I'm certainly opening to adding it if there's a compelling use-case and a decent interface.

[dmarjenburgh]

The return value is the summary data from clojure.test, which you get when you run clojure.test/run-tests from the REPL. For example:

stateful-check.example=> (clojure.test/run-tests)

Testing stateful-check.example

FAIL in (example) (form-init6016606865532893737.clj:11)
Sequential prefix:
  #<1> = (:put "" 0)  = 0
  #<4> = (:get "a")  = 3

Seed: 1590414055502

expected: all executions to match specification
  actual: the above execution did not match the specification

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
=> {:test 1, :pass 0, :fail 1, :error 0, :type :summary}

I checked through the code a bit and saw all the data is already here: https://github.com/czan/stateful-check/blob/891faf20eeb2e776e933fb6674e04ff8ed5a27ac/src/stateful_check/core.clj#L208-L209

I also saw that you can get the entire quick-check result by calling run-specification. If I extract and transform that a bit, I should be able to get the commands and system responses out 🙂.

[czan]

Yeah, so {:test 1, :pass 0, :fail 1, :error 0, :type :summary} is what clojure.test/run-tests returns. That's the test runner, not stateful-check returning that value. I'm not aware of any way to inject more data into that map.

The trace data definitely exists, but it's quite inconvenient to access at the moment. If it were exposed bystateful-check, what would you like to be able to do with the trace data?

[dmarjenburgh]

The full quick-check result is already exposed by stateful-check by calling run-specification, so now I'm using that. Then I'm feeding it into this function to get the sequence of failed commands:

(defn- failed-command-sequence [qc-result]
  (let [smallest (ex-data (get-in qc-result [:shrunk :result]))]
    (mapv (fn [[[handle cmd & args] trace]]
            {:handle (pr-str handle)
             :command (:name cmd)
             :args (vec args)
             :trace trace}) (:sequential smallest))))

This is something I can inspect and if I want I can apply the failed commands one-by-one with another helper function:

(defn- apply-failed-command! [failed-command]
  (let [f (get-in system-specification [:commands (:command failed-command) :command])]
    (apply f (:args failed-command))))

This happens to work fine for me because of the way the commands are setup. So the issue isn't that relevant for me anymore, as I've found a workable solution. You can close it if you want.

[czan]

Can you show me a concrete example of how you're using this trace and the two functions that you've written, in practice? Could you maybe give me a run-through of a specific failure, and how these functions helped you to work through it?

[czan]

Without more information, this issue isn't really actionable. I'm going to close it, but if you can give me more information I'm still interested in exposing this data more helpfully.