cdsmith / HMock

Mock framework for testing in Haskell
BSD 3-Clause "New" or "Revised" License
23 stars 4 forks source link

Consider supporting the log-and-verify approach #14

Open cdsmith opened 3 years ago

cdsmith commented 3 years ago

There are two different approaches to modern mocking. Libraries like gMock are plan-and-test, where you first declare an execution plan, then run the test and the execution plan is verified during and after the test run. Libraries like Mockito are log-and-verify, where the test runs first and then assertions are made about the logged call list.

Currently, HMock follows the plan-and-test approach. I think there are advantages to doing it this way, because the desired response to actions is usually tied to why those actions are expected in the first place. So the log-and-verify approach sometimes requires repeating yourself to declare a return value at the beginning and then verify the actual call at the end. There are two different languages for this (Mockito's when versus verify)

HMock can do both, though. It can keep a log of actions as the test runs, and also allow the user to verify that call log against another set of expectations after the test has completed. Verification can just be another ExpectContext instance, so that the same primitives are available for verification as for advance expectation.

For example:

runMockT $ do
  allowUnexpected $ ReadFile_ anything |-> "contents"
  allowUnexpected $ WriteFile anything anything

  copyFile "foo.txt" "bar.txt"

  verify $ inSequence [
    ReadFile "foo.txt",
    WriteFile "bar.txt" "contents"
    ]

The idea is that the user can choose to be very permissive when setting up responses, but then run a more specific verification (maybe even several of them!) after the fact. If the verify block has return values, it should fail at runtime. (We cannot compare, because we don't have Eq instances.)

cdsmith commented 3 years ago

The semantics here are unclear. A few options:

  1. Verify the full action sequence against the expect set passed to verify. In a complex test, though, it's unlikely that you want to verify the full action sequence each time.
  2. Verify that some subset of the action sequence matches. Aside from possible exponential blowup of matching, though, this leaves no way to say "this was never called", since verify will always match the subsequence that ignores that action.
  3. Mockito's verification is per-method, and non-matching methods are ignored. The equivalent here would be to ignore unmentioned methods. This is the "uninteresting" trick again, but applied only within the scope of verify. It has some of the same compositionality problems, but they are mitigated by the fact that you can call verify multiple times for independent expectations.
cdsmith commented 3 years ago

I'm deciding on option 3.