zachallaun / mneme

Snapshot testing for Elixir
https://hex.pm/packages/mneme
100 stars 5 forks source link

Make variable bindings accessible outside of `auto_assert` #1

Closed zachallaun closed 5 months ago

zachallaun commented 1 year ago

Currently, variables bound in the match pattern of an auto_assert are not accessible except in guards within that same assertion. For instance:

auto_assert pid when is_pid(pid) <- self()
pid # error, pid is not bound here

The work around is that the assertion returns the value, so if you really needed the result, you could do this:

pid = auto_assert pid when is_pid(pid) <- self()

Optimally, variables would be accessible in the block containing the assertion, as is the case with ExUnit's assert:

assert %{foo: foo, bar: bar} = some_call()
foo # foo and bar are available

How ExUnit does it

ExUnit statically extracts variables from the match expression and ensures that they are returned in the same order when the assertion is run. This is pseudo-code for demonstration purposes, but you can imagine the above assert example expanding to something like this:

value = some_call()
[foo, bar] = __assert_match__(value)
value

This means the assertion would return the value, which is desired, but would also bind the variables in the current scope.

Challenges for Mneme

The fundamental challenge is that Mneme doesn't know the vars that should be bound until runtime. Let's consider three cases:

# 1) new assertion
auto_assert self()

# 2) existing assertion
auto_assert pid when is_pid(pid) <- self()

# 3) incorrect assertion that will be updated
auto_assert pid when is_pid(pid) <- make_ref()

It turns out that cases 1) and 2) are possible to deal with. The second case can use the same technique that ExUnit uses, and the first can be handled by maintaining a sort of "private binding" that Mneme can draw from for future assertions. Since Mneme knows what new variables will be introduced by a new pattern, it can use those to e.g. pin that variable for future patterns in that test.

auto_assert pid when is_pid(pid) <- self()
auto_assert [^pid] <- [self()]

The third case is where things get really problematic, since a change in the value of an existing assertion can result in code that has a completely different set of variables. I'm not sure that there is an elegant solution to this.

I did some experimentation with this on this branch.

zachallaun commented 1 year ago

If this ends up being possible, we could also switch to using = instead of <- at that point. (One of the reasons I chose <- is to signal that the binding semantics are like with/for/etc.)

tcoopman commented 1 year ago

I would probably ignore the 3 case, as it can lead to any possible outcome.

zachallaun commented 1 year ago

One option for handling case 3 would be to always fail if an existing variable will no longer be present after updating, then prompt the user to re-run tests (or force-recompile and do it ourselves). If a variable was previously used that no longer exists, something will blow up.