search

zachallaun / mneme

Snapshot testing for Elixir
https://hex.pm/packages/mneme
100 stars 5 forks source link

Proposal: Introduce `mix mneme` to run tests with Mneme prompts instead of "hijacking" `mix test` #21

Closed zachallaun closed 1 year ago

zachallaun commented 1 year ago

I've been thinking that it may be worth introducing a mix mneme task instead of modifying the behavior of mix test, which is what currently happens. I'll lay out the proposed changes and then some pros/cons as I see them.

Proposed changes

  1. Mneme would now run in two modes - interactive and non-interactive. Mneme will be interactive (prompt the user with changes) if Mneme.start/1 has been called. In non-interactive mode, new/changed auto-assertions will simply fail the test.
  2. A new task mix mneme would become the recommended way to use the library. It would essentially call Mneme.start/1 and then delegate all arguments to the mix test task.
  3. It would no longer be recommended to call Mneme.start() in test/test_helper.exs. This means that Mneme would be in non-interactive mode when running mix test.
    • Note: you still could call Mneme.start() in your test helper, in which case mix test would work as it does with Mneme today, prompting for new/changed assertions. This makes this proposal backwards-compatible with existing workflows.

Advantages

  1. One less thing to set up -- no changes to test/test_helper.exs required.
  2. Less "buy-in" necessary to try Mneme since it would no longer change the way your tests run when you run mix test. This may give people more confidence to try it out.
  3. mix test will be slightly faster when Mneme is run in non-interactive mode. Currently auto-assertions may have to wait for a small amount of time for messages from ExUnit containing information about the active test to come in.
  4. Less concern for CI environments. Mneme currently auto-rejects all new/changed auto-assertions if the env var CI=true is set, and it would continue to do that in interactive mode, but this would side-step the issue altogether as CI runners would be running mix test and not mix mneme.

Disadvantages

  1. Certain editor integrations may require a bit more setup if you wish for them to run tests in interactive mode -- you'll need to update the test command to use mix mneme.
  2. One more thing to remember -- it doesn't just "work" when you run the thing you're trained to run, you have to run another command instead. I expect that I will accidentally run mix test quite a few times, see an error, sigh, then run mix mneme which is what I meant to do in the first place.

Any thoughts/reactions to this, positive or negative, would be most welcome!

mindreframer commented 1 year ago

I'm fine with the small setup-fee of 1 line config in test_helper. So honestly, I would try to keep dev UX as simple as possible, and current flow seems good enough for me. Running another mix task to run my tests might mess with muscle memory for many devs, I do not want to think whether I should run mix mneme or mix test. Also, alternatives require maintenance and more code and more documentation. Maybe keep it simple?

Just my 2 cents.

zachallaun commented 1 year ago

Thanks for sharing your thoughts @mindreframer! Thinking about this more over the last week, I think I'm going to incorporate some of the proposed changes, but not the default way to run tests with Mneme: