RFC: DSL to ExUnit

[mgwidmann]

The primary purpose of this tool is to provide a way to execute Gherkin files as code. This needs to be done so transparently and as simply as possible. The best way to do this is to leverage all the capabilities provided by ExUnit.

The major issues with this project:

As discussed here: https://github.com/meadsteve/white-bread/issues/67 and issues like https://github.com/meadsteve/white-bread/issues/74 and https://github.com/meadsteve/white-bread/issues/80 will be resolved by this proposal

• No clear understanding of where code begins executing
  • Context files are required first and then the config.exs, which is required because they are mentioned in the config.exs file
  • Runs in dev by default (this can be fixed though by setting @preferred_cli_env)
• Setup of additional test only dependencies and/or startup/shutdown procedures must be done inside a callback like scenario_starting_state callback, which is awkward since this callback is intended to return the beginning state of a scenario
  • Starting of dependencies like hound or ex_machina or any other test dependency occurs here since they're not started by mix. Similar to the first point
  • These must be done on a per context level in an idempotent fashion (using Application.ensure_all_started(:hound) typically)
• Custom DSL to perform similar actions in ExUnit
  • setup_all == scenario_starting_state, setup == feature_starting_state, on_exit == scenario_finalize

To resolve this, users need a clear and transparent understanding of how feature files become tests. Elixir's macro system is powerful enough to provide this translation at compile time.

Example

The feature file from the README, unchanged. Notice its moved under the test/features directory to show closer integration with ExUnit. test/features/coffee.feature

Feature: Serve coffee
  Coffee should not be served until paid for
  Coffee should not be served until the button has been pressed
  If there is no coffee left then money should be refunded

  Scenario: Buy last coffee
    Given there are 1 coffees left in the machine
    And I have deposited £1
    When I press the coffee button
    Then I should be served a coffee

The feature file is ignored by ExUnit while the test below is executed. No need to provide suite setup callbacks as everything is provided by ExUnit. test/features/coffee_test.exs or anywhere you want really

defmodule MyApp.Features.CoffeeTest do
  # base directory should be configurable, assumes "test/features/" is prepended
  # remaining options are passed directly to `ExUnit`
  use WhiteBread.Feature, async: false, file: "coffee.feature"

  # Provided by `use WhiteBread.Feature` to "copy" in reuseable steps
  # Can be discussed if this should be recursive to work more than one level deep
  import_steps_from MyApp.Features.Global

  # Instead of providing a callback, use exunit to handle this
  # `setup_all/1` provides a callback for doing something before the entire suite runs
  setup do
    on_exit fn ->
      IO.puts "Scenario completed, cleanup stuff"
    end
    {:ok, %{my_starting: :state, user: %User{}}}
  end

  # This works just like `ExUnit`'s `test/3` macro, so it should provide a similar feel
  # All `defgiven/4`, `defand/4`, `defwhen/4` and `defthen/4` takes a regex, state, matched data, and lastly a block
  defgiven ~r/^there (is|are) (?<number>\d+) coffee(s) left in the machine$/, %{user: user}, %{number: number} do
    # `{:ok, state}` gets returned from each callback and a compiler error is thrown otherwise
    {:ok, %{user: user, machine: Machine.put_coffee(Machine.new, number)}}
  end

  defand ~r/^And I have deposited £(?<number>\d+)$/, %{user: user, machine: machine}} = state, %{number: number} do
    {:ok, Map.put(state, :machine, Machine.deposit(machine, user, number))}
  end

  # With no matches, the map is empty. Since state is unchanged, its not necessary to return it
  defwhen ~r/^I press the coffee button$/, state, %{} do
    Machine.press_coffee(state.machine) # likely instead would be some `hound` dsl
  end

  # Since state is unchanged, its not necessary to return it
  defthen ~r/^I should be served a coffee$/, state, _ do
    assert %Coffee{} = Machine.take_drink(state.machine) # Obviously some fake code, but the point is clear
  end
end

This can all work by using macros, to compile to the following:

defmodule MyApp.Features.CoffeeTest do
  use ExUnit.Case, async: false

  setup do
    on_exit fn ->
      IO.puts "Scenario completed, cleanup stuff"
    end
    {:ok, %{my_starting: :state, user: %User{}}}
  end

  # Each scenario would generate a single test case
  @tag :white_bread
  test "Buy last coffee", %{my_starting: :state, user: user} do
    # From the given
    state = %{user: user, machine: Machine.put_coffee(Machine.new, number)}
    # From the and
    state = Map.put(state, :machine, Machine.deposit(machine, user, number))
    # From the when
    Machine.press_coffee(state.machine)
    # From the then
    assert %Coffee{} = Machine.take_drink(state.machine)
  end
end

This is now run via mix test. If a user wants to run them all the time, they can, or they can either via the command line do mix test --exclude white_bread or in their test/test_helper.exs put ExUnit.configure exclude: [:integration] to exclude it by default. The end user has control, and all the heavy lifting is done by ExUnit.

This project then only handles translating feature files to tests. It doesn't have callbacks, or handle setting up the test environment (with or without starting the application). Theres a clear starting point (defined by ExUnit in the test/test_helper.exs). Output is handled by ExUnit based upon the assertions the user writes (though it may be a nice to have to add an ExUnit formatter).

In the end this DSL will result in fewer bugs and a clearer understanding & full transparency of what is running at test time. Any additional features surrounding the lifecycle and/or setup and teardown can and should be provided by ExUnit, which means this project won't have to develop them, it'll just inherit these future additions and features. Why try to keep up and build your own testing framework when we can just leverage ExUnit?

If you're too busy, I'd like to see if what I can come up with when I get a moment.

[meadsteve]

@mgwidmann I definitely don't have time at the moment to do this but I do really like it as an idea.

It feels like it'd be easier to run this up from scratch as a new project (obviously stealing any ideas from whitebread as needed). If it works well I'd happily redirect people to this new tool and write some tools to help with migration from whitebread. What do you think?

[mgwidmann]

Let me see if I can come up with something...

[mgwidmann]

So I have the above example working in a separate project... I've extracted out :gherkin as a separate project since I used that to help parse the feature files... I'm going to try to publish it by extracting your commits from this repository...

But anyway, heres the new project thus far: https://github.com/mgwidmann/cabbage

[mgwidmann]

I've extracted gherkin to a separate project. Since you and the contributors to this project wrote it, I can transfer the repository to you if you'd like and post it to hex under your name.

https://github.com/mgwidmann/gherkin

[meadsteve]

Looks really good so far. Love how much less code it needs. Regarding gherkin the other alternative is to create a cabbagex org and put both projects there. I'm happy with either really. The important thing for me is that we credit the people who made pull requests.

[meadsteve]

Looks really good so far. Love how much less code it needs. Regarding gherkin the other alternative is to create a cabbagex org and put both projects there. I'm happy with either really. The important thing for me is that we credit the people who made pull requests.

[mgwidmann]

Yeah so I managed (not without difficulty though) to keep the history behind all the commits in Gherkin, so everyone who contributed to the gherkin parsing is still listed as a contributor. I can make an organization to keep it all together.

[mgwidmann]

Started an organization and published the initial version to hex. https://hex.pm/packages/gherkin https://github.com/cabbage-ex/gherkin https://hex.pm/packages/cabbage https://github.com/cabbage-ex/cabbage

🎉 🎈 :clinking_glasses: 🎊

[meadsteve]

Amazing work!

[meadsteve]

I'll do a point release of white bread to use the gherkin lib so we don't have any duplication. And add a mention of cabbage to the readme here.

[meadsteve]

I've linked to cabbage from this README and whitebread is now using the cabbage gherkin lib. There's still an outstanding question of how/who should and what migration should happen from whitebread to cabbage . I'm looking for new maintainers to help out with this project (#88) so that would be something to discuss with the group that picks this up.

[mgwidmann]

Awesome, I still have a few features to support, namely scenario outlines aren't supported yet. I'll try to put together a small road map.

[diabolo]

First of all I'm an Elixir newly, but I am very experienced with Cucumber.

My gut tells me that this approach of converting scenarios directly into ex tests via macros has fundamental problems.

For a start its a completely different approach to the other Cucumber implementations as I understand them. These leverage the platform code to create a test environment and then work within that to execute code against the environment. My extremely limited understanding of Elixir would suggest that an Elixir cucumber should use the environment that is created for an Ex test, but that is very different from actually creating a test itself for each scenario.

The fundamental purpose of Cucumber is to create a pipeline from a natural language statement to a call that runs some code. The problems I see with this macro approach is that now you have an additional step in this pipeline. With Ruby Cucumber this pipeline is

scenario -> step_definition -> code execution`

If you do this with some elegance you get

scenario -> step_definition -> call to helper module -> code execution

so you get out of the Cucumber space as fast as possible.

What this macro approach is doing is

scenario -> step_definition -> macro conversion to ex unit test -> code execution

with the elegant version still having an extra step

scenario -> step_definition -> macro conversion to ex unit test -> call to helper module -> code execution

These extra steps and the physical entities they produce (files) have to be navigated to for every individual scenario, and all this is just so we can execute our calls in an environment that is test friendly.

This macro approach seems very focused on the scenario being the key structure in Cucumber. But the real key to Cucumber is the translation to a call, and the way to implement features effectively is to manage the API of the calls you can make, using your context to control their naming, and working at higher levels of abstraction . This is quite different from the usual Unit testing experience.

To conclude the short term benefits of reducing the amount of code to setup the environment are just not sufficient to justify the extra work required to understand how each individual scenario works.

[mgwidmann]

I'm not sure what you're suggesting, but the macro conversion happens unknowingly to the user who is writing all the features/tests. Basically all https://github.com/cabbage-ex/cabbage is doing is utilizing the standard test environment in Elixir so that we're not writing a testing framework twice and having people learn two different frameworks to test their code.

If you have an idea on how things should change, please feel free to submit an issue.

[danielfoxp2]

I have to agree with @diabolo. This approach is far from what cucumber is (afaik). It breaks the pattern found across cucumber implementations.

In SpecFlow (the .Net version of cucumber), for example, I am able to run my SpecFlow Scenarios with MSTests, Nunit, xUnit. In Ruby you have minitest, rspec... In javascript once was possible to work with jasmine to make the assertions (I did it couple of years ago as a pet project - if my memory is not fooling me) - today you can use assertion, chai and others.

What if the dev like/need/prefer/ more ESpec over ExUnit?

To me, it would need to be something like: guerkin => steps in steps I use whatever I want to make my assertions(ESpec, ExUnit, name it).

That brings flexibility and simplicity. Right?

[mgwidmann]

I think if you'd like to support other frameworks like ESpec, it would be easier done in cabbage-ex/cabbage. If you open an issue there I would consider it if we got enough support for it.

[meadsteve]

I agree with @mgwidmann ^ . I disagree that this is "far from what cucumber is" but cabbage is definitely a neat approach for fitting in with existing tools