Proposal: Property based testing

[sascha-wolf]

Elixir plans to include Property Based Testing in the future. As such it would make sense to think about how to include it into ESpec.

Elixir aims to merge stream_data into the language in the future. Taking a deeper look into Property Based Testing and stream_data would prepare us for integrating it into ESpec. This issue should serve as a platform for discussion on how this can be achieved while keeping the spec-based approach which makes ESpec such a pleasure to use.

[sascha-wolf]

Sidenote: At my company we consider putting a heavier fokus on Property Based Testing; I hope this will allow me to do some research and maybe even prepare a talk for ElixirEU. Integrating Property Based Testing into ESpec would be a blast of a topic I think.

[antonmi]

Yeah! I love the idea!

[sascha-wolf]

@antonmi ExUnitProperties in stream_data provides a check all macro which internally uses ExUnit configuration and assertions errors to easily allow for Property Based Testing.

I'm currently thinking about how ESpec should go about providing the same functionality but with a spec based approach. I've taken a look at a few RSpec-based solutions, such as Propr, Generative and Rantly.

While Rantly is the most popular solution, it makes heavy use of blocks which aren't a particular good fit for ESpec, Propr is the same. Generative on the other hand is using aliases and tags for test cases which seem like a better fit.

As such a possible ESpec approach could look like this:

describe ".reverse" do
  data string: generate(:string) # There is probably a better way of doing this

  subject do
    string()
    |> reverse()
    |> length()
  end

  property "does not change the length of the string" do # Alias for it with `property: true` or sth similar
    should(eq length(string()))
  end
end

This is obviously only a rough first draft but I think this direction is the way to go. Any input on this?

[antonmi]

Hi, @Zeeker ! Sorry for the late response, I needed some time to investigate the problem.

Let me share my suggestions.

For now, all the Elixir PBT functionality is in stream_data package. The package also implements a set of ExUnit specific macros in ExUnitProperties module

I think we shouldn't reinvent a wheel and move the same way - create an analog of ExUnitProperties.

Then we may kindly ask @whatyouhide to add this ESpec analog to "stream_data".

I'm sure this is the easiest and most comfortable way for ESpec users to start using PBT.

[whatyouhide]

@antonmi what "ESpec analog" should be added to stream_data? FWIW you should be able to build whatever you want on top of stream_data without us changing anything :).

[antonmi]

Hi, @whatyouhide ! Yeap, you are right! There is no need to make changes on your side. Sorry to bother you.

[sascha-wolf]

@antonmi So you would suggest an approach similar to this?

it "bin1 <> bin2 always starts with bin1" do
  check all bin1 <- binary(),
            bin2 <- binary() do
    expect(bin1 <> bin2).to(starts_with bin1)
  end
end

To be honest, I don't see a lot of value in that for ESpec. It effectively removes let (and as such subject) from the picture; or I can't see how they would fit in there. It would reduce ESpec to the matchers - which are nice but not the main reason we use ESpec - and ignore the structural strengths.

stream_data builds Streams. One could handle them manually in specs but I think some kind of DSL to write spec-based tests using Streams would go a long way to provide an alternative property based testing approach compared to ExUnit.

[sascha-wolf]

But in general I agree, we can use a lot from ExUnitProperties such as the ready made generators etc. but I wouldn't implement a 1to1 copy.

[antonmi]

Hi @Zeeker !

Thank you for investigating!

Absolutely agree with you about 'let' feature. My main point is - we definitely need to build PBT functionality on top of stream_data package. And I agree that coping ExUnitProperties macros is not the best idea.

If we can recreate most of the functionality with other macros, it will be awesome! Could you please prepare some kind of draft implementation using syntax you proposed earlier?

Thank you!

[sascha-wolf]

I'll do, looking forward to it. :)

[bittelc]

@dominiquejb v imp @braedongough