Look into property-based testing (such as ponycheck) for testing topology setup

[nisanharamati]

After discussion with @slfritchie, we figured that some of the bugs we recently uncovered with integration testing could have also been found via property based testing tests, such as with https://github.com/mfelsche/ponycheck.

For example, the bugs uncovered in https://github.com/WallarooLabs/wallaroo/pull/1662 seem like good candidates for such a test:

• they are input-based
• some inputs produce valid topology setups, while others do not (e.g. missing links in routers)
• invalid topology setups result in a Fail()

The purpose of this issue is to:

1. look into ponycheck, and perhaps other property-based-testing tools, and determine whether they are a good fit for this kind of testing in Wallaro
2. Identify areas of Wallaroo code that could benefit from this type of testing

For starters, I think that most of what's being done in testing/correctness/topology_layout could benefit from property based testing (in addition to the integration testing that's being done).

[slfritchie]

FWIW I've created an annotated version of a test that I'd created for a prototype for managing complex computation workflows for Basho Pipe. It was based on the declarative framework skel. However, Skel didn't have any backpressure implementation, so the prototype mixed in a credit-based scheme also.

The annotated source is at https://gist.github.com/slfritchie/4b5120e7a83b961ab3dc920fb53c275d, and the original is at https://github.com/basho-labs/skel/blob/slf/riak_pipe_ify/test/sk_basic_eqc.erl.

Skel's tutorial can be found at https://chrisb.host.cs.st-andrews.ac.uk/skel-test-master/tutorial/bin/tutorial.html. My prototype had implemented only 3 of Skel's skeletons: Sequential (a single processing stage), Pipeline, and Farm (roughly similar to a Wallaroo stateless partition for runtime work assignment but stateful like Wallaroo stateful partitions).

Before looking at this test today, I had believed that this used a QuickCheck stateful model. In fact, it does not use a stateful QC model. Thus adapting to ponycheck would be a lot less work, if Nisan/I/whoever decided to try it.

[mfelsche]

In case you want to try ponycheck and/or run into issues or find some features missing (i guess there are quite many of them), please don't hesitate to ping me.

The most severe limitation right now in ponycheck is that it only supports synchronous tests by now, no notion of long tests as in Ponytest yet. This is my highest priority in terms of ponycheck right now.

I'd love to see this library being useful to you.

[nisanharamati]

Thanks @mfelsche! We're probably still a few weeks away from starting on this, but I'll be sure to let you know how it goes once we start.