Marble Testing

[thomas-jeepe]

Marble testing for me is one of the large advantages of rxjs, it is very clear and easy to use and describes the code well. The current solution is using a virtual timer and ticking forward x milliseconds, which works fine, but I don't find it as declarative.

I decided to make a most-marble do see if I could get it to work, and I got it working pretty well.

Here is an example:

itenv('should test a periodic function', 1000, presume => {
  const period$ = periodic(1000)
    .constant(1)
    .scan((acc, v) => acc + v, 0)
    .skip(1)
    .take(5)
  presume(period$).toBe('1234(5|)')
})

When compared to a similar stream tested with only the virtual timer, it looks far more declarative:

it('is easy to use', () => {
      const stream = most
        .periodic(1000, 0)
        .scan(x => x + 1, 0)
        .skip(1)
        .take(2);

      const env = run(stream);

      return env.tick(/* advance by 1ms */)
        .then(result => {
          // Make sure the stream didn't terminate
          assert.ok(!('end' in result));
          assert.ok(!('error' in result));

          // Ensure that the initial `periodic` value was emitted
          assert.equal(result.events.length, 1);
          assert.equal(result.events[0], 1);

          // Advance 1000ms; the stream should emit the second event and complete
          return env.tick(1000);
        })
        .then(result => {
          assert.equal(result.events.length, 1);
          assert.equal(result.events[0], 2);
          assert.ok(result.end);
        });
    });

The repo for most-marble is here. As a quick note, I am pretty unfamiliar with most's internals, so I probably implemented the Scheduler and MarbleSource incorrectly, although it works.

So, there are current limitations on my marble testing as specified at the end of the README, however, I wanted to get the mostjs contributors' opinions on what I am doing and to put something out there.

Also, Rxjs 5 generates very nice picture graphs from marble tests that are used as documentation. I was thinking that if most had something similar, it could make some of the documentation nicer and friendlier to newcomers.

Summary

Marble testing is awesome, I implemented a subset of marble testing with most and it is really nice. I would like to work with mostjs contributors to make it full and make it more official and I would also like to see if generating diagrams from marble tests for operator documentation would be possible and wanted.

I also will begin to use this on my own project which I built it for, meaning the repo might change over time.

[thomas-jeepe]

https://github.com/ReactiveX/rxjs/blob/5a2266af8770463156a67527ee3aca990cfd0365/spec/helpers/tests2png/painter.js Seems to be the file handling the painting of the graph

[briancavalier]

Hi @thomas-jeepe, it's awesome that you put together a POC for this. Thanks!

I haven't had time to look at it thoroughly yet, but I'll try to do that soon. In the meantime, have a look at @most/core (the new core of what will be most.js 2.0), specifically https://github.com/mostjs/core/pull/61. We've started to standardize a virtual event stream on which marble testing could be based.

It isn't specifically tailored for testing--for example, it doesn't have something like your presume--although testing is definitely a use case we want it to support. Visualization to marble diagrams, as well as parsing from marble diagrams are also things we want it to support.

I wonder if we could find a way to align that and your work on most-marble. I'd love to hear any thoughts you have.

[thomas-jeepe]

Ok, so I looked through your POC. I'll outline the differences.

First, my package isn't too much different under the hood. The marble diagram is simply a way to convert data into an array of events, which is a similar implementation to yours. presume also just captures all events from a stream and puts it into events array to make easy assertions. So, parseMarbles operates like this:

const events = parseMarbles('123|', undefined, 10)
expect(events).toEqual([
  new Next(1, 0),
  new Next(2, 10),
  new Next(3, 20),
  new End(30)
])

Next, Err and End are what are considered an "event". The definitions are really simple. Events have a frame/time and possibly a value depending on if they are Next or Err.

I don't currently have a class such as Infinite, Finite or Errored, I only use an array to represent the events. Rxjs uses an array as well. The reason for this is that I assume that all events must error or be finite (subject to change in the future.)

For virtual-stream you might want to decide how to handle infinite streams. Also, I saw the comment on the PR about a data structure for the events, arrays should be fine as RxJs uses them as well.

Creating a stream from the events is very similar between both implementations. Mine is rather simplistic and simply schedules delays then runs the events. It seems to generate valid enough streams.

The next part is getting events from a stream, I use a CaptureSink which, again, is the same concept as your Collect sink.

However, in order to bypass the timing, I used a TestScheduler while you used a VirtualTimer. I think for manually pushing or ticking forward time, a VirtualTimer is better while if you want to run the whole stream and collect all the events, a TestScheduler is better. But, since you have a better understanding of what is under the hood, it would probably be a good idea if you made that decision.

The rest of my api is simply wrappers around TestScheduler, CaptureSink, MarbleStream (generates a stream from events) and parseMarbles.

stream parses the marbles into events and creates a stream from the events.

stream = (marbles: string, values?: { [name: string]: any }) => {
  const events = parseMarbles(marbles, values, this.timeFactor)
  return new MarbleStream(events)
}

For running tests, I use a flushing approach similar to RxJs. When you presume something, it begins capturing the events from the stream and then flush returns a promise which resolves when all tests are ready to be compared, then compares them.

testenv handles creating an environment and flushing.

So, I ended up reimplementing many sections of your virtual-stream. The rest of this package is really a wrapper on creating streams from events and creating a list of events from streams, making it easy to create compare streams and make assertions on streams.

If virtual-stream has a working TestScheduler, CaptureSink and toStream for events, we could wrap the streams with just about any kind of assertion library, because from that point it is just comparing an array of events.

[briancavalier]

Thanks for all the detail @thomas-jeepe. It does seem like the two approaches have a lot in common! I'm planning to respond in more detail in the next day or so ... hoping we can keep moving this forward.