Statebus needs documentation on testing

[karth295]

• Unit tests

  • Client
  • Server

• Integration tests

  • Client
  • Server

[toomim]

Thanks Karthik.

You run tests with npm test. This just runs node extras/tests.js.

We'd like to test client.js, server.js, and statebus.js. However, since these tests run under node, they don't test anything in client.js. I was thinking a good way to test that is a html page, that you can double-click to open in a browser, kind of like the smiley face acid2 test for CSS.

I've put this info here: https://wiki.invisible.college/statebus/dev

Anything else we should add to that page?

[karth295]

Ah that's not what I was thinking of -- I meant document how to test a statebus app (like tawk).

[toomim]

Oh. Do we know anything about testing statebus apps that's different from testing other apps?

[karth295]

E.g. React has this page as a starting point, and plenty of blogs discussing testing strategy.

[karth295]

My PR has an example that might bite you -- bus.save() does synchronously trigger reactive functions watching that state, even on the server. So that impacts how you would test a server. I don't think we need a test framework, just best practices / examples.

[toomim]

Oh, I see — I need to take on this testcase then and examine the issue.

[karth295]

*does not

[toomim]

Ok! I've investigated a bunch of this, and you've actually run into a research question for Statebus: How can we track the ripples of reactive state changes throughout the bus?

What you wanna do in the "should increment when connection leaves space" test is make a change to the bus, wait for it to trigger some reactions in other parts of the statebus mesh network, and then once all reactions have "settled down", read the value and see what it is.

Here, I've distilled it down to this example:

state = bus.state
state.foo = 0

// 1. Let's modify foo in reaction to changes in bar
bus(_=> { state.foo = state.bar + 1 })

// 2. Now let's test it
test(_=> {
  state.bar = 3
  expect(state.foo).to.be.equal(4)  // But state.foo hasn't been updated yet!
})

In this particular case, Statebus's behavior is to execute the reactive function 1. in a setTimeout(.., 0) after function 2. completes.

How to track reactions in general? I'm not sure, but in developing the universal synchronizer, we will end up with an ack(version) message, and a History DAG. The former can tell us when a peer has finished processing a version. The latter can track the causal relationships between versions. So perhaps Statebus will be able to know automatically when all events have finished propagating and reacting.

In the meantime, it's simple to just wait a while before you read the result. You can do this:

// 2. Now let's test it
test(_=> {
  state.bar = 3
  setTimeout(_=> expect(state.foo).to.be.equal(4),
             100)   // Enough time for state.foo to react
})

This is the kind of code I write in extras/tests.js. I also wrote a helper function called delay() that you can use like this:

function test_stuff () {
  state.bar = 3
  delay(10, _=> assert(state.foo == 4))   // At 10ms, do this
  delay(10, _=> state.bar = 4)            // At 20ms, do this
  delay(10, _=> assert(state.foo == 5))   // At 30ms, do this
}

Is that what you want?

[toomim]

I'll generalize extras/tests.js into a test framework.

[toomim]

Ok, it's committed and in npm. Now you can write tests like this:

// Include the new test helpers
var {test, run_tests, log, assert, delay} = require('statebus').testing

// Setup your tests
bus = require('statebus')()
state = bus.state
bus(_=> { state.foo = state.bar + 1 })

// Define a test
test(function foo_reacts_to_bar (done) {
    log('Setting bar to 3!')
    state.bar = 3
    delay(10, _=> assert(state.foo == 4))
    log('Setting bar to 4!')
    delay(10, _=> state.bar = 4)
    delay(10, _=> { assert(state.foo == 5); done() })
})

// Define another test
test(function another_test (done) {done()})

// Run tests
run_tests()

If you then run node <filename.js>, you'll see this:

Testing: foo_reacts_to_bar
   Setting bar to 3!
   Setting bar to 4!

Testing: another_test

Done with all tests.

[toomim]

How's this? I've added it to the bottom of https://wiki.invisible.college/statebus

[karth295]

The delay function is pretty nice -- much clearer than setTimeout. Thanks for adding it to the readme.

Why invent log/assert/test functions? The benefit of using the fancy expect(var).to.be.equal(10) syntax is that if a test fails you'll see a message like expected 123 to be 10, rather than just a generic assertion failed message.

[toomim]

Well, because this doesn't need mocha. Those functions are just wrappers around console.log and console.assert.

To get "expected 123 to be 10", you can write this:

assert(x != 10, `Expected ${x} to be 10`)

It's not as fancy as expect(var).to.be.equal(10), but it's more general and the implementation is really small.

[karth295]

Fair enough.

Also, an alternative solution to the async fetch/save problem is promises. If you return promises, the tests can also use the fancy async/await syntax (in both JS and CoffeeScript).

[toomim]

You can also use delay() with mocha. Just be sure to run delay.init() at the beginning of each test function.

[toomim]

One of the goals of Statebus is to eliminate the need for promises. Check out https://wiki.invisible.college/statebus#eliminate-callbacks-with-reactive-functions

In other words, Statebus promises promise-free code.

Anyway, the issue here wouldn't be solved by promises— the issue is that we don't have a way of even knowing when a reaction has completed in general. Any change to state can trigger a function to run, which might run other callbacks, and eventually come back and change the state. I think we will eventually allow Statebus programmers to link together their transactions semantically across time. But until then, we wouldn't know when to return a promise. (Or call a callback, or tell await to stop waiting, etc.)

[toomim]

The Statebus method is simpler and slicker than even React's brand new async rendering, which uses promises: https://reactjs.org/blog/2018/03/27/update-on-async-rendering.html

[toomim]

FYI here's the documentation I added to the statebus wiki: https://wiki.invisible.college/statebus#testing-reactive-code

[toomim]

Consider, instead of this brand-new react javascript:

// After
class ExampleComponent extends React.Component {
  state = {
    externalData: null,
  };

  componentDidMount() {
    this._asyncRequest = asyncLoadData().then(
      externalData => {
        this._asyncRequest = null;
        this.setState({externalData});
      }
    );
  }

  render() {
    if (this.state.externalData === null) {
      // Render loading state ...
    } else {
      // Render real UI ...
    }
  }
}

We get the same with just:

dom.EXAMPLE = ->
  data = fetch('/data')
  # Render the UI

Even the loading indicator happens automatically. No promises necessary.

[karth295]

Oh I 100% agree on the client -- that's why statebus is awesome. But in tests you're going to write something like this:

function my_test() {
  state.foo = 'bar'
  delay(10, _ => assert state.foo == 'bar')
}

Or something like this (if your ack api is implemented this way):

function my_test() {
  bus.save({key: 'foo', _: 'bar'}, () => {
    assert state.foo == 'bar'
  });
}

The first one is a hack, and the second is ok. But a promise version would be even shorter, especially when you have to save multiple keys:

async function my_test() {
  await bus.save({key: 'foo', _: 'bar'})
  assert state.foo == 'bar'
}

It would be even nicer if there was a way to use state.foo = bar and have that be synchronous -- like a unit testing mode.

[toomim]

Let me say this simpler:

• In order to implement the async version you're promoting, you'd need a new feature that doesn't exist in Statebus yet.
• By the time that feature is implemented, you won't need the async/await, and your test will look like this:

  function my_test() {
  state.foo = 'bar'
  assert(state.foo == 'bar')
  }

  Because Statebus eliminates the need for promises, async, and await.

But there's a flaw with this example—it works right now! It's just reading the value of a state you just changed. You don't need to wait via async, await, callbacks, or promises to read the value of something you just changed. You only need to wait when reading the value of some other state that's changed in a reactive function listening to the state that you're changing. Those reactive functions can go over the network, execute in any order, and all sorts of stuff, and you don't know when they'll re-run. Anything could set them off!

[karth295]

Actually, I'm confused on how universal sync makes your example possible with reactive state changes. Is one of those magically a blocking statement?

[toomim]

Well, my example works right now. You can try it! This works with the v6 Statebus in npm:

function my_test() {
  state.foo = 'bar'
  assert(state.foo == 'bar')
}

But perhaps you aren't talking about my example, but rather in general how we can simulate blocking saves using reactive functions? Here's an example for creating an account and logging in:

// Imagine this is all embedded within a reactive function

var name = 'john', pass = 'pooh bear'
var u = fetch('/current_user')
u.create_account = {name, pass}

save(u).wait()          // Waits until account is created before trying to log in

u.login_as = {name, pass}
save(u).wait()          // Waits until logged in before trying to post

save({key: '/initial post', author: u.user, post: 'hello world'})

This function waits for two round trips before finally saving the /initial post.

The new part we'd have here is save().wait(), which will wait until the transaction has completed on the server and been acknowledged back to us before continuing. Under the hood, the save() will create a new transaction, with a version id, and the .wait() will just crash until the version id has been acknowledged by the server. When the version comes back in a {ack: "key", version: "id"} message, the function will re-run, and then it will move along farther. The old save() won't do anything the second time around, because it remembers that it's already been run.

We will probably need to know not just that the server has seen the version, but that it has brought its reactions up-to-date with the new version. I think this information could be encoded somewhere in the history DAG, but we haven't fleshed out what all that looks like yet, and there may be multiple meanings to "up-to-date" that we could allow a programmer to specify.

Am I answering your question?

[karth295]

Ah yes -- so this will work using the same crash/recover magic that makes React components work. That does answer the question.

[toomim]

Ah, great! :) I'm happy to reach completion.