RFC: better approaches for dealing with timers in tests

[machty]

EC makes it so easy to build polling loops that unfortunately it also makes it easy for those polling loops to infinite delay test "settlement" (since ember-testing is waiting for all Ember.run.later timers to settle and timeout uses Ember.run.later), causing tests to timeout. Or at the very least, short timers are still awaited before continuing. Either way, the result is a slow test suite that takes hacks to work around.

There are a bunch of hacky weak solutions on the docs site but it'd be nice if we had something better.

I personally don't use ember acceptance tests in my app (FYI I use Capybara for reasons that are beyond the present scope), so I don't have the day to day experience fueling my ideas for a better testing API, but I figured I'd open the issue to collect feedback on better approaches. I'll seed this issue with a few ideas of my own.

Eager/Capybara-style testing

One reason I've delayed addressing this is that I've coalesced my experiences with Capybara into some concrete feedback for the Ember Testing Unification RFC that should make some of these problems go away, but there's no clear roadmap as to whether/when those suggestions will actually be implemented.

I've been working on a next branch for EC, in which I've experimented with some testing utilities that approximate my comments in the above RFC. For example, here's an acceptance test that correctly a timer loop:

https://github.com/machty/ember-concurrency/blob/next/tests/unit/test-utilities-test.js#L95

Please check out some of the other examples in that file. In short, there's an alternate approach to asserting against the state of your app that sidesteps Ember's concept of waiters and test settlement entirely that's inspired by my Capybara experiences.

So, we could consider using the experimental API above, which may in fact be included as part of the Ember Testing Unification RFC, but there are some other ideas too:

labeled timeout()s

We could do something where tests declare the long/looping timers they expect to fire, and then await/yield them, making it possible to step through each "iteration" of a timer:

export default Component.extend({
  fooValue: 0,
  looper: task(function * () {
    while(true) {
      // oh no, a timer loop
      yield timeout(500, "x-foo#looper-tick");
      this.incrementProperty('fooValue');
    }
  }).on('init')
})

test('x-foo', function(assert) => {
  visit('/');

  return waitForTimer("x-foo#looper-tick").then((timer) => {
    assert.equal(find('.fooValue').text(), 0);
    return timer.fire();
  }).then(() => {
    assert.equal(find('.fooValue').text(), 1);
  });
});

Or if we had the generator function syntax from the next branch, we could just do

test('x-foo', function * (assert) => {
  visit('/');

  let timer = yield waitForTimer("x-foo#looper-tick");
  assert.equal(find('.fooValue').text(), 0);
  yield timer.fire();
  assert.equal(find('.fooValue').text(), 1);
});

Which seems pretty elegant.

We could also automatically derive a label for timers based on, I dunno, the name of the component class it lives on plus the name of the task that's yielding it.

Drawbacks

I like both approaches (generalized assertion-driven eager testing semantics and tagged timers), I just don't want to introduce anything that'll fragment the ecosystem depending on where Grand Testing Unification RFC winds up.

[john-kurkowski]

Before ember-concurrency, my team would hit this same acceptance-test-runs-forever problem with Ember.run.later. We worked around it by following 1 of 2 suggestions from emberjs/ember.js#3008: we switched to window.setTimeout and wrapped Ember-specific code in its callback in an Ember.run. This was preferable to the other workaround of maintaining test-specific code in app-code. The only drawback was slightly more code and the non-obviousness of using native window methods, when we tend to reach for Ember methods first. ember-concurrency could abstract this away. What would happen if ember-concurrency switched to window.setTimeout + Ember.run under the hood?

[gilest]

I know this issue is super old, but I wanted to suggest an API that I'd find useful.

I agree with @john-kurkowski that there are some async tasks I want to be ignored by ember's test helpers.

In general I love that ember's test helpers wait for ember-concurrency tasks to settle. But if you could mark specific tasks as "untracked" that would be very useful to me.

Primary use case is the classic wait 5000ms for tracking code to init.

Could be exposed through a modifier like .untracked().

Not sure of the implication details, but I'd be OK with the user being responsible for using Ember.run correctly within "untracked" tasks. Though it'd be even sweeter if this was unnecessary too.

Edit: someone in the discord suggested using rawTimeout which is pretty nice 👍

[maxfierke]

Yeah, in general I think rawTimeout is probably the best tool for the particular use-case you raised.

Can you think of other use-cases (i.e. non-timeout) where an escape from the runloop is a common need?