Errors in `before-all` are silently ignored

[lastquestion]

Consider this test:

;;; -*- lexical-binding: t; -*-

(describe
 "A suite"
 (before-all
  (message "well")
  (setq f (1+ f))
  (message "this is unexpected"))

 (it
  "contains a spec with an expectation"
  (expect t :to-be t)))

The output is this:

~/tmp $ emacs -batch -f package-initialize -L . -f buttercup-run-discover
Running 1 specs.

A suite
well
  contains a spec with an expectation (7.30ms)

Ran 1 specs, 0 failed, in 11.24ms.

That is unexpected. I would expect that that buttercup should catch all errors in before and after handlers, and print that they fail, and also skip the suite if it fails in a before.

After all, that is what Mocha does:

describe("A test", () => {
  before(() => {
    throw "Wow, I'm evil";
  });

  it("works", () => {});
});

~/tmp $ mocha test.js

  A test
    1) "before all" hook for "works"

  0 passing (6ms)
  1 failing

  1) A test
       "before all" hook for "works":
     Error: the string "Wow, I'm evil" was thrown, throw an Error :)
      at processImmediate (internal/timers.js:456:21)

[snogge]

I've verified that this happens, it's probably a bug.

It's due to the fact that errors in before-all and before-each are stored in the suite record instead of the spec record. And failure states for suites are then ignored.

As you know buttercup is modeled on jasmine, not mocha. And I'm not knowledgeable in either of them. So I'll have to look into how that will work in Jasmine. Your conclusions look entirely valid though.

[lastquestion]

As you know buttercup is modeled on jasmine, not mocha. And I'm not knowledgeable in either of them. So I'll have to look into how that will work in Jasmine. Your conclusions look entirely valid though.

Oh so sorry! I confused the two and thought buttercup was based on Mocha. My bad!

In general I think both work the same way regarding before* hooks of the various kinds. I think in some corner cases mocha may have deviated from jasmine. But I haven't used jasmine in many years; mocha had pretty much won the unit testing war (though it is now being supplanted by jest...)

The major tricky corner cases that have caused man-years of heartache are around what to do if the before- fails, or an after- fails, e.g. should you continue running more tests. Like, if a before-each fails, ought you to run the tests themselves? And should after-each failures stop further tests in that suite from executing?

Should after-* failures even be considered failures? (Probably.)

Thanks for your quick reply!