Write a Chapter: Automated Testing

[toolness]

• Prefer Mocha with expect.js. Mocha provides good error feedback that's easy to read, and it's quite flexbile. Both of these also work in the browser, which is nice.
• If necessary, use Sinon.JS for mocking and stubbing.
• If it's easy, consider using PhantomJS to run lightweight functional tests. If your code needs to bend over backwards to work in Phantom, though, don't worry about it.

Other questions:

• What about cross-browser functional testing? Should we use Sauce Labs or BrowserStack or something else?

[k88hudson]

Test frameworks/helper libraries

• does expect.js have good error feedback? v.s. should?
• use the spec reporter -R spec
• i use mockrequire https://github.com/mateodelnorte/mockrequire for browserify

Baseline test coverage

• stuff that can be easily unit tested (i.e. no human interaction/DOM) should be separated out and unit tested
• new features that involve human interaction/DOM should have manual test scripts written

Integration testing

• we need this for our core services (e.g. login) + their node clients
• not necessary for everything

Testing in staging/production

• continue to do back end error monitoring/logs (always set up newrelic and messina for new projects)
• we should set up front end error reporting (what will that be? we don't know yet.)
• when you deploy a new tag, watch for:
  • increased error percentages in loggins and/or newrelic
  • opsview alerts (when shit goes down)
  • newrelic change reports:
  • slower load times/response times
  • overall performance for the application
  • 
  • broken routes in newrelic:
  • 

Philosophy

• writing tests should be easier than not writing tests. TDD should actually help when designing services or a nodejs API
• try to separate front-end code that is purely programmatic (no DOM etc.) into browserify modules so they can be separately tested
• JSON and component configuration can be validated against a schema (e.g. https://github.com/geraintluff/tv4)

Good resources on testing that align with our values: lets add some

[toolness]

Oh, we should also consider measuring test coverage, and plug the stats in to some kind of dashboard, to incentivize improving test coverage.

[simonwex]

@toolness, @k88hudson hell yeah. :metal:

@toolness do you mind filing some of this functionality against @adamlofting's repo? https://github.com/adamlofting/webmaker-metrics

@Pomax will be stubbing out a sample project that we'll all use to implement these opinions here: https://github.com/MozillaFoundation/unicorn

[jbuck]

Since we're looking at alot of the other Facebook libs for dev, how about their test runner, jest?

[toolness]

I haven't used jest, but I'm cool with investigating it--I am not tied to Mocha. Honestly, though, I don't even mind different projects using different testing tools--which is why I wrote "prefer" in the draft decisions--as long as metrics like coverage % still get passed back to the build dashboard.

[jbuck]

Yeah, I think that's a better metric -- Do we have tests? Do they cover enough of the thing to be useful?