Investigate running tests in node

[jasonLaster]

We are currently working on running our unit tests with xpcshell because it would be easier to integrate when we merge the experiment back.

We've made a lot of progress in this effort. We've made it possible to run xpcshell-unit-test from any directory. We've also added additional shams to ff-devtools-libs for running async tests in an xpcshell environment.

In the process, it's becoming clear that our test infrastructure would rely on the xpcshell environment in a similar way as the gecko debugger client relies on browser internal libraries. This conflicts with one the the primary goals of devtools.html, which is to run the client in any browser context and potentially debug any JS environment.

With this in mind, it would be nice to explore running the debugger's unit tests in node. The primary benefit of node, would be that we would be constrained to writing tests that could potentially run in any environment.

This approach would add additional work when it comes time to merge. I see two possible approaches:

1. we add some hooks in our test infrastructure to run in xpcshell
2. we add try node test runner. This option might be more in vogue with project tofino.

One of the side-effects of using node is the lack of xpcshell test runner. If we go with node, we could either use mocha or write a small test runner with similar semantics to xpcshell test. There has been significant interest in using mocha before, so we should consider that option and see if we would be comfortable using mocha post merge.

[jlongster]

Yeah, I was thinking this too. Yesterday when we were working on tests it was clear that even with all my work to smooth it over, there are still serious issues. My test harness is really simple and we'll probably have to keep investing in it. And it's doubtful we'll be able to hook it up to travis; if we do, it will be a lot of work.

Mocha on try has been done before. The Hello team has done it. We will have to work that out when we merge back in, and we definitely are breaking the workflow of normal devtools, so I don't know. Breaking the workflow is the biggest deal; technically we can do this but other people on the team will have to learn how to run our tests. All of our docs and getting started guides won't apply to us.

But I think it's probably worth it. Worst case, we will have to convert our tests to xpcshell/mocha tests but that can be aided with code transform tools. But if we pave the way for mocha, most likely we'd rather go the other way and convert xpcshell to mocha.

[jasonLaster]

I'm going to add a couple sub-tasks here because it's a lot of work, but I want to capture the additional steps so they're not missed:

• [x] travis support
• [x] web runner (i want to debug our debugger tests in the debugger ofcourse)
• [x] karma runner (lets start running our tests in multiple platforms) #58
• [ ] watcher (run tests as we work) stretch
• [ ] coverage (it might be nice to get some coverage stats on the project early on) stretch

[jasonLaster]

Currently, we're using webpack to build for node, which is unecessary because we're using commonjs. This should be fixed soon.

• [ ] remove webpack dependency

[jasonLaster]

Just spent tried to remove webpack and it's definitely possible, but it's not trivial. Basically, there are two things that you need to do to run our tests:

1. transform code to es6 and stage-0
2. alias ff-devtools-libs to devtools

To transpile, you need to use babel presets as well as babel-register. babel-register, is babel's hook into require and import to be able to transpile on the file

To alias, you need to use babel-plugin-module-alias, which provides a similar system to webpack aliases

One way to do this is to have a test/index.js file, which requires tests

"use strict";

require("babel-register")({
  ignore: false
});

require("../js/actions/tests/sources.js");

and a babelrc file.

  "presets": ["es2015", "stage-0"],
  "plugins": [
      ["babel-plugin-module-alias", [
        { "src": "./node_modules/ff-devtools-libs", "expose": "devtools" },
        { "src": "./node_modules/ff-devtools-libs/sdk", "expose": "sdk" }
      ]]
  ]

This approach, works but is lame for two reasons:

1. there's a project level babelrc config for just node tests. It'd be nicer to pass these options to babel programmatically
2. The test paths are hard coded in index.js and it would be nice to pass them in via the command line e.g. js/actions/tests or js

I think one way to overcome this is to use a mocha compiler, which we could write with the correct options.

[jasonLaster]

Closing this issue because we now have the node test script working. Added two smaller issues for enhancements.