build jsdom capybara driver

[sfcgeorge]

JSDom is a DOM implementation that works in any JS runtime (mini_racer perhaps) and can be used for doing functional testing without a browser. I'm thinking this would be faster and more reliable than headless-chrome et al.

Really this wouldn't be a Hyperloop specific tool, more just another Capybara driver somewhere between Rack and Selenium in capabilities.

Looking at the Capybara driver base class it looks like it would be surprisingly easy to implement. I'm thinking of giving it a go when I find the time but opening an issue here anyway for reference and thoughts.

Testing framework options - all would just be different Capybara drivers:

• chrome-headless standard Capybara setup: full functionality, slow, buggy
• Hyperloop remote browser: full functionality, simple, better debugging, slow
• JSDom browserless (this issue): speed, full DOM & JS, fast, no rendering
• [React Test Renderer](): fastest unit tests, not acceptance tests

[catmando]

Good analysis.

Given you are using hyperloop, then the Hyperloop Remote browser (#5) would always be used rather than standard capybara.

I think if #5 were done properly, it would fast enough to make the other options (JSDom and TestRenderer) not very interesting. Sure they might be a bit faster, but would it be worth maintaining 2 more tools, and adding to the confusion of which tool to use when?

So my feeling is that we get #5 done and see if that doesn't fix the problems we are having with speed and reliability.

[catmando]

I just noticed that elsewhere you said your priority would be to get JSDom working first. I may be missing something here, but Hyperloop remote browser seems the much easier problem to solve. I.e. its just a matter of building a Hyperloop specific capybara driver. The actual communications is well understood (see hyper-console for example) and is built ontop of hyperloop operations.

On the otherhand with JSDom you are going to have to do the same amount of work in terms of writting the capybara interface, plus there is going to be some finagling to get it work with hyperloop and react, etc.

Plus you are now testing in an environment that is radically different from production. With the hyperloop capybara driver, you are using all your production code (hyperloop stack) without change, and you are running on the same base browser code that many of your users will execute on.

[sfcgeorge]

I agree with your points. I think it comes down to tradeoffs and constraints with none of the solutions being perfect; so choosing will come down to personal preference and what works best for the application being developed.

Motivations

The easier and faster writing tests can be the better. That's why I like hyper-spec and a big reason I chose Hyperloop. You just swap visit for mount and that's it, nothing new to learn, no separate test framework, just the same Capybara you know and tolerate. It's a beautifully streamlined setup.

CI builds are slow, a few minutes are setup and deploy, then you add all the test time. So you can't QA a feature right away, you have to come back to it later, and by then you have multiple pull requests, builds queued, out of date branches, merge conflicts. Anything I can do to get CI out of my way faster the better. Especially if a flakey browser spec makes you restart CI.

Development cycle. If tests are instant you can run the spec while developing for a nice tight feedback loop. It works for unit tests, but as soon as a browser is involved that goes out the window. Browser specs seem to take ~4s just to boot, then ~1s per spec. Not pleasant.

Basically, I'll never be happy with the speed of tests and always looking for ways to speed them up and increase programmer happiness. In our app I tend to prefer faster unit style tests wherever possible—even databaseless in the few places we can make that work—then a few higher level feature tests from a user perspective too. And even those are mostly Rack driver whenever possible. Those few JS tests running in chrome-headless take the majority of test suite run time despite being fewest in number.

Choice

RSpec Rails has tons of very similar test types: controller, request, feature, routing, system. "routing" is basically useless even. Which is crazy, but we actually have found different uses for and use each of: controller, request and feature specs. I wouldn't like to see us go as far as RSpec but some options would allow making the right tradeoffs given particular situation.

E.g. I've enforced controller specs to be databaseless, mainly for quickly testing auth, redirection, status codes; all the possible paths through a controller. Then feature specs for happy path testing the key things a user can do. And finally integration specs we use on occasions we want to run a full request while still having visibility inside the controller to stub / check instance variables; rarely needed or the best way, but when it is it's great to have.

The point of this driver should be a simple driver swap that increases speed and reliability across all JS tests in the app (not just hyper-spec ones) at the expense of being less close to a production browser. I would see myself using it to test individual components quicker, not quite true unit tests but less than full acceptance test.

Outcomes

We basically get the Selenium driver for free, so that's fine for certain tests like acceptance style perhaps. Then maybe we could have 1 or 2 additional drivers that meet different constraints - faster or more reliable.

Having looked at Capybara drivers, JSDom and mini_racer I have this hunch that it might all be quite easy to cobble together. But that does sound like famous last words. JSDom is commonly used for React testing, but there may still be snags. If it turns out to be a big mess, or unreliable, or slow, then I definitely wouldn't pursue it.

I don't see a JSDom browser needing anything Hyperloop specific so it needn't be a part of the Hyperloop project, could be developed separately and just dropped in.

To me, making a driver that uses operations with a regular browser sounds like more work cobbling, but I don't know the Hyperloop code at all. If any of these end up getting built, then we can compare and see how well they work. They may all be terrible and we begrudgingly stick with Selenium :P

[catmando]

huh... so I went back and took a little closer look at JSDOM, and perhaps I understand a little better.

So the idea here is that all the JS DOM APIs are presented, and because its just written in JS you can run it inside of mini-racer for example. (right?)

Thus the communication transport between the JSDOM and Capybara is just simple ruby method <->js function calls (Hyperloop does this today during prerendering, so there is example code to look at.)

Okay let's compare to using a browser, and hyperloop operations for the transport. In both cases you have to write a capybara driver to communicate with the JS/browser side of things. In both cases, you get immediate benefit of not having selenium. And both cases are going to enable (with some work) running each spec without a full tear down and setup.

JSDOM

• no hyperloop dependency

• uses execjs (miniracer / rubyracer / etc) which is just a gem so dependency can be bundled.

• your test environment is quite different from the actual production environment

• you will still want some ability to do full acceptance style testing, which means you have the whole selenium integration headache.

• no fun for debug (but might be fixable by adding some console like features, but still no visual of what is going on.)

Browser (any brand or headless) + Hyperloop Operations for Communication with Capybara

• test as you fly, fly as you test

• switch from headless to normal browser for easy debug

• gets rid of selenium even for full acceptance testing

• depends on external software (i.e. browser / headless browser for test)

• requires you include hyperloop in test - not sure if this would be a big deal or not

In terms of ease of implementation, I think Hyperloop Operations is going to be easier, but I could be wrong. My belief is because there is simple working code (HyperConsole) that does very close to what a capybara driver would do. I.e. uses ServerOps to receive and broadcast async data.

My conclusion is that I can live easily with the two negatives for using Browser + Operations, but the three negatives for JSDom are show stoppers.

My question is am I missing any +/-'s

[sfcgeorge]

For me I don't care about drivers visually rendering as you don't test anything based on how it looks, just that elements exist, can be clicked, etc.

The big point for me is speed. Using Operations is still using a full browser so I doubt it will be any faster. Maybe Selenium adds a little overhead, but most time is spent by the browser rendering the page.

Biggest downside for me of Operation driver is that it would only work for testing Hyperloop parts of the app. Most of our app is not Hyperloop so we'd need a regular Selenium driver to test those. As I assume the Operation driver would be sending clicks and whatnot to React. Unless you're suggesting it would work for non Hyperloop pages with vanilla Javascript as well, I don't know how you're thinking of implementing.

[sfcgeorge]

I spent a couple nights scrappily writing my proposed JSDOM driver and I have a basic implementation working. It has enough methods implemented to do simple Capybara stuff like visit "/foo" and expect(page).to have_css("p").

Thus I was able to do a very artificial benchmark of JSDOM vs Selenium vs RackTest and (ignoring startup time) the JSDOM version is about 5x slower than Rack and 17x faster than Selenium, so between the two and about what I hoped.

I'll keep going and see if it works in a real app not just a tiny dummy Sinatra one, and what the performance and capabilities look like then. No idea if it will handle everything Hyperloop needs like WebSockets. I'll put it on GitHub too once I've done a bit more.

[catmando]

FYI re: operation based solution. Yes I was assuming that it would simulate events at the lowest possible level. Wouldn't a JSDOM driver have to do the same?

So operations would just be the communication transport.

[sfcgeorge]

Oh I see, fair enough then. Yes with the JSDOM driver you basically just send JS commands to the "browser", it's been easy so far - click() and the like.

Welp, with a lot of fiddling I got my JSDOM driver working on the work app. Well, on 1 feature spec involving clicking and AJAX anyway. Selenium takes 1.5s, JSDOM takes 0.17s, so about 9x faster. I think this is promising enough for me to continue with.

[sfcgeorge]

Alpha version pushed to Github. Needs more Capybara API fleshing out (not hard, just time). Also needs testing on more, well, tests. I see all of the Hyperloop log messages coming through so it looks like it's working at least a little, but I haven't tried testing Hyperloop through it yet.