search

awatson1978 / starrynight

Scaffolding and Testing tool optimized for Meteor apps.
http://starrynight.meteorapp.com
53 stars 11 forks source link

On documentation #56

Closed alesch closed 9 years ago

alesch commented 9 years ago

Hi Abigail,

Thanks for your efforts! In the docs I did not find your email, so I started this issue instead.

I've just given starrynight a try and these are my notes on the docs.

starrynight.meteor.com

I see in the generated nightwatch.json that tests can be run with phantomjs. How?

As starrynight did not install on OSX, I created a Meteor project in cloud9 (Ubuntu). But I did not manage to get Nightwatch connecting to selenium. It would be good with a checklist of things to check for troubleshooting.

At the end of the day, I still did not manage to get nightwatch tests running.

alesch commented 9 years ago

I managed to run nightwatch unittests by calling nightwatch directly. I had to add ./tests/nightwatch/unittests folder to src_folders in nightwathc.json.

nightwatch --env unittests --config .meteor/nightwatch.json

How do you pass --env to nightwatch?

alesch commented 9 years ago

Looking at the code, I assume that the way to do it would be something like:

starrynight [run-tests --framework nightwatch || --type unit]

Unit is described in the help output, but the case for it is not defined. Lines 113, 142 and 148 are confusing: the text does not match the launched framework.

awatson1978 commented 9 years ago

Hi! Great questions. Generally speaking, you're jumping into the deep-end of the pool by starting with the unit testing, since Nightwatch was originally written as an acceptance testing framework. But that's fine. :)

You'll need to edit your nightwatch.json config though. StarryNight can do that for you, if you starrynight generate-autoconfig. It will leave a `.meteor/nightwatch.json file for you that you can then edit.

alesch commented 9 years ago

Thanks for your answers. I got unittests running like this:

  1. starrynight generate-autoconfig
  2. As my unittests folder is at the same level than ./tests/nightwatch/walkthroughs, I added it to src_folders.
  3. starrynight run-tests --framework nightwatch --env unittests --autogenerated

For phantomjs: starrynight run-tests --framework nightwatch --env phantomjs --autogenerated

awatson1978 commented 9 years ago

Sweet!
To be perfectly honest, I've personally never tested it with that particular configuration before (phantom + unittests + autogenerated). I've certainly used each component in isolation, but not together. So that's great to hear.

If you got all that working, the rest of the Nightwatch/StarryNight features should all fall into place, like tagging, screenshots, logging, chai expect syntax, scaffolding, environment analysis, refactoring, etc.

Thanks for the feedback and have fun!

alesch commented 9 years ago

I think that claimed victory too soon. How do I access the Meteor code that I want test, from my unittests? Say I define a global variable X, how do I get access to it from a test?

awatson1978 commented 9 years ago

Interesting question. There's a couple of different approaches, depending on whether you're wanting to test server, client, or package code.

The primary issue though, is whether or not you've written your code to be accessible in the first place. And that generally means naming functions, and cleanly attaching them to known objects (ie. Meteor, Templates, or on an isomorphic object exposed from a package). But lets assume you are.

Take a look at the following code, which uses unittests in the acceptance test runner, and which can hook into a running app and test code on the client and from packages. That's one way to access running code. (This is akin to running an engine diagnostics meter while your car is running.)

https://gist.github.com/awatson1978/e9fd118781e68c5e851e

To access static non-running code, the functions need to be attached to an object that can be found by the runner. This is similar to opening the hood of your car and looking at the engine while the engine if off. It's also where I'm a little less clear on how to use Nightwatch, as it's a new feature and being developed by others. But I'm more than happy to help.

If we look at the unittest example, we see the following line:

var Utils = require('lib/util/utils.js');

We need to extend this with something Meteorish...

var Foo = require('client/components/foo.js');

But that foo.js file isn't entirely guaranteed to have a module.export or otherwise export any objects. So, we may need to write a wrapper for it. If we're lucky it maybe has a reference to Template.foo.x and we can test on the Template object. But I wouldn't hold my breath. We may need something more like this:

var Template.foo = require('client/components/foo.js');

My guess is we're getting close to needing to stub the various default Meteor objects. Velocity has a project which is relevant; and I'm thinking we may want to grab the following file and figure out how it gets integrated.
https://github.com/meteor-velocity/meteor-stubs/blob/master/index.js

So, here's what I'd recommend trying:

See if you can get anywhere with that. I'm writing documentation, and will be trying to get some of these answers written up and posted.

alesch commented 9 years ago

Maybe this is naive but... would it be possible to ask the meteor tool to build the source for us? and then use that as our context for the test?

awatson1978 commented 9 years ago

Oh, I don't think that's naive at all. Not 100% sure it will work. But it may be an excellent approach. For that, we'd need to reference files in the .meteor/local/build directory. We have a file scanning tool that's working great. We just need to figure out what else to maybe add to the scanning rules. If you can dig around in .meteor/local/build and make any progress putting files from there into the src_folders directory, please let me know.

alesch commented 9 years ago

I've been trying munit and package testing (both tintytest and velocity). My conclusions so far are:

Integration tests and scope issues

Regular package testing seems to be much better supported than velocity integration tests. So maybe the solution is to package my app into several packages, and then I get to choose which parts of the ecosystem should be available to my tests. This is indeed a bit of extra work, but tests run quicker than with regular velocity.

Which runner to use?

I definitely do not want to rewrite my tests because I choose another runner. The ideal would be to write the tests in one framework, and then be able to choose runner freely. For example, as a Webstorm user I'd like to have a runner that is integrated with my IDE.

At the moment it seems that jasmine is the best supported runner in velocity, and velocity seems to be MDG's choice (right or wrong, that is the reality). So I'd like to write my tests in jasmine (or mocha) and be able to choose a runner afterwards.

My guess is that there many other users in my situation, and if you'd like a wider adoption of starrynight, a path compatible with velocity could a wise strategy.

awatson1978 commented 9 years ago

Hi Alex, Thanks for the feedback. MDG might be choosing Velocity, but there's a portion of the Meteor community that doesn't agree with that decision, the Velocity architecture, or how the management of the Velocity project has been handled. Long story short, I've been funded specifically (among other things) to provide an FDA ready testing solution. And since some members of the Velocity project chose to team up and vote out our contributions to what was advertised as a community project, I'm basically now being funded to provide a community alternative to Velocity.

As you may have read about, my clients and our partner organizations are in the process of publishing a Clinical Meteor Track, where we'll be adding D3 back in, and replacing Velocity with Nightwatch. Because of the way that management of the Velocity project was handled, it's actually going to get murkier as the Clinical Meteor Track is published, since Velocity is not going to be the Clinical Meteor Track's official runner. (Similar to how some people use React or Angular instead of Blaze).

As for compatibility between StarryNight and Velocity, it would be fairly straight-forward to integrate the two. Simply need to add Velocity as a framework in the same way TinyTest was added. If somebody else wants to write that code, I can provide a pattern and files to follow, and would accept pull request for it. But I'm also not going to personally waste any more of my own time trying to integrate it.

As for getting additional people using StarryNight, there's already the healthcare working group and our partner organizations who are invested in using its features. It will grow simply because we've got a stable and growing base of clients, are currently interviewing and hiring developers for StarryNight/Nightwatch, it's going to be the official runner of the Clinical Meteor Track, and graduate students and interns are being assigned to use StarryNight instead of Velocity.

alesch commented 9 years ago

Thanks for the extra info. Great to know that you've got funding to get this working. I love D3 and agree that Velocity is too complex. I'll be eagerly watching your team's progress.