search

xolvio / jasmine-unit

A jasmine-based unit-testing framework designed for the Velocity test runner
9 stars 4 forks source link

Auto package stubber #3

Closed xolvio closed 10 years ago

xolvio commented 10 years ago

After the meteor-stub is loaded and before any user stubs and any meteor code being loaded, we could do the following:

  1. Scan the package.js of every package in the app directory

  2. For each Package.export('theExportedObject')

    a. Scan that package's files and find theExportedObject

    b. Look for all methods on theExportedObject and stub them with an empty function

    c. Drill deep into any objects on theExportedObject, and stub methods in there

The theory here is that we'll be able to get most cases covered so the unit testing will load up first time without the user needed to stub the **\ out of the packages they include.

alanning commented 10 years ago

Would this really be useful? It seems like automatically stubbing packages wouldn't help much because in order for the unit tests to pass the package in use would have to actually do something useful too. So the users would have to make custom stubs for that package function anyways.

I just did a small test with the velocity example project:

1. $ mrt add moment 2. leaderboard.coffee

if Meteor.isServer
  root = exports ? this
  root.cincoDeMayo = -> 
    moment("2014-05-05T09:30:30.000").format("MMM, DD YYYY")
  console.log root.cincoDeMayo

3. stubs/package-stub.js

moment = function () {
  return {
    format: function () {
      return ""
    }
  }
}

4. tests/leaderboard-rtd-unit.js

    describe("moment package", function () {

      it("is stubbed properly", function () {
        var may5 = cincoDeMayo()
        console.log('XXX', typeof may5)
        expect(may5).toBe("May, 05 2014");
      });

    });

Output of running tests:

Failures:
1) moment package is stubbed properly
Message:
Expected '' to be 'May, 05 2014'.

In order for the test to pass, I need to write a custom stub for moment().format()...

xolvio commented 10 years ago

So the purpose of auto stubbing is to allow the user's meteor code to run statically, as opposed to automatically create spies.

If you have a package the user depends on, say iron-router, and the app code has something like:

Router.map(function() {
    this.route('home', {path: '/'})
    this.route('about');
});

Without stubbing Router.map to be an empty function, starting up the app code (vm.runInContext) will fail because Router doesn't exit.

In the test the user would still have to use spies and what have you to assert the expected behavior and so fourth.

On a side note, iron-router's map function would actually require a stub and not just a spy as we're interested in what happened at the app's startup (just like Meteor.startup and xxx.template.events). In this case, after auto-pacakge-stubbing, we'd apply a proper iron-router stub.

It's actually quite a lot of hard work getting unit testing to really work with Meteor and it's due to the startup-dependencies. Any framework that taps in to startup configuration would actually require a more involved stub.

alanning commented 10 years ago

Ok, gotcha. Thanks for the explanation.

This is a bit of a hairy problem. I see a few ways of approaching it:

I'd expect loading the packages then stubbing to be the easiest. We'd lose most of the speed benefit of unit testing compared to running in-context but by stubbing after loading we'd maintain isolation of the unit test code from issues in the packages themselves.

Any suggestions on how to load the packages, maybe based on whatever mocha-web is doing?

xolvio commented 10 years ago

Sorry I missed your question!

So I have another option. Not so kosher but could just work :)

What about trying to do the loading in a try/catch clause and dynamically building a stub from undefined exceptions at loading time?

That would allow the so to at least load up in a stubbed-context. Should also be pretty fast.

Thoughts?

alanning commented 10 years ago

Yeah, that's one I had considered but I discarded it in favor of the load the packages and then stub option. Reason being that I think we'd have to re-load the code each time an undefined was found, which would hurt performance, so in that case why not just load the packages directly and walk the resulting objects?

What do you think about having Meteor load the packages as normal, walk the package objects to build the stub, re-load the app with packages stubbed?

Any suggestions about how to do the load the packages as normal part?

samhatoum commented 10 years ago

Hmm... You have access to an in-context app in main.js before rerunTests is executed. You could run your static analysis in here, create a dynamic-stub-helper.js, which jasmine-node would load automatically (any file like xxxx-helper.js is auto loaded).

samhatoum commented 10 years ago

On the try/catch solution, I still think it may have some legs without a performance hit. Consider the vm.runInContext we're using for coffee files. This currently just loads up a compiled coffee/raw js file as a string and runs it in the vm. We could wrap it with a try/catch dynamically here and if it fails, there's no reloading since the file is already loaded as a string and is still in memory. We'd stub as needed and then re-run vm.runInContext.

The change here would be to also load non-coffee in the same way as above (string first). Perhaps there's a way to extend node's require to do this stuff but to start we could try the above.

I'll attempt this route on a new branch and you can continue down the static analysis route. We can then compare our learnings and go from there.

I hope we nail this, it would be so cool that this would always work on startup. Once the app is loaded statically out-of-context, I think it's fair then to expect a developer to spy/mock/stub when they're writing tests.

xolvio commented 10 years ago

I tried this last night and it's not trivial! The errors coming from the VM don't have line numbers as the code is being loaded as a string. It can solve for "Reference Error: moment is not defined" but after that, it becomes tough. For example, moment is a constructor and the exception can't really distinguish between that and an object, and object depths is not easy to see either.

Perhaps a combination of try/catch plus code analysis might do the trick but I'll hold back until you have have results.

alanning commented 10 years ago

Static analysis is also not trivial. Loading meteor packages as normal, walking the object graphs and building stubs, then using stubs in actual app seems like it would be the easiest.

What do you think? Any suggestions on where to start there?

Sent from my phone

On May 13, 2014, at 11:49 AM, "xolv.io" notifications@github.com wrote:

I tried this last night and it's not trivial! The errors coming from the VM don't have line numbers as the code is being loaded as a string. It can solve for "Reference Error: moment is not defined" but after that, it becomes tough. For example, moment is a constructor and the exception can't really distinguish between that and an object, and object depths is not easy to see either.

Perhaps a combination of try/catch plus code analysis might do the trick but I'll hold back until you have have results.

— Reply to this email directly or view it on GitHub.

samhatoum commented 10 years ago

Something like:

alanning commented 10 years ago

Cool. For step 2, how do we run a vm with the meteor stuff in-context?

Sent from my phone

On May 13, 2014, at 3:13 PM, Sam Hatoum notifications@github.com wrote:

Something like:

Read the packages file in .meteor, then for each package look at the package.js file and look for api.export('xxxxx');

Walk that object in the meteor context (both client server) and build the stub

Pass that stub to the non-meteor context as a helper file on the file system, in a .stubs directory (we can add this to the jasmine-cli runner to see it)

OPTIONAL: Only rebuild that auto-stub file if packages file changes - Perhaps create a hash and call the helper the hash maybe. So the file would be called 8743b52063cd84097a65d1633f5c74f5-helper.js and you'd compare the hash to save you walking the packages again. We can optimize this per package even if this works out

— Reply to this email directly or view it on GitHub.

samhatoum commented 10 years ago

Ah. Before this line runs, you can walk the object as you're in-context here (server side of course).

alanning commented 10 years ago

Ok, thanks. I'll work with that.

Any ideas on doing the same for the client side? If we're in context before that line then that also means the app has been bundled (dev version, at least) and we'd have access to load the client-side resources and do the same graph-walk/stubbing?

Sent from my phone

On May 13, 2014, at 10:32 PM, Sam Hatoum notifications@github.com wrote:

Ah. Before this line runs, you can walk the object as you're in-context here (server side of course).

— Reply to this email directly or view it on GitHub.

samhatoum commented 10 years ago

Indeed we do have a loaded client. I envisage we can do this:

Caching is looking more and more like a necessary step now!

I'd wait until we know that the results of the server walker before porting it to the client as that's a (little) more involved.

alanning commented 10 years ago

Jest, a new js testing framework built on jasmine and from Facebook, also does auto mocking. They do it the way we're heading by loading dependencies, walking graph, and using stubs/mocks for actual unit tests. :-)

http://facebook.github.io/jest/docs/automatic-mocking.html#content

Sent from my phone

On May 13, 2014, at 10:32 PM, Sam Hatoum notifications@github.com wrote:

Ah. Before this line runs, you can walk the object as you're in-context here (server side of course).

— Reply to this email directly or view it on GitHub.

samhatoum commented 10 years ago

That's AWESOME! Nice find :+1:

samhatoum commented 10 years ago

Do you think we can repurpose it?

alanning commented 10 years ago

Yeah, definitely can gank stuff. Better yet, find some way to use it directly. Have to look more into it.

It's made to work with commonjs modules but if they haven't already exposed the parts we need I'd be pretty sure they would if we asked.

Sent from my phone

On May 14, 2014, at 6:00 PM, Sam Hatoum notifications@github.com wrote:

Do you think we can repurpose it?

— Reply to this email directly or view it on GitHub.

alanning commented 10 years ago

Woohoo! Got a simple auto-stubber working! Check out the 'auto-stub' branch to play with it.

samhatoum commented 10 years ago

Can't wait to try this!!

alanning commented 10 years ago

In process of extracting out stubbing code. New smart package to perform auto-stubbing can be found here: https://github.com/alanning/meteor-package-stubber

NOTE: jasmine-unit auto-stub branch not updated yet

alanning commented 10 years ago

Process complete!