Better documentation for the plugin API

[ORESoftware]

Hey all, I am writing Suman, which is a test harness like Mocha and Jasmine, but much improved, especially for backend testing. I am looking to run Suman tests in the browser. I am almost there, but I want the Suman test harness to work with Karma. Note that Suman does not use global variables, so I am hoping that Suman can call the Karma API, not the other way around.

I posted this question on SO: https://stackoverflow.com/questions/46868214/how-do-i-make-a-karma-compliant-test-harness-for-the-browser

My guess is that the Karma test runner does not hook in Mocha and Jasmine, rather the reverse.

so I assume Mocha and Jasmine call something like this:

window.__karma__.testCasePassed(data); window.__karma__.testCaseFailed(data);

and then Karma will report these events.

I am writing Suman and I want to figure out if I can make Suman Karma compliant.

When I look through the Mocha and Jasmine codebase however, I am not seeing the evidence I am looking for.

My question is - does Karma have some documentation as to how to hook into Karma? I am hoping that Karma has an API that I can call. It would be to my dismay if Karma hooks into Mocha and Jasmine's globals, instead of Mocha and Jasmine calling Karma directly.

[ORESoftware]

I see this: https://developers.soundcloud.com/blog/writing-your-own-karma-adapter

but it's from 2013, is there something more up-to-date?

[ORESoftware]

Ok it looks like I should follow the examples here:

https://github.com/karma-runner/karma-jasmine https://github.com/karma-runner/karma-mocha

and I should create:

https://github.com/karma-runner/karma-suman

is that about right? Any advice?

[johnjbarton]

Yes follow the examples. Advice: think about karma as an orchestrator and transport for a nodejs to browser system. The job of your adapter is to fire the tests and pass the results through the karma callbacks to the reporters. So the interface to karma-runner is very minimal. The test framework and the reporters are more complex that karma.

[ORESoftware]

@johnjbarton thanks, just one question for you

I am looking for more complete documentation on the karma API

https://karma-runner.github.io/1.0/dev/plugins.html

if you look at that link, it only lists the signature for the result() call, but I really could use more info on the exact API for the complete(), error() and info() calls.

.complete the client completed execution of all the tests .error an error happened in the client .info other data (e.g. number of tests or debugging messages)

How can I get more info on the signatures for these API calls?

[johnjbarton]

AFAIK there is no more information on this API. The API is not "exact". Broadly the calls are sent from your framework plugin to a reporter plugin. karma-runner does not care very much what those two ends talk about. It just not karma-runners business, so does not get involved.

There are some constraints. For example info info.total should not be null: https://github.com/karma-runner/karma/blob/de55bc63205c656eb5f5534894aa4ae92228efb8/lib/browser.js#L129 But for the most part the API that matters is the reporter plugin API.

Which of course is also not documented. I would look at the dots reporter and get your framework to work with it. Then pick up the reporter you expect your users to use.

I guess the result API is more like

var oneTestResult = {
                description: test.name,
                log: errMsgs,
                skipped: false,
                pending: false,
                success: errMsgs.length === 0,
                suite: [],
                time: test.getElapsedTime()
            };

[ORESoftware]

ahh ok good idea, I will look at the dots reporter - but I don't see a dot reporter in this org's projects

ok nvm, I found it, it's in lib/reporters

you can close this issue if you want: https://github.com/karma-runner/karma/issues/2851

[ORESoftware]

@johnjbarton as I get karma-suman closer to a production-ready state, is there some test suite I can use to verify the implementation behaves properly? Do you know if there are some good pre-existing tests that I can use to test karma-suman, the way that karma-mocha, karma-jasmine are tested? Thanks.

[johnjbarton]

@ORESoftware I wouldn't expect so: unit test would just involve your karma-suman files and e2e tests would involve passing a suman file to karma in a config and verifying the reporter or log output. You should be able to find a e2e test to use a a template.