code coverage with istanbul, mocha, and Selenium Webdriver running on Node.JS (WebDriverJS)

[attodorov]

Hey, I have the following scenario where I would like to collect code coverage and I would be glad to get some feedback about potential solutions using istanbul:

• I have a web app - HTML, CSS * Javascript - that needs to be tested
• it is mounted with ExpressJS and runs on some port, for instance 8080. There isn't any server-side logic, it's just js/css/html
• the tests are written in mocha and use WebDriverJS to bring up the index.html, do whatever interactions the test needs to do (such as drag & drop , fill a form, etc.)

I can instrument my javascript resources with istanbul, then load the app using the instrumented scripts, but I am missing the link between the coverage collection and the reporting using my setup. when I look at the window object at runtime, I can see the code coverage metadata and its maps, but how do you think should I tackle collecting and reporting this data? Is there anything available or should I write my own lib to do that?

I cannot use istanbul's integration with mocha, because my app is actually loaded by WebDriver and lives in its own context there (including the window object and everything else). I actually have requirejs modules - my app is modularized, and I am testing as much as I can using pure unit tests with node's requirejs, as well as jsdom, but for a big part of the app that's not possible because it's highly interactive, so my closest bet is WebDriver.

On a separate note, I don't want to use tools like PhantomJS because it looks like they're quite unstable and I have no control on the browser. For example I get different pass rate when I run on Phantom vs real Chrome. I also can't run it on anything else than whatever phantom uses internally, I can't do FF, IEs, and so on.

I've also tried using other code coverage libs like Blanket, but it has the same issues with respect to my scenario. Istanbul has proven to be a lot easier to use and I use it all over in my other tests that don't rely on selenium's web driver.

Thanks Angel

[swatiagarwal1284]

I also have the similar requirement and looking for the solution. Please update this post if you find any solution or workaround.

Thanks, Swati

[gotwarlost]

Have you taken a look at istanbul-middleware for ideas on the general approach?

https://github.com/gotwarlost/istanbul-middleware/

It won't be directly usable since your app is not a node app but here is one set up that I can see working.

1. Run your app on port, say 8000
2. Run a basic express app with istanbul-middleware on port 8001
3. Have a proxy on port 80 that forwards the /coverage path to the express app and all others to your main app
4. At the end of your webdriver tests (for each page, before navigation) post the coverage object to /coverage
5. After all your tests have run, download the coverage reports

[attodorov]

Thanks, i got it working in the following way. (Btw, i am not sure why I need step 3. from your list).

1. Create a simple node express app, this is the whole app:

var coverage = require('istanbul-middleware');
var express = require('express');
var app = express();

 app.use('/coverage', coverage.createHandler());

 app.listen(8888);

1. Add the following logic to every test suite which uses selenium-webdriver:

after(function (done) {
    // post coverage info
    driver.switchTo().defaultContent();
    driver.executeScript("return window.__coverage__;").then(function (obj) {
        var str = JSON.stringify(obj);
        var options = {
            port: 8888,
            host: "localhost",
            path: "/coverage/client",
            method: "POST",
            headers: {
                "Content-Type": "application/json",
            }
        };
        var req = http.request(options, function (res) {
            console.log("\nFinished sending coverage data.");
            done();
        });
        req.write(str);
        req.end();
    });
})
test.after(function () {
    // at the end of the *whole* test run, download all coverage data and place it somewhere
    console.log("Closing browser.");
    driver.quit();
})

The only assumption that exists is that the js code loaded by the app under test needs to be manually instrumented with istanbul. In order to see report details, i am using the following command:

istanbul instrument src.original.js --output src.js --embed-source true

If i don't give it the "--embed-source true" option, I can't see detailed reports, just the summary. Makes sense, I guess.

Thanks, Angel

[mahuntington]

Hi @attodorov and @gotwarlost I got most what you suggested working, but I'm unclear how to get the reports. In the example @attodorov gives just above, I console.log(res) within the http.request callback, and it prints out JSON. As far as I can tell this isn't the coverage report, though. Any suggestions?

Thanks,

Matt

[mekdev]

Guys this is great!

I have been looking all over and this is the only relevant info I have found for doing istanbul code coverage using selenium-webdriver to drive the UI.

I have some questions.

1. @attodorov can you please briefly explain what you guys have to do for the step below ? // at the end of the whole test run, download all coverage data and place it somewhere
2. Did you guys experience any skew in the report numbers, does the coverage report include the libraries that the app is using e.g. stuff in node_modules ? Did you have to so some source mapping to get this to work with accurate numbers ?
3. @mahuntington were you guys able to get this to work after following the example ?

Thank you!

• Mek

[mahuntington]

hey @mekdev! I was able to get it running. To be honest, it was a while ago, so I don't remember what I did specifically. Take a look at https://github.com/mahuntington/e2e-istanbul, though. See if that solves any of your issues.

[dailystore]

That's great! I'm looking around and this thread seems to be relevant to my problem also. @mahuntington can you kindly elaborate how to get the instrumented code run please?

Thank you!

• An

[mahuntington]

hey @dailystore, I've updated my repo (https://github.com/mahuntington/e2e-istanbul) with a README. Let me know if you have any other questions!

P.S. I've also updated the code a bit to work with the latest browser drivers

[dailystore]

@mahuntington Thank you! I'm able to get the coverage data against my own node server as well but the coverage report is not working. In the report detail, instead of showing lines of code with a nice format, it shows the instrumented code with no format so there is no way to know which line is covered or not. Are you aware of this issue?

[mahuntington]

Hmm, it worked for me. Can you put it up on github and paste a link?

[mahuntington]

Actually, @dailystore, I don't want to bug anyone else who's following this issue if there's a problem with my repo. Can you create an issue on my repo and we'll figure it out there?

[ORESoftware]

this was a GREAT thread...and I got it to work!

I wrote up an article on how to do this:

https://medium.com/@the1mills/front-end-javascript-test-coverage-with-istanbul-selenium-4b2be44e3e98

@attodorov I gave you credit for my findings :) thanks dogg

[kagarwal29]

Hello Guys,

I am taking help of this thread to get JS coverage report.

I am using code mentioned by @attodorov to setup server & posting coverage report from broser to server.But when I try to execute this line : 'driver.executeScript("return window.coverage;")', I get following error :

ReferenceError: window is not defined
    at /Users/kapaga/Desktop/Node/webdriver.js:23:19
    at ManagedPromise.invokeCallback_ (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/promise.js:1384:14)
    at TaskQueue.execute_ (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/promise.js:3092:14)
    at TaskQueue.executeNext_ (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/promise.js:3075:27)
    at asyncRun (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/promise.js:2935:27)
    at /Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/promise.js:676:7
    at process._tickCallback (internal/process/next_tick.js:109:7)

It seems Node doesn't support 'window' instead there is something called 'global' for Node.But when I try to use 'global', I get following error :

WebDriverError: unknown error: global is not defined
  (Session info: chrome=59.0.3071.86)
  (Driver info: chromedriver=2.28.455517 (2c6d2707d8ea850c862f04ac066724273981e88f),platform=Mac OS X 10.12.4 x86_64)
    at WebDriverError (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/error.js:27:5)
    at Object.checkLegacyResponse (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/error.js:517:15)
    at parseHttpResponse (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/http.js:509:13)
    at doSend.then.response (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/http.js:441:30)
    at process._tickCallback (internal/process/next_tick.js:109:7)
From: Task: WebDriver.executeScript()
    at thenableWebDriverProxy.schedule (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/webdriver.js:816:17)
    at thenableWebDriverProxy.executeScript (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/webdriver.js:887:16)
    at Object.<anonymous> (/Users/kapaga/Desktop/Node/webdriver.js:20:13)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:393:7)

Can you please help me out here? I am new to Javascriot world ( & to be precise to the Coding world, as I am a QA) so it's highely likely that I might be missing out something very silly here.

Thanks

[mahuntington]

hey @kagarwal29. Node doesn't have a window, as far as I know, since it's purely server-side. Sounds like you're trying to run your tests in node, but in reality what you want to do is run browser tests. I have a walkthrough of something similar here

[kagarwal29]

Thanks @mahuntington for pointing me to you project. I imported the project & it works on my local. :) Your code is proving to be of great help to me.

[mahuntington]

great! @kagarwal29

[kagarwal29]

@mahuntington Can you please look into my query in following post ? Your response would be very helpful.

https://github.com/gotwarlost/istanbul-middleware/issues/45#issuecomment-308631620

[rsoman]

@attodorov , @mahuntington , @gotwarlost Great thread indeed! I got it working to the point of sending data successfully to node express server. However, when trying to view or download coverage report, I am getting RangeError in JSON.stringify(). Error stack is as follows:

RangeError: Maximum call stack size exceeded
    at Object.stringify (native)
    at MemoryStore.Store.setObject (D:\HTML5 Unit Testing\Functional Coverage\node_modules\istanbul-middleware\node_modules\istanbul\lib\store\index.js:116:35)
    at D:\HTML5 Unit Testing\Functional Coverage\node_modules\istanbul-middleware\node_modules\istanbul\lib\collector.js:72:23
    at Array.forEach (native)
    at Object.Collector.add (D:\HTML5 Unit Testing\Functional Coverage\node_modules\istanbul-middleware\node_modules\istanbul\lib\collector.js:67:31)
    at Object.render (D:\HTML5 Unit Testing\Functional Coverage\node_modules\istanbul-middleware\lib\core.js:172:15)
    at D:\HTML5 Unit Testing\Functional Coverage\node_modules\istanbul-middleware\lib\handlers.js:47:14
    at Layer.handle [as handle_request] (D:\HTML5 Unit Testing\Functional Coverage\node_modules\express\lib\router\layer.js:95:5)
    at next (D:\HTML5 Unit Testing\Functional Coverage\node_modules\express\lib\router\route.js:137:13)
    at Route.dispatch (D:\HTML5 Unit Testing\Functional Coverage\node_modules\express\lib\router\route.js:112:3)

Ironically, when I am calling stringify on window.coverage object using executeScript() in selenium test class, it works without an issue.

I feel like I am missing something obvious and would appreciate some guidance.

Thanks in advance.

[kagarwal29]

@gotwarlost I have a quick query.How do I un-minify the JS present in the report? I want the JS present in the report to be in readable format.Currently it all comes in minified format.

I have tried passing debug:true,embedSource:true,noCompact:true to Instrumenter constructor but it is not doing anything.

[reeteshranjan]

@attodorov This is what I was looking for to do my cross-browser testing. I eventually built a tool of my own. If anyone wants to do cross-browser testing on platforms like browserstack, saucelabs and crossbrowsertesting and collect code coverage, you can use https://www.npmjs.com/package/cross-browser-tests-runner.

[ORESoftware]

I wrote up an article on how to do this:

https://medium.com/@the1mills/front-end-javascript-test-coverage-with-istanbul-selenium-4b2be44e3e98

[silkentrance]

@gotwarlost since there is an article about this by @ORESoftware, why not close this for good?

[japborst]

Could we use the same approach to track coverage for electron apps?

[mahuntington]

I don’t think selenium webdriver will work, but the rest should

On Sep 15, 2018, at 4:39 AM, Jelmer Borst notifications@github.com wrote:

Could we use the same approach to track coverage for electron apps?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[pavan447]

I am trying to find out code coverage for our backend application which is written on node js and my automation code written in Java using rest assured library. Both are in two different github repositories. Is there any way to find out code coverage for backend application using rest assured, please suggest.

[alex028502]

I made another example that might come in handy for somebody. It also merges the selenium code coverage with the unit test coverage.

https://github.com/alex028502/istanbulseleniumexample

[samsonbarukula]

@attodorov This is what I was looking for to do my cross-browser testing. I eventually built a tool of my own. If anyone wants to do cross-browser testing on platforms like browserstack, saucelabs and crossbrowsertesting and collect code coverage, you can use https://www.npmjs.com/package/cross-browser-tests-runner.

Hi RateeshRajan @rateeshrajan , Can you please share a repo in which you have used this cross-browser-tests-runner working with mocha pls . thank you.

[samsonbarukula]

@reeteshranjan , Can you please share a repo in which you have used this cross-browser-tests-runner working with mocha pls . thank you.

[samsonbarukula]

Or can you please share the steps in detail to make it work. Thanks in advance.

[reeteshranjan]

@samsonbarukula only jasmine v1 is supported at this time

[ilovepumpkin]

Hi, guys, I am able to get the coverage report based on the above steps, but I have one question - how do you make the report containing all the javascript files instead of only the loaded javascript files?

For example, I have three javascript files: a.js, b.js, c.js. And my selenium test only have tests for a.js and b.js, now in my coverage report, I only a.js and b.js, no c.js. What should I do to show c.js (all coverage are 0%) in my coverage report?

At beginning I thought this is caused by webpack build all my code in several truck files. So I updated my webpack configuration so only one single bundle.js is generated now. But I see see the same.

I am using babel-plugin-istanbul to instrument my code. And I also tried to add "all":true in plugin options, but it does not work.

Anyone has any idea about this issue?

[silkentrance]

@ilovepumpkin this project seems to be dead. there is istanbuljs which is actively maintained. see https://github.com/istanbuljs

[ilovepumpkin]

@silkentrance , thank you! I will raise my question in https://github.com/istanbuljs

[Mohammed786836]

Hey team! i have wirtten my code in java in java eclipse classic now i have to integrate it with istanbul the codes are test codes and all are written in java. can you help me what i should do to integrate with it and find out the code coverage...? Thanks....