tns test <platform> include coverage?

[dtopuzov]

From @NickIliev on June 20, 2016 11:31

From @NickIliev on June 20, 2016 10:56

From @georgeedwards on June 17, 2016 12:46

Feature Request:

Can the tns test <platform> command include coverage reporting as part of it's output? Karma-Coverage seems to be quite a nice way of doing this. To add support for Typescript, adding in remap-istanbul looks to be the most straight forward solution.

Copied from original issue: NativeScript/NativeScript#2333

Copied from original issue: NativeScript/nativescript-cli#1865

Copied from original issue: NativeScript/nativescript-unit-test-runner#12

[dtopuzov]

From @casche on October 14, 2016 15:0

@NickIliev @georgeedwards is anyone working on this or is there an eta ⌛️ ?

I've tried to get this executing with istanbul:

    files: [
      'app/**/*.js',
    ],

    preprocessors: {
      'app/**/!(*spec).js': ['coverage']
    },

    reporters: ['progress', 'coverage'],

But it just generates an empty report. I can't really tell what tns test is really doing to debug it. Or am I just missing something simple ? 😞

[dtopuzov]

From @joeskeen on October 17, 2016 22:12

@casche I was really struggling with this as well, but I just got it working. Could you post your full karma.conf.js file and I can compare it with what I have?

Edit: I got my Karma config file cleaned up and here is the config that is working for me:

(Note that I am just using karma-coverage. I'm trying to do remapping to TS source files but I haven't gotten that working yet.)

module.exports = function(config) {
  config.set({

    // base path that will be used to resolve all patterns (eg. files, exclude)
    basePath: '',

    // frameworks to use
    // available frameworks: https://npmjs.org/browse/keyword/karma-adapter
    frameworks: ['jasmine'],

    // list of files / patterns to load in the browser
    files: [
      'app/**/*.js',
    ],

    // list of files to exclude
    exclude: [
    ],

    // preprocess matching files before serving them to the browser
    // available preprocessors: https://npmjs.org/browse/keyword/karma-preprocessor
    preprocessors: {
      'app/**/!(*spec).js': ['coverage']
    },

    // test results reporter to use
    // possible values: 'dots', 'progress'
    // available reporters: https://npmjs.org/browse/keyword/karma-reporter
    reporters: [
      'progress',
      'spec',
      'junit',
      'coverage'
    ],

    // web server port
    port: 9876,

    // enable / disable colors in the output (reporters and logs)
    colors: true,

    // level of logging
    // possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
    logLevel: config.LOG_DEBUG,

    // enable / disable watching file and executing tests whenever any file changes
    autoWatch: false,

    // start these browsers
    // available browser launchers: https://npmjs.org/browse/keyword/karma-launcher
    browsers: [],

    customLaunchers: {
      android: {
        base: 'NS',
        platform: 'android'
      },
      ios: {
        base: 'NS',
        platform: 'ios'
      },
      ios_simulator: {
        base: 'NS',
        platform: 'ios',
        arguments: ['--emulator']
      }
    },

    coverageReporter: {
      // specify a common output directory
      dir: '.reports/coverage',
      reporters: [
          { type: 'lcov', subdir: '.' },
          { type: 'cobertura', subdir: '.' }
      ],
      includeAllSources: true
    },

    // Continuous Integration mode
    // if true, Karma captures browsers, runs the tests and exits
    singleRun: true
  })
}

The report was successfully generated for me (looks like I have some work to do on coverage ;) ):

One option that made the difference for me was coverageReporter.includeAllSources. Without it, I just got an empty report.

Update: after looking it over, the coverage results I'm getting out of this are not right. It seems that the test code isn't causing any increase of coverage, as the only lines reporting as 'covered' are the lines that the classes are declared on, but none of the functions I am calling in my test show any coverage.

[dtopuzov]

From @casche on October 20, 2016 19:11

@joeskeen thanks! 👍 I added those lines you suggested and now I get the same report as you. That is, files are listed but coverage is not increasing.

[dtopuzov]

From @phattranky on May 2, 2017 4:56

My Karma config:

Got same issue, the coverage reporter always empty. If add the coverageReporter.includeAllSources, it listing all file, but the report values not changes, and always 0 like @casche

  config.set({

    // base path that will be used to resolve all patterns (eg. files, exclude)
    basePath: '',

    // frameworks to use
    // available frameworks: https://npmjs.org/browse/keyword/karma-adapter
    frameworks: ['jasmine'],

    // plugins: ['karma-jasmine', 'karma-nativescript-launcher', 'karma-typescript-preprocessor', 'karma-typescript', 'karma-coverage', 'karma-remap-coverage'],

    // list of files / patterns to load in the browser
    files: [
      'node_modules/nativescript-angular/zone-js/dist/zone-nativescript.js',
      'node_modules/zone.js/dist/long-stack-trace-zone.js',
      'node_modules/zone.js/dist/proxy.js',
      'node_modules/zone.js/dist/sync-test.js',
      'node_modules/zone.js/dist/jasmine-patch.js',
      'node_modules/zone.js/dist/async-test.js',
      'node_modules/zone.js/dist/fake-async-test.js',
      'app/**/*.ts'
    ],

    // list of files to exclude
    exclude: [
    ],

    // preprocess matching files before serving them to the browser
    // available preprocessors: https://npmjs.org/browse/keyword/karma-preprocessor
    preprocessors: {
       'app/**/*.ts': ['typescript'],
       'app/**/!(*spec).ts': ['typescript', 'coverage']
    },

    typescriptPreprocessor: {
      // options passed to the typescript compiler 
      options: {
        project: './tsconfig.json'
      },
      // transforming the filenames 
      transformPath: function(path) {
        return path.replace(/\.ts$/, '.js');
      }
    },

    // test results reporter to use
    // possible values: 'dots', 'progress'
    // available reporters: https://npmjs.org/browse/keyword/karma-reporter
    reporters: ['progress', 'coverage', 'remap-coverage'],

    // web server port
    port: 9876,

    // enable / disable colors in the output (reporters and logs)
    colors: true,

    // level of logging
    // possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
    logLevel: config.LOG_INFO,

    // enable / disable watching file and executing tests whenever any file changes
    autoWatch: true,

    // start these browsers
    // available browser launchers: https://npmjs.org/browse/keyword/karma-launcher
    browsers: [],

    customLaunchers: {
      android: {
        base: 'NS',
        platform: 'android'
      },
      ios: {
        base: 'NS',
        platform: 'ios'
      },
      ios_simulator: {
        base: 'NS',
        platform: 'ios',
        arguments: ['--emulator']
      }
    },

    /*
    coverageReporter: {
      // specify a common output directory
      dir: './coverage',
      reporters: [
          { type: 'lcov', subdir: '.' }
      ],
      includeAllSources: true
    },*/
    coverageReporter: {
      type: 'in-memory'
    },

    remapCoverageReporter: {
      'text-summary': null,
      html: './coverage/html',
      cobertura: './coverage/cobertura.xml'
    },

    // Continuous Integration mode
    // if true, Karma captures browsers, runs the tests and exits
    singleRun: true
  })
}``

[geertjanb]

I have the same issue as @dtopuzov

any update on this matter?

[pkoleva]

Hi @geertjanb

We currently work on making livesync and debug better, but we've summarized what we need to do with the tns test command, so we can plan it for a future release. We'll make sure to update this issue when we plan it for a specific milestone.

[Guatom]

@dtopuzov: could you make it work somehow?

[mcrvaz]

any updates on this?

[Guatom]

I gave up on this and I used node-ts + Jasmine, which worked OK. I have to say that I could see the magic behind running tests in the phone: in the node-ts + Jasmine approach I had to mock (proxyquire) every single import; tests using NativeScript framework worked just out of the box.

[SmailHammour]

@pkoleva Any news on this?

[pnahtanoj]

Coverage would be very helpful - any updates for this?

[rosen-vladimirov]

Hey all,

It's been some time we've not worked on our unit testing story. Most of you, who have tried it, know that it has some issues, but still it can be used to test your application. Currently we are working on some other tasks and we may not be able to implement this feature in the next few months. However, we have created a POC that you can use and even improve it and send us a PR. Before deep diving in the details described below, please read carefully this article and especially the Run the tests part. It describes what happens when tns test command is executed.

Karma coverage - enabling the feature

So back to the coverage feature.

NOTE: Information below is for a project that already has unit tests, i.e. you have executed tns create myApp && cd myApp && tns test init.

Karma allows you to have coverage information by adding the karma-coverage plugin as devDependency of your project: https://karma-runner.github.io/0.8/config/coverage.html So you have to install it with:

npm i --save-dev karma-coverage

As described in the article, you need to add some code in your karma.config.js. The following configuration should work for your NativeScript application.

files: [
  'app/**/*.js'
],

preprocessors = {
  'app/**/*.js': ['coverage']
},

reporters = ['progress', 'coverage'],

coverageReporter: {
  type: 'html',
  dir: 'coverage/',
  includeAllSources: true
},

At this point you have everything setup, so you can try tns test <platform> --justlaunch. (--justlaunch will ensure the command exits after the execution of unit test finishes). After that you can check the <project dir>/coverage directory. You'll notice that there is some information there, but the generated coverage report is not correct.

Discover the issue or why the code coverage is not correct

The actual issue here is with the execution of the tests. In the previous point we have added a preprocessor - coverage. What does it do? Well, it modifies the files that will be served by Karma in a specific way. This specific way in our case is adding of lines inside all files that are used by the karma-coverage to detect how many times each line is called. This is done through istanbul package. It is important to notice that the files are modified in-memory, their content is not written anywhere on your hard disk. So, we know Karma is working fine and the files are preprocessed, but why the coverage information is not correct on our side? Let's get back to the way NativeScript CLI executes the unit tests. When tns test <platform> command is executed CLI does the following:

1. The CLI starts a Karma server on the development machine.
2. The CLI prepares, builds and deploys your project, if not already deployed. If already deployed, the CLI synchronizes changes to the application package.
3. The CLI embeds the NativeScript unit test runner and your host network and Karma configuration in the deployed package.
4. The CLI launches the main module of the NativeScript unit test runner instead of launching the main module of your app.
5. The NativeScript unit test runner uses the embedded network configuration to try to connect to the Karma server on the development machine.
6. When the connection between the NativeScript unit test runner and the Karma server is established, the test runner begins the execution of the unit tests.
7. When the execution completes, the NativeScript unit test runner reports the results to the Karma server.
8. The Karma server reports the results on the command line.

There's some additional things that happen on step 6. As you can see here (this is the code that is actually running on the device) once a connection is established (i.e. after NativeScript CLI had already uploaded the application and all of the files on device), the application on device gets a file called context.json from the Karma server. This file contains information about the current test run including which files should be included in it. These are all the files that match our files array in the karma.config.js and some specific files required for testing, like mocha and chai files in case you've selected to use mocha framework for testing. As you can see, the application gets the list of files and executes some logic for each of them. In case the file is part of the application, the application does nothing. But in case this is a file from "node_modules" (as the paths that are written in the context.json are from the local execution of the Karma and they contain node_modules, which does not exist on device), the application calls the Karma server to get the content of the file. Getting back to the way CLI and Karma work, we start to see what might be the issue - CLI had uploaded the original files of the application. Meanwhile the Karma server had preprocessed the local files (the same ones that CLI had already uploaded), but keeps them in memory. Once the application on device starts executing the tests, it does not call Karma server to get the contents of the project files, it calls it only for the files that are not already uploaded by CLI. So the preprocessed files are never taken to the device and so the coverage information cannot be correct.

Fixing the coverage issue

So how to fix this behavior. The easiest way we've found is to overwrite the project files with the content that Karma server will give us. This will allow us to include other preprocessores in case we need. And here's a branch that implements exactly this logic (this is our POC) of this feature: https://github.com/NativeScript/nativescript-unit-test-runner/tree/vladimirov/coverage-poc (check the latest commit) This way the files on the device will be overwritten with the preprocessed content and the coverage information that Karma will create inside the local project will be correct.

(Potential) Issues with the suggested implementation

1. Using platform specific files will not work - the Karma server will use the files from the local app directory, which may have main-view-model.android.js and main-view-model.ios.js instead of just a single file. Once the application on device starts, it will receive the list of files, including the two platform specific files. After that it will ask Karma to get the content of these two files and will write them in the application with the same names - main-view-model.android.js and main-view-model.ios.js. Neither file will require these files, as CLI had already uploaded a file called main-view-model.js. There are various ways to fix this behavior, for example:

   • implement a logic once the application gets the file list from Karma and parse the file names based on the platform of the device (you can use isAndroid/isIOS methods. After that skip the files that are not for this platform and replace the name of the files that are for current platform, i.e. filter the main-view-model.ios.js and set the name of the main-view-model.android.js to main-view-model.js.

2. Overwritten files may remain in the application after executing tns test <platform> and tns run <platform> after that as CLI may not transfer all files. Workaround: uninstall the app from device.

Notes

The nativescript-unit-test-runner is written in TypeScript, so when you apply changes in it, you have to execute the following steps to get them transpiled:

git clone https://github.com/NativeScript/nativescript-unit-test-runner.git
cd nativescript-unit-test-runner
git checkout vladimirov/coverage-poc
npm i
# apply your change in the code

grunt pack 

NOTE: The grunt pack command will produce some warnings, but don't worry about them, it will generate a .tgz file that you can install in your application instead of the official version of nativescript-unit-test-runner.

So that's it. You should be able to get the coverage information you need. In case anyone is interested in finishing this POC, just drop a line. We'll try to help you with the implementation and any issues you face.

[rosen-vladimirov]

Another idea for handling the platform specific files - in case we start the Karma server from the <project dir>/platforms/<platform>/.../app/ directory, it will work with already prepared files and there will be no need to apply additional logic in the app - just pull the files from the karma server.

[veerendervoskula]

@rosen-vladimirov

I did a POC based on the steps you suggested,please find here https://github.com/veerendervoskula/Nativescript-Unit-Testing-Code-Coverage.

Thanks now am able to generate coverage report(snapshot taken from the generated coverage report)

But generated coverage report shows 100% Branch coverage(highlighted in red).

Is there anything am missing from my end to add? Any help would be appreciated.

[adrian-niculescu]

Hi! What is the progress on this? Coverage is important, especially for writing reliable NativeScript plugins.

[rosen-vladimirov]

Hey @adrian-niculescu , We have not worked on this feature as we are preparing the next major version of NativeScript 6.0.0. We fully understand the high importance of this feature and we'll prioritize it after 6.0.0 release.

[anshuranjan]

HI team, I am using Nativescript code sharing project with angular. How we can produce the code coverage for ios device? if team have some document please provide.

[Fatme]

Hey @anshuranjan,

We do not support code coverage officially. However, the project could be configured with code coverage using the following steps:

Install the required packages:

1. npm i istanbul-instrumenter-loader
2. npm i --save-dev karma-coverage
3. npm i --save-dev karma-coverage-istanbul-reporter

NOTE: It is required to use tns-core-modules@next in case you are with @angular/cli@8.3.0 or greater.

karma.conf.js

1. Add the required plugins:

   plugins: [
     'karma-jasmine',
     'karma-coverage-istanbul-reporter',
     'karma-coverage',
     'karma-webpack',
     'karma-nativescript-launcher'
   ]

2. Update reporters:

   reporters: ['progress', 'coverage-istanbul']

3. Setup coverage-istanbul-reporter :

   coverageIstanbulReporter: {
     dir: require('path').join(__dirname, 'coverage'),
     reports: ['html', 'lcovonly', 'text-summary'],
     fixWebpackSourcePaths: true
   }

4. Add coverage preprocessor:

     options.preprocessors[file].push('coverage');

5. Add istanbul-instrumenter-loader:

   options.webpack.module.rules.push(
       {
         test: /\.ts$/,
         use: { loader: 'istanbul-instrumenter-loader', options: { esModules: true } },
         enforce: 'post'
       }
   );

Include test files into typescript compilation from tsconfig.tns.json:

   "files": [
      "src/main.tns.ts",
      "src/tests/example.ts"
   ]

Limitations:

• We support test files only from src/tests folder. Meaning, all test files should be moved/created to that folder in order to be executed.

I prepared a sample project demonstrating all the required changes - https://github.com/Fatme/code-coverage-shared-app

[fasterinnerlooper]

@Fatme How did you get the project to build with the .spec files in there? It's not something that NativeScript provides by default, and being able to run tests from the command line using ng test would be a great help.

[NidheeshMv]

Hi Team, i working on writing test cases for Nativescript. Am getting below error.

TypeError: frame_1.Frame.topmost is not a function

Can you guys please help on this.

[adrian-niculescu]

Hi! Any news about this?

[VoBichTo]

Hi! Any support with generate coverage for typescript file ?

[adrian-niculescu]

Hi! Any news about this? Asking from the point of view of NativeScript plugin developers where split files .ios.ts / .android.ts are prevalent.