search

kiwix / kiwix-js

Fully portable & lightweight ZIM reader in Javascript
https://www.kiwix.org/
GNU General Public License v3.0
299 stars 124 forks source link

Introduce code coverage with Codecov #528

Open kelson42 opened 5 years ago

kelson42 commented 5 years ago

Code coverage solutions allow to clearly measure and visualise the parts of the code which are involved by the automated tests. It is traditionally one part of the Continuous Integration (CI) system. Like the other tools of the CI, its goal is to control new development, bring knowledge and securities to increase the overall code quality.

We have just started an effort within Kiwix & openZIM to generalise automated testing using Travis-CI & code coverage using codecov.io. Codecov provides a third-party app which is really easy to integrate with Github & Travis-CI. We have achieve so far to integrate it to the MWoffliner, Kiwix-Android and Zimfarm repositories. Here have a look how it looks like for MWoffliner for example: https://codecov.io/gh/openzim/mwoffliner.

I know that both of you are sensitive to quality and - this is really fortunate - you already have introduced automated testing with Travis-CI. Adding Codecov would allow you to more precisely evaluate your automated testing strategy and evolution over time. That is why I write this ticket and would recommend to make this new step. I think this should be relatively easy, taking example on MWoffliner might be enough to achieve this within 1-2 hours.

mossroy commented 5 years ago

Very interesting. I'm trying to activate it.

I think I've set it up for kiwix-js, but for now nothing appears on the dashboard. I created a dummy pull request to trigger a Travis build, but it did not help. Maybe I missed something, or it needs some time to sync with github (and it might work on next PR or next push)

mossroy commented 5 years ago

I've added a step in .travis.yml to run codecov in https://github.com/kiwix/kiwix-js/pull/533. It runs well but does not find any coverage report.

I don't know if it's possible to create one with the way we run Unit and UI tests. The javascript does not run inside Node (where one can easily plug some code coverage analysis, as in the many other supported languages : https://docs.codecov.io/docs/supported-languages) : it is run inside actual browsers, started inside SauceConnect, and the scenario is run by nightwatchjs. Based on https://nightwatchjs.org/guide/#code-coverage , nightwatchjs does not provide any code coverage for now. Nightwatchjs is based on Selenium so maybe it's possible (see https://stackoverflow.com/questions/9495625/selenium-is-there-any-js-javascript-code-coverage-tool-which-we-can-integrate), but it's certainly not a 1-2 hour work.

kelson42 commented 5 years ago

@mossroy OK, that is sad. But that also means that Kiwix JS does not have any unit tests? @julianharty Any info which could be helpful here?

mossroy commented 5 years ago

kiwix-js has both unit and UI tests. Both are currently run with the same infrastructure.

Unit tests are focused on the "backend" that reads the ZIM file (which we will hopefully be able to replace with an emscripten version of kiwix-lib, in the future). But they are run inside a browser with https://qunitjs.com/ (you can run them with https://kiwix.github.io/kiwix-js/tests.html for example) I suppose these unit tests might be run in a nodejs environment, but we would need to add some glue to replace the https://developer.mozilla.org/fr/docs/Web/API/File API with an equivalent one from NodeJS. It might also be possible to make QUnit generate some code coverage report that could be read by Codecov, but it's probably not easy.

UI tests can only be run on actual browsers, obviously.

kelson42 commented 5 years ago

@mossroy Thank you for all these explanations. I agree here it does not sound straight forward anymore. A bit of engineering work would be needed to get this code coverage numbers. Luckily this is not a urgent things.

Bam92 commented 4 years ago

Any help needed here?

mossroy commented 4 years ago

@Bam92 : this seems complicated to implement and I'm not sure it is the highest priority. Regarding the UI tests, I don't see an easy way to plug them on CodeCov. And they're probably the most interesting ones. The unit tests might be plugged with CodeCov (IMHO, #554 should be implemented first, so that to make them run on nodeJS), but they "only" test the ZIM backend. I would prefer to work on replacing this backend with libzim (see #116, that has been split in many other github issues)

Bam92 commented 4 years ago

@mossroy Good. I try to guess what you mean.