[Code Coverage] Node Tests - Generate reports for live tests

HarshaNalluru commented 3 years ago

TO DO

The goal is to have the .nycrc config for all the following packages that are consistent.
This includes consistency for the integration-test:node command for all the packages as well.
With the proper .nycrc setup, each package should generate a cobertura-coverage.xml file which would be picked up by the DevOps to publish the report.
Example report below - https://github.com/Azure/azure-sdk-for-js/issues/12935#issuecomment-747345414
Example PR that fixes code coverage for storage-blob https://github.com/Azure/azure-sdk-for-js/pull/12903

Packages

[x] /sdk/template/ - #12968
[x] /sdk/appconfiguration/ - #12968
[x] /sdk/communication/ - #12968
[x] /sdk/eventhub/ (event-hubs and eventhubs-checkpointstore-blob)
[x] /sdk/servicebus/
[x] /sdk/keyvault/ - #12968
[x] /sdk/storage/
- [x] /sdk/storage/storage-blob - #12903
- [x] /sdk/storage/storage-file-datalake - #12968
- [x] /sdk/storage/storage-file-share
- [x] /sdk/storage/storage-queue
[ ] /sdk/tables/ - #12976
[x] /sdk/textanalytics/ - #12968
[x] /sdk/formrecognizer/
[x] /sdk/metricsadvisor/
[x] /sdk/search/ - #12968
[ ] .nycrc -> .nycrc.json so that they are formatted better with prettier
[ ] Investigate if the "extends" keyword is supported to allow creating a common config that would be extended by the configs in each of the SDKs.

Note

ARM packages will be ignored, the ones we have the live test pipelines will be the priority
We only do this for the live test pipeline and not the unit-tests because instrumenting with nyc adds an overhead for the unit-tests which results in being slower. The overhead would be negligible for the live tests as they tend to run for longer periods.

Packages without live tests

[ ] sdk/core (many packages)
[ ] /sdk/identity/
[ ] /sdk/test-utils/
[ ] As they don't have the live test pipelines, figure out a way to show the code coverage for the unit-test pipeline instead, by potentially adding nyc to the unit-test commands.
[ ] Or use rush or our yml trickery to show code coverage reports for unit-tests without updating unit-test commands.

Others

[ ] SDKs such as /sdk/cognitiveservices/, sdk/anomalydetector either not have the test folder or missing the live test pipeline or not have an integration-test:node command. Discuss with the team and come with a plan for the rest of the SDKs. To begin with, one good way to prioritize is to pick the packages with the live test pipeline but with an unorganized setup.

cc @KarishmaGhiya @ramya-rao-a @AlexGhiondea

HarshaNalluru commented 3 years ago

Example

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=656757&view=codecoverage-tab

KarishmaGhiya commented 3 years ago

@HarshaNalluru @ramya-rao-a @AlexGhiondea List of packages that do not support code coverage in MQ -

[ ] synapse - It is autogenerated code with no test folder. Source code and tests will be added in future

List of packages with no live test pipeline -

[ ] /sdk/eventgrid/
[ ] /sdk/digitaltwins
[ ] /sdk/cosmosdb
[ ] /sdk/batch/

ramya-rao-a commented 3 years ago

@HarshaNalluru, Lets log individual issues for the general, package agnostic follow up items in this issue and then close this one

ramya-rao-a commented 3 years ago

@KarishmaGhiya At this point, please log separate issues for packages that have live tests, but no code coverage and then close this issue

ramya-rao-a commented 3 years ago

SDKs such as /sdk/cognitiveservices/, sdk/anomalydetector either not have the test folder or missing the live test pipeline or not have an integration-test:node command. Discuss with the team and come with a plan for the rest of the SDKs. T

The ones under sdk/cognitiveservices are Track 1 packages, they do not have tests and so we can keep them out of our current list

Any package that is not a core package and is on our rush pipeline should have a live test + code coverage. If not, then please log an issue for them

KarishmaGhiya commented 3 years ago

@ramya-rao-a sdks that fall under the category of having recent live tests and no code coverage -

sdk/eventgrid : https://dev.azure.com/azure-sdk/internal/_build?definitionId=2321&_a=summary Issue filed - https://github.com/Azure/azure-sdk-for-js/issues/13585
sdk/digitaltwins : https://dev.azure.com/azure-sdk/internal/_build?definitionId=2313 Issue filed - https://github.com/Azure/azure-sdk-for-js/issues/13586

Azure / azure-sdk-for-js