search

databrickslabs / cicd-templates

Manage your Databricks deployments and CI with code.
Other
201 stars 100 forks source link

Run testing coverage on integration tests #61

Open Halpph opened 3 years ago

Halpph commented 3 years ago

Hello, I'm trying to check my tests code coverage using coverage.py, it's easy to do it for the unit tests because they are launched using pytest: pytest --cov=standard_datatamers_lib tests/unit --cov-report=xml:cobertura.xml, but I'm struggling to find a way to do it with the integration tests, because they are launched using dbx: launch --job=standard_datatamers_lib-sample-integration-test-feature --trace. Do you have a way to do that? If not, are you planning to create something in regard to it? Thanks in advance :)

renardeinside commented 3 years ago

Hi @Halpph . Thanks for raising an issue.

This issue is quite an interesting task tbh because it requires the following:

  1. Generation of the cov-report is done on the driver-side during an integration test run
  2. We need somehow to save the report to dbfs (?) or as an MLflow artifact (??)
  3. We need to export this artifact after the job run

Theoretically, it's possible but requires some fiddling around the launch mechanics. I'll try out and ping you in this thread as soon as I get some results.

dead commented 3 years ago

Hi guys! Not sure if @Halpph is still looking for a solution to this, but I was able to create a simple POC code to execute and get the coverage.xml from the integration test run.

renardeinside commented 3 years ago

hi @dead , I must admit I was thinking that the solution was hardly achievable until some changes on the API side, but your solution works perfectly.

Could you please create a separate .md file with instructions and contribute it to the repository as a PR?