search

jpmorganchase / nbcelltests

Cell-by-cell testing for production Jupyter notebooks in JupyterLab
Apache License 2.0
83 stars 21 forks source link

Measure actual coverage, not "cell coverage" #156

Open ceball opened 4 years ago

ceball commented 4 years ago

Current cell coverage (num cells tested)/(num cells) is a proxy for actual coverage.

nbval includes a coverage measure - should investigate that.

Current coverage is relatively easy to fake, e.g.

if False:
    %cell

Something related to "what actually ran?" (e.g. detection that cell source was submitted to kernel) might be better than a static check.

timkpaine commented 3 years ago

@vidartf is this something you can help with? I see you've done it for nbval https://github.com/computationalmodelling/nbval/pull/134

since we have the %cell injection, I would think all we need to do is maintain a map of nb line # to test line #, inject coverage.py into the test itself, then match back the covered line using the mapping.

vidartf commented 3 years ago

@timkpaine The nbval coverage is all about getting coverage data in python modules, from the execution of the notebook. It doesn't currently attempt to measure coverage in the notebook. The problem is that coverage will not be available in the same scope as the coverage gathering was started, i.e. the notebook global scope. There might be a clever work-around somehow, but I haven't dived deep enough into it to find anything.