Support pytest-cov with ipytest

[westurner]

%%ipytest --cov= --cov-report=term-missing
%%ipytest --cov=. --cov-report=term-missing
%%ipytest -v --cov=. --cov-report=term-missing

• https://pytest-cov.readthedocs.io/en/latest/reporting.html
  • --cov-report=term-missing text/plain report of which lines or expressions lack coverage
  • --cov-report=html text/html report of '
  • [ ] ipytest: _repr_html_() or _repr_mimebundle_?
• https://github.com/chmp/ipytest/issues/71
  • Could there be an %%ipytest__all or similar to run all the tests at the end?
• https://github.com/pytest-dev/pytest-cov/issues/202

[westurner]

Current errors:

%ipytest -v --cov= --report=term-missing
[...]

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:507: CoverageWarning: Module  was never imported. (module-not-imported)
  self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
/usr/local/lib/python3.8/dist-packages/coverage/control.py:858: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")
/usr/local/lib/python3.8/dist-packages/pytest_cov/plugin.py:311: CovReportWarning: Failed to generate report: No data to report.

  warnings.warn(CovReportWarning(message))

%ipytest -v --cov=. --cov-report=term-missing
[...]

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name                                    Stmts   Miss  Cover   Missing
---------------------------------------------------------------------
t_ddc434a77aa64527808c1ce068113911.py       0      0   100%
---------------------------------------------------------------------
TOTAL                                       0      0   100%

===================================== short test summary info ======================================
FAILED t_ddc434a77aa64527808c1ce068113911.py::test_information_metric_626__001[1-None] - assert NotImplemented == None
FAILED t_ddc434a77aa64527808c1ce068113911.py::test_information_metric_626__001[2-None] - assert NotImplemented == None
FAILED t_ddc434a77aa64527808c1ce068113911.py::test_information_metric_626__001[3-None] - assert NotImplemented == None
FAILED t_ddc434a77aa64527808c1ce068113911.py::test_information_metric_626__001[10-None] - assert NotImplemented == None
=================================== 4 failed, 1 passed in 0.16s ====================================

/usr/local/lib/python3.8/dist-packages/coverage/control.py:858: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")

[chmp]

Hi @westurner,

My feeling is that it could be hard or even impossible to support coverage from notebooks, due to the way notebooks operate. In any case, I will have to a take a closer look on how pytest-cov works in the coming days.

[westurner]

Probably easy enough; though maybe impractically verbose for many use cases

The --cov= parameter somehow gets set to t_ddc434a77aa64527808c1ce068113911.py

[chmp]

The weird file name comes from the way ipytest works: it creates a random temporary module and registeres the current interpreter session as this module. In your case the random module was named t_ddc434a77aa64527808c1ce068113911. See here for details.

The problem is: the file is empty and there is no (easy) way to fill it with your source code. Jupyter per its design does not offer a way to get the source code.

[westurner]

filename = 't_uuidxyz.py'
modname, ext = os.path.splitext(filename)

%%ipytest --cov-report=term-missing --cov=$modname 

• IIUC, here, cell is the cell text?: https://github.com/chmp/ipytest/blob/f1b8f698f289872598cb0a7d19969100297c05e8/ipytest/_impl.py#L131
• https://ipython.readthedocs.io/en/stable/config/custommagics.html
• https://github.com/ipython/ipython/blob/3339fad91de3e3da1f02b0b209afef20f021ca33/IPython/core/tests/test_magic.py#L1006-L1024
• https://github.com/pytest-dev/pytest-cov/blob/2ea944212d62841cbd95ce49ef4586c8450435fd/src/pytest_cov/plugin.py#L93-L96
  • cov_source
  • https://github.com/pytest-dev/pytest-cov/blob/2ea944212d62841cbd95ce49ef4586c8450435fd/src/pytest_cov/plugin.py#L129-L136

[chmp]

Your comment is a bit sparse. Could you maybe add one or two sentences of what you are asking? As is, I don't get it, TBH.

IIUC, here, cell is the cell text?

Yes, but only of the test cell. Any other cell can define arbitrary Python code. These cells are not known to ipytest.

[westurner]

Is it necessary to have the source of the other cells if we have the locals() dict?

From https://ipython.readthedocs.io/en/stable/config/custommagics.html#accessing-user-namespace-and-local-scope :

Accessing user namespace and local scope¶ When creating line magics, you may need to access surrounding scope to get user variables (e.g when called inside functions). IPython provides the @needs_local_scope decorator that can be imported from IPython.core.magics. When decorated with @needs_local_scope a magic will be passed local_ns as an argument. As a convenience @needs_local_scope can also be applied to cell magics even if cell magics cannot appear at local scope context.

[chmp]

After looking a bit how coverage.py works, I'm pretty sure it cannot be supported in ipytest. I will close this issue for now.

[chmp]

The new coverage version is using settrace and seems work at least in principle inside notebooks. See here. However, I am not yet sure how to properly use pytest-cov.

Update: Draft PR

[chmp]

Implemented in ipytest==0.14.1 via the ipytest.cov coveragepy plugin

[westurner]

Thanks. A Pull Request Review: https://github.com/chmp/ipytest/pull/110#pullrequestreview-2004658898

[chmp]

Thanks for the PR review :)