running selected jupyter notebooks as part of a test suite

[NicolasRouquette]

I would like to contribute additional test suites for running a selection of jupyter notebooks to verify compatibility w.r.t the Pacti API. This would require running them in order, for example:

case_studies/space_mission/space_mission_power.ipynb case_studies/space_mission/space_mission_navigation.ipynb case_studies/space_mission/space_mission_science.ipynb case_studies/space_mission/space_mission_3viewpoints.ipynb

The first 3 notebooks write contracts to json files that the 4th notebook reads for subsequent analysis.

[iincer]

We can execute the notebooks from the command line. Using this functionality, we could write a python test that runs shell commands to execute the three notebooks in order. When it gets to the fourth, I hope we would get an error status from jupyter if the execution ends abnormally.

[NicolasRouquette]

Running command-line scripts could make the tests fragile. Wouldn't it be preferable to take advantage of the support for writing tests in python?

Would this package help achieve this? https://github.com/nteract/testbook

[iincer]

I agree. To execute a notebook from within a python script we could do [this](https://stackoverflow.com/questions/67811531/how-can-i-execute-a-ipynb-notebook-file-in-a-python-script#:~:text=To%20execute%20the%20cells%20of,the%20code%20using%20exec()%20.&text=Save%20this%20answer.,-Show%20activity%20on). Their most voted answer uses the imports

import nbformat
from nbconvert.preprocessors import ExecutePreprocessor

Then the code executes the notebooks and stores the results in a variable.

[NicolasRouquette]

I tried to write a unit test:

https://github.com/FormalSystems/pacti/blob/space_mission/tests/case_studies/space_mission/test_3viewpoints.py

When I run it with make test, it seems to be hanging:

nfr@nfr-desktop:/opt/local/github.formalsystems/pacti$ make test
pdm run duty test
> Running tests

Is there a way to tell what's going on?

In VSCode, when I run this file, it returns quickly; presumably because it is not doing anything at all.

I realize that for this kind of test, I should run all but the last cells of the notebooks so that it does not save the viewpoint contracts; however, this will make the test depend on the number of cells in the notebook. This is getting awkward. Any suggestions?

[iincer]

This is the command that make test runs

pdm run pytest -c config/pytest.ini -n auto -k "" "tests"

Perhaps pytest has a verbose option to see progress?

[NicolasRouquette]

According to this: https://stackoverflow.com/a/15630454/1009693

We need an empty __init__.py file.

nfr@nfr-desktop:/opt/local/github.formalsystems/pacti$ pdm run pytest
================================================================================================= test session starts ==================================================================================================
platform linux -- Python 3.9.16, pytest-7.2.1, pluggy-1.0.0
Using --randomly-seed=1178118921
rootdir: /opt/local/github.formalsystems/pacti
plugins: anyio-3.6.2, cov-4.0.0, xdist-3.1.0, randomly-3.12.0
collected 26 items

tests/case_studies/space_mission/test_3viewpoints.py

It is stuck... so I stopped it and got this:

=================================================================================================== warnings summary ===================================================================================================
.venv/lib/python3.9/site-packages/jupyter_client/connect.py:20
  /opt/local/github.formalsystems/pacti/.venv/lib/python3.9/site-packages/jupyter_client/connect.py:20: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs
  given by the platformdirs library.  To remove this warning and
  see the appropriate new directories, set the environment variable
  `JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`.
  The use of platformdirs will be the default in `jupyter_core` v6
    from jupyter_core.paths import jupyter_data_dir, jupyter_runtime_dir, secure_write

tests/case_studies/space_mission/test_3viewpoints.py::test_viewpoints
  /opt/local/github.formalsystems/pacti/.venv/lib/python3.9/site-packages/jupyter_core/utils/__init__.py:153: RuntimeWarning: coroutine 'NotebookClient._async_cleanup_kernel' was never awaited
    inner = coro(*args, **kwargs)
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/opt/local/github.formalsystems/pacti/.venv/lib/python3.9/site-packages/jupyter_core/utils/__init__.py:153: KeyboardInterrupt
(to show a full traceback on KeyboardInterrupt use --full-trace)
=========================================================================================== 2 warnings in 288.54s (0:04:48) ============================================================================================

[iincer]

Would that work for us? It seems the __init__ trick is meant to alleviate an issue with unittest. We are using pytest.

Regarding the jupyter notebooks, the fact it works for vscode, but not for make test, is puzzling to me. Can we see what command vscode is running?

[iincer]

I just reread the

presumably because it is not doing anything at all.

If using nbconvert from within python is challenging, it seems our options are to go to the command line (brittle) or to convert the notebook to python (via a shell command)--see #222.

[NicolasRouquette]

Trying to execute notebooks in python seems too advanced and too brittle to be reliable.

Converting the notebooks to python with a git commit hook seems interesting, assuming that jupytext works as advertised:

https://jupytext.readthedocs.io/en/latest/using-pre-commit.html

[NicolasRouquette]

It seems that it is possible to do this, but this will likely require using both nbconvert and jupytext.

https://github.com/FormalSystems/pacti/blob/space_mission/case_studies/space_mission/README.md#syncing-case-study-notebooks-to-python-files-for-testing

[iincer]

(#334 is relevant to this issue.)