This is meant to be a powerful tool for debugging, allowing us to inspect exactly what happened following a test failure.
Unfortunately, it currently doesn't work exactly as expected. For cells marked with raises-exception (i.e. cells performing checks) we get the desired behaviour; the notebook goes on executing and the errors in those cells are saved in its outputs. If an error is raised in one of the other cells, though, this won't work. In fact, it will look like that cell executed without any output.
One possible patch for that would be using the allow_errors flag, which resembles adding the raises-exception tag everywhere, meaning that should an error be raised in any cell, the execution will continue. Although this would ensure the outputs are saved to the original notebook, it is not ideal, since it can drown us in a chain of errors, something that would not happen when executing the notebook manually (execution stops when a cell breaks).
Goal
Investigate an alternative solution for saving all cell outputs, but still interrupting the notebook execution when an error is raised in a cell not marked with raises-exception.
After executing each test notebook we attempt to write its final state back to the original file: https://github.com/canonical/charmed-kubeflow-uats/blob/53e0389f070a7d26e97e9eb81473693e526275dd/tests/test_notebooks.py#L38-L40
This is meant to be a powerful tool for debugging, allowing us to inspect exactly what happened following a test failure.
Unfortunately, it currently doesn't work exactly as expected. For cells marked with
raises-exception
(i.e. cells performing checks) we get the desired behaviour; the notebook goes on executing and the errors in those cells are saved in its outputs. If an error is raised in one of the other cells, though, this won't work. In fact, it will look like that cell executed without any output.One possible patch for that would be using the
allow_errors
flag, which resembles adding theraises-exception
tag everywhere, meaning that should an error be raised in any cell, the execution will continue. Although this would ensure the outputs are saved to the original notebook, it is not ideal, since it can drown us in a chain of errors, something that would not happen when executing the notebook manually (execution stops when a cell breaks).Goal
Investigate an alternative solution for saving all cell outputs, but still interrupting the notebook execution when an error is raised in a cell not marked with
raises-exception
.