canonical / charmed-kubeflow-uats

Automated UATs for Charmed Kubeflow
Apache License 2.0
6 stars 2 forks source link

Write back notebook output #5

Open phoevos opened 1 year ago

phoevos commented 1 year ago

After executing each test notebook we attempt to write its final state back to the original file: https://github.com/canonical/charmed-kubeflow-uats/blob/53e0389f070a7d26e97e9eb81473693e526275dd/tests/test_notebooks.py#L38-L40

This is meant to be a powerful tool for debugging, allowing us to inspect exactly what happened following a test failure.

Unfortunately, it currently doesn't work exactly as expected. For cells marked with raises-exception (i.e. cells performing checks) we get the desired behaviour; the notebook goes on executing and the errors in those cells are saved in its outputs. If an error is raised in one of the other cells, though, this won't work. In fact, it will look like that cell executed without any output.

One possible patch for that would be using the allow_errors flag, which resembles adding the raises-exception tag everywhere, meaning that should an error be raised in any cell, the execution will continue. Although this would ensure the outputs are saved to the original notebook, it is not ideal, since it can drown us in a chain of errors, something that would not happen when executing the notebook manually (execution stops when a cell breaks).

Goal

Investigate an alternative solution for saving all cell outputs, but still interrupting the notebook execution when an error is raised in a cell not marked with raises-exception.

phoevos commented 12 months ago

Closing and reopening to trigger Jira automation.