I suggest we use --nbval-lax in the make test-notebooks command, which only checks that cells do not fail, not that they generate a specific output.
Experience suggests that insisting on exact matches, even with sanitizing regexps, leads to many false positives and requires lots of minor tweaks to avoid failures.
By default, notebooks are thus run as smoke tests, and when desired, #NBVAL_CHECK_OUTPUT can be inserted at the top of a cell to force an exact comparison with the expected output. This output should be robust to minor library updates, ie. native python objects with as little formatting as possible.
Description
I suggest we use
--nbval-laxin themake test-notebookscommand, which only checks that cells do not fail, not that they generate a specific output.Experience suggests that insisting on exact matches, even with sanitizing regexps, leads to many false positives and requires lots of minor tweaks to avoid failures.
By default, notebooks are thus run as smoke tests, and when desired,
#NBVAL_CHECK_OUTPUTcan be inserted at the top of a cell to force an exact comparison with the expected output. This output should be robust to minor library updates, ie. native python objects with as little formatting as possible.