Foundations health checks fail to identify failing code

[brian-rose]

We recently discovered an example of code in a Foundations notebook that is failing (#343). We discovered this by accident while perusing the published pages. It's not clear how long that code has been failing.

This is basically a bug in our nightly health checks. We currently use allow_errors: True in the Foundations config file, so any run-time errors are silent and do not show up as CI failures.

The reason for using allow_errors: True is because we have cells in notebooks that are supposed to fail for demonstration purposes.

But we need a better way of identify unexpected failures.

We discussed the same question for Cookbooks here: https://github.com/ProjectPythiaCookbooks/cookbook-template/issues/53. The consensus there was to use allow_errors: False but set raises-exception flags on cells that are expected to fail.

It's unclear if anyone has tested this and if it will play well with JupyterBook.

I think we should investigate this option for Foundations.

[brian-rose]

Scanning through all pages in Foundations, here are places where I found that we have an expected failure for a cell:

• https://foundations.projectpythia.org/core/numpy/numpy-basics.html#arithmetic
• https://foundations.projectpythia.org/core/numpy/intermediate-numpy.html#array-comparisons
• https://foundations.projectpythia.org/core/numpy/numpy-broadcasting.html#what-is-broadcasting
• https://foundations.projectpythia.org/core/pandas/pandas.html#extending-to-the-dataframe

I will open a PR that sets the raises-exception flag for all those cells, and sets allow_errors: False for the Foundations jupyterbook build.

[brian-rose]

Actually it appears that all those cells in the notebooks I identified above already have the raises-exception tag.