tbirdso
commented
1 year ago One consideration for notebook testing is how the wheel under test is made available for testing examples.
- A wheel can be fetched from PyPI, but it will be the most recently tagged/published version and will not reflect changes made in the PR under test. This means that the notebook check result will not be particularly useful for a given PR and may even be misleading. This approach is currently used in external module implementations and its pitfall is noted in https://github.com/KitwareMedical/ITKUltrasound/issues/177
- In theory a wheel artifact can be fetched from the
build-test-python-packagereusable workflow after it runs. The wheel artifact in question would reflect the PR changes under test. This would require that the notebook workflow runs after wheel builds complete, which appears possible but nontrivial to set up. (xref https://stevenmortimer.com/running-github-actions-sequentially/) - A dependent job can be added in the
build-test-python-packageworkflow to optionally run notebook tests if enabled. This would be more difficult to discover, but seems to be more straightforward to implement.
I am in favor of and currently investigating the third approach.
Adapt from workflow in https://github.com/InsightSoftwareConsortium/ITKParabolicMorphology/pull/40/files