The power spectrum integration tests are useful ways of catching when code changes affect the output. However, there are various issues with them:
They are slow, which causes a lot of friction for development.
They fail often, because even updating the code for bugfixes etc changes the output w.r.t. the old output, and so then the data has to be re-run. This can be done, but again creates a lot of friction.
Sometimes they fail for bizarre reasons that can't be figured out (e.g. in #335) or reproduced locally. This probably amounts to the tests themselves being poorly formed (testing for ~1% precision on the power over a wide range of k).
I think we should try to move as many of these tests as we can out of being integration tests, and instead have more of them as unit tests. We can still have some integration tests that are run on-demand when we want to check major version updates etc. But we should think about this more carefully.
The power spectrum integration tests are useful ways of catching when code changes affect the output. However, there are various issues with them:
I think we should try to move as many of these tests as we can out of being integration tests, and instead have more of them as unit tests. We can still have some integration tests that are run on-demand when we want to check major version updates etc. But we should think about this more carefully.