One of the things the Good Scientific Code Workshop teaches is writing good unit tests. I have to admit, ChaosTools.jl suffers a lot from "bad tests", partly because it is the oldest thing in the organization, partly because it is a package really hard to test due to the vast amount of content and the fact that all content is atomic. Nevertheless, these aren't real excuses, and little by little we should try to improve the test suite. I am pasting here the slide with "good advice on writing tests":
Actually unit: test atomic, self-contained functions. Each test must test only one thing, the unit. When a test fails, it should pinpoint the location of the problem. Testing entire processing pipelines (a.k.a. integration tests) should be done only after units are covered, and only if resources/time allow for it!
Known output / Deterministic: tests defined through minimal examples that their result is known analytically are the best tests you can have! If random number generation is necessary, either test valid output range, or use seed for RNG
Robust: Test that the expected outcome is met, not the implementation details. Test that the target functionality is met without utilizing knowledge about the internals. Also, never use internal functions in the test suite.
High coverage: the more functionality of the code is tested, the better
Clean slate: each test file should be runnable by itself, and not rely on previous test files
Fast: use the minimal amount of computations to test what is necessary
Regression: Whenever a bug is fixed, a test is added for this case
Input variety: attempt to cover a wide gambit of input types
One doesn't have to worry about re-writing all tests. In fact, a PR "correcting" a single test file is already very much welcomed!
One of the things the Good Scientific Code Workshop teaches is writing good unit tests. I have to admit, ChaosTools.jl suffers a lot from "bad tests", partly because it is the oldest thing in the organization, partly because it is a package really hard to test due to the vast amount of content and the fact that all content is atomic. Nevertheless, these aren't real excuses, and little by little we should try to improve the test suite. I am pasting here the slide with "good advice on writing tests":
One doesn't have to worry about re-writing all tests. In fact, a PR "correcting" a single test file is already very much welcomed!