Automatically present outputs and diagnostics on representative examples

[axch]

The ideal web page for this project will have a section of worked examples advertising the model's capabilities (not necessarily the same as a tutorial on getting started, which is presumably also example-oriented, but with different emphases).

I am envisioning examples up to the complexity, resolution, and duration of reproducing previously published research results, with

• Graphics reproducing the published figures
• Sanity checks with simulation-specific diagnostics
• Sanity checks with generic diagnostics, like volume conservation
• Reports on how long the runs take and how much memory they use
• If feasible, measuring discrepancies against an equivalent setup in a different model, either MIM itself at a (perhaps slightly) different resolution, or a different model entirely
• Feature (but not necessarily code) coverage

Perhaps also a dashboard summarizing all the examples, showing, e.g., aggregated information on volume conservation. Perhaps also archives of the same across MIM versions.

Ideally, this display can be mechanically rebuilt for any desired version of MIM, and the project page features one for the current "stable release", and maybe one for the current "nightly build".

A display like this serves four purposes:

• Advertising the model's scope, capabilities, and performance;
• Validating the model's correctness;
• Demonstrating to the models' users that the developers care about its correctness and performance; and
• Teaching new users basic sanity checks they can use for their own work, including calibrating what they should expect to see on realistic cases.

Additionally, it effectively constitutes the results section of the MIM paper.

The down sides:

• This sort of thing takes work to set up
• Doing many realistically-sized MIM runs may be more computation than Travis-CI will be willing to just donate, so it may be necessary to either reduce the resolution of the examples or the frequency with which the report is built, and/or to find another source of cycles for producing it.

[edoddridge]

If we want this report to include simulations that recreate recently published results it will certainly take a fair amount of CPU time to produce - I'd be very surprised if Travis-CI let us get away with that.

Other than that concern, I'm onboard with all of this. I think we should also focus on making it as automatic as possible to reduce the maintenance burden in the future.

[edoddridge]

The expanding documentation (see e.g. #133, #155, #157) is making progress on this issue.

As a larger library of images and animations becomes available, it would be good to place a curated selection on the landing page for the documentation (or the project's main webpage if it becomes large enough to deserve something separate from the docs).