Sim Regression Testing Framework

[dandelany]

Background

As part of our work on Incremental Simulation (see #1414) we identified the need for strong validation & verification, specifically in these three areas:

• Regression testing: validate that a full simulation produces the same results as a simulation of the same model/plan on a previous Aerie version
• Consistency testing: validate that a partial re-simulation (using incremental sim) produces the same results as a full re-sim.
• Internal behavior testing: validate that incremental sims are actually incremental when they should be, ie. that it correctly understands when it needs to re-sim or not.

The last point here will largely be handled by unit tests on the branch itself. However, for the first two, there seems to be an opportunity for us to make a testing tool that we can use for this purpose and also re-use for testing future changes to sim semantics.

Requirements

These are a WIP draft so feel free to edit as we shape requirements further.

• The tool should allow us to test two different branches/versions of Aerie against each other
• There should be a way to set up test "fixtures" ie. models and plans that are the same in both environments, to make sure both are starting from the same data.
• The tool should have a way to trigger a simulation run in both environments
• We need a comparison tool/library for easily comparing the results of both simulation runs to check for significant changes between then
  • "significant changes" is an important caveat - there may be things like IDs which are different between runs but should be ignored and treated as valid/equivalent

[ewferg]

That comparison tool/library almost feels like a feature that could be exposed to our users in the future for doing plan "diff"ing. This has been a highly requested feature in the past.