Add Solver Statistics

[niklasdewally]

Based on #278.

Add global and solver specific stats objects and save these to {filename}-stats.json for each integration test.

This provides the necessary json and infrastructure for the performance monitoring work for @PedroGGBM.

Adding Minion node count using this will be in the next PR.

See commit log for details.

[niklasdewally]

@PedroGGBM this should provide the basic structure for what you need.

I will add minion node count sometime tomorrow, this will be called nodeCount within the solver runs section of the json

[niklasdewally]

Will look something like this for each test.

{
  "solver_runs": [
    {
      "conjure_solver_wall_time_s": 0.00949225
      "node_count": TODO
    }
  ]
}

[niklasdewally]

I've also made the context serializable so that you can have things like the file name. @PedroGGBM

{
  "extra_rule_set_names": [],
  "file_name": "tests/integration/basic/bool/01/bool-01.essence",
  "stats": {
    "solver_runs": [
      {
        "conjure_solver_wall_time_s": 0.004314042,
        "node_count": TODO
      }
    ]
  }
}

[github-actions[bot]]

Documentation Coverage

This PR

``` +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | crates/conjure_macros/src/lib.rs | 2 | 66.7% | 1 | 33.3% | +-------------------------------------+------------+------------+------------+------------+ | Total | 2 | 66.7% | 1 | 33.3% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | solvers/kissat/src/lib.rs | 0 | 0.0% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 0 | 0.0% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | ...m_compatability_macro/src/lib.rs | 2 | 100.0% | 1 | 50.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 2 | 100.0% | 1 | 50.0% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | crates/uniplate_derive/src/lib.rs | 1 | 50.0% | 1 | 50.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 1 | 50.0% | 1 | 50.0% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | crates/uniplate/src/biplate.rs | 9 | 56.2% | 0 | 0.0% | | crates/uniplate/src/lib.rs | 1 | 100.0% | 1 | 100.0% | | crates/uniplate/src/tree.rs | 0 | 0.0% | 0 | 0.0% | | crates/uniplate/src/uniplate.rs | 9 | 69.2% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 19 | 55.9% | 1 | 3.4% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | solvers/minion/src/ast.rs | 11 | 11.2% | 0 | 0.0% | | solvers/minion/src/error.rs | 8 | 100.0% | 0 | 0.0% | | solvers/minion/src/lib.rs | 1 | 100.0% | 1 | 100.0% | | solvers/minion/src/run.rs | 2 | 100.0% | 1 | 100.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 22 | 20.2% | 2 | 11.8% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | ...onjure_core/src/ast/constants.rs | 0 | 0.0% | 0 | 0.0% | | .../conjure_core/src/ast/domains.rs | 0 | 0.0% | 0 | 0.0% | | ...jure_core/src/ast/expressions.rs | 19 | 95.0% | 0 | 0.0% | | crates/conjure_core/src/ast/mod.rs | 0 | 0.0% | 0 | 0.0% | | ...ure_core/src/ast/symbol_table.rs | 0 | 0.0% | 0 | 0.0% | | ...onjure_core/src/ast/variables.rs | 0 | 0.0% | 0 | 0.0% | | crates/conjure_core/src/context.rs | 0 | 0.0% | 0 | 0.0% | | crates/conjure_core/src/error.rs | 1 | 14.3% | 0 | 0.0% | | crates/conjure_core/src/lib.rs | 0 | 0.0% | 0 | 0.0% | | crates/conjure_core/src/metadata.rs | 0 | 0.0% | 0 | 0.0% | | crates/conjure_core/src/model.rs | 1 | 6.2% | 0 | 0.0% | | ...core/src/parse/example_models.rs | 2 | 100.0% | 0 | 0.0% | | ...es/conjure_core/src/parse/mod.rs | 0 | 0.0% | 0 | 0.0% | | ...re_core/src/parse/parse_model.rs | 0 | 0.0% | 0 | 0.0% | | ...jure_core/src/rule_engine/mod.rs | 5 | 71.4% | 5 | 71.4% | | ...src/rule_engine/resolve_rules.rs | 3 | 100.0% | 0 | 0.0% | | ..._core/src/rule_engine/rewrite.rs | 1 | 33.3% | 0 | 0.0% | | ...ure_core/src/rule_engine/rule.rs | 3 | 25.0% | 0 | 0.0% | | ...core/src/rule_engine/rule_set.rs | 4 | 100.0% | 0 | 0.0% | | ...njure_core/src/rules/constant.rs | 1 | 100.0% | 0 | 0.0% | | ...es/conjure_core/src/rules/mod.rs | 0 | 0.0% | 0 | 0.0% | | ...re/src/solver/adaptors/kissat.rs | 1 | 100.0% | 0 | 0.0% | | ...re/src/solver/adaptors/minion.rs | 1 | 100.0% | 0 | 0.0% | | ..._core/src/solver/adaptors/mod.rs | 1 | 100.0% | 0 | 0.0% | | ...s/conjure_core/src/solver/mod.rs | 14 | 33.3% | 1 | 4.2% | | ...ore/src/solver/model_modifier.rs | 7 | 70.0% | 0 | 0.0% | | ...onjure_core/src/solver/states.rs | 7 | 63.6% | 0 | 0.0% | | ...es/conjure_core/src/stats/mod.rs | 0 | 0.0% | 0 | 0.0% | | ...e_core/src/stats/solver_stats.rs | 3 | 37.5% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 74 | 39.4% | 6 | 8.1% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | conjure_oxide/src/find_conjure.rs | 1 | 50.0% | 0 | 0.0% | | conjure_oxide/src/lib.rs | 0 | 0.0% | 0 | 0.0% | | conjure_oxide/src/utils/conjure.rs | 0 | 0.0% | 0 | 0.0% | | conjure_oxide/src/utils/json.rs | 2 | 66.7% | 0 | 0.0% | | conjure_oxide/src/utils/misc.rs | 0 | 0.0% | 0 | 0.0% | | conjure_oxide/src/utils/mod.rs | 0 | 0.0% | 0 | 0.0% | | conjure_oxide/src/utils/testing.rs | 0 | 0.0% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 3 | 12.0% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | ...05d26364/out/chuffed_bindings.rs | 0 | 0.0% | 0 | 0.0% | | solvers/chuffed/src/lib.rs | 0 | 0.0% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 0 | 0.0% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ ```

Main

``` +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | crates/uniplate_derive/src/lib.rs | 1 | 50.0% | 1 | 50.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 1 | 50.0% | 1 | 50.0% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | crates/conjure_macros/src/lib.rs | 2 | 66.7% | 1 | 33.3% | +-------------------------------------+------------+------------+------------+------------+ | Total | 2 | 66.7% | 1 | 33.3% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | ...m_compatability_macro/src/lib.rs | 2 | 100.0% | 1 | 50.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 2 | 100.0% | 1 | 50.0% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | crates/uniplate/src/biplate.rs | 9 | 56.2% | 0 | 0.0% | | crates/uniplate/src/lib.rs | 1 | 100.0% | 1 | 100.0% | | crates/uniplate/src/tree.rs | 0 | 0.0% | 0 | 0.0% | | crates/uniplate/src/uniplate.rs | 9 | 69.2% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 19 | 55.9% | 1 | 3.4% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | solvers/minion/src/ast.rs | 11 | 11.2% | 0 | 0.0% | | solvers/minion/src/error.rs | 8 | 100.0% | 0 | 0.0% | | solvers/minion/src/lib.rs | 1 | 100.0% | 1 | 100.0% | | solvers/minion/src/run.rs | 2 | 100.0% | 1 | 100.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 22 | 20.2% | 2 | 11.8% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | solvers/kissat/src/lib.rs | 0 | 0.0% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 0 | 0.0% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | ...onjure_core/src/ast/constants.rs | 0 | 0.0% | 0 | 0.0% | | .../conjure_core/src/ast/domains.rs | 0 | 0.0% | 0 | 0.0% | | ...jure_core/src/ast/expressions.rs | 19 | 95.0% | 0 | 0.0% | | crates/conjure_core/src/ast/mod.rs | 0 | 0.0% | 0 | 0.0% | | ...ure_core/src/ast/symbol_table.rs | 0 | 0.0% | 0 | 0.0% | | ...onjure_core/src/ast/variables.rs | 0 | 0.0% | 0 | 0.0% | | crates/conjure_core/src/context.rs | 0 | 0.0% | 0 | 0.0% | | crates/conjure_core/src/error.rs | 1 | 14.3% | 0 | 0.0% | | crates/conjure_core/src/lib.rs | 0 | 0.0% | 0 | 0.0% | | crates/conjure_core/src/metadata.rs | 0 | 0.0% | 0 | 0.0% | | crates/conjure_core/src/model.rs | 1 | 6.7% | 0 | 0.0% | | ...core/src/parse/example_models.rs | 2 | 100.0% | 0 | 0.0% | | ...es/conjure_core/src/parse/mod.rs | 0 | 0.0% | 0 | 0.0% | | ...re_core/src/parse/parse_model.rs | 0 | 0.0% | 0 | 0.0% | | ...jure_core/src/rule_engine/mod.rs | 5 | 71.4% | 5 | 71.4% | | ...src/rule_engine/resolve_rules.rs | 3 | 100.0% | 0 | 0.0% | | ..._core/src/rule_engine/rewrite.rs | 1 | 33.3% | 0 | 0.0% | | ...ure_core/src/rule_engine/rule.rs | 3 | 25.0% | 0 | 0.0% | | ...core/src/rule_engine/rule_set.rs | 4 | 100.0% | 0 | 0.0% | | ...njure_core/src/rules/constant.rs | 1 | 100.0% | 0 | 0.0% | | ...es/conjure_core/src/rules/mod.rs | 0 | 0.0% | 0 | 0.0% | | ...re/src/solver/adaptors/kissat.rs | 1 | 100.0% | 0 | 0.0% | | ...re/src/solver/adaptors/minion.rs | 1 | 100.0% | 0 | 0.0% | | ..._core/src/solver/adaptors/mod.rs | 1 | 100.0% | 0 | 0.0% | | ...s/conjure_core/src/solver/mod.rs | 15 | 35.7% | 1 | 4.2% | | ...ore/src/solver/model_modifier.rs | 7 | 70.0% | 0 | 0.0% | | ...onjure_core/src/solver/states.rs | 7 | 58.3% | 0 | 0.0% | | ...conjure_core/src/solver/stats.rs | 0 | 0.0% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 72 | 41.1% | 6 | 8.5% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | conjure_oxide/src/find_conjure.rs | 1 | 50.0% | 0 | 0.0% | | conjure_oxide/src/lib.rs | 0 | 0.0% | 0 | 0.0% | | conjure_oxide/src/utils/conjure.rs | 0 | 0.0% | 0 | 0.0% | | conjure_oxide/src/utils/json.rs | 2 | 66.7% | 0 | 0.0% | | conjure_oxide/src/utils/misc.rs | 0 | 0.0% | 0 | 0.0% | | conjure_oxide/src/utils/mod.rs | 0 | 0.0% | 0 | 0.0% | | conjure_oxide/src/utils/testing.rs | 0 | 0.0% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 3 | 12.5% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ +-------------------------------------+------------+------------+------------+------------+ | File | Documented | Percentage | Examples | Percentage | +-------------------------------------+------------+------------+------------+------------+ | ...05d26364/out/chuffed_bindings.rs | 0 | 0.0% | 0 | 0.0% | | solvers/chuffed/src/lib.rs | 0 | 0.0% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ | Total | 0 | 0.0% | 0 | 0.0% | +-------------------------------------+------------+------------+------------+------------+ ```

Code Coverage

This PR

  lines......: 73.3% (4328 of 5907 lines)
  functions..: 50.3% (456 of 907 functions)
  branches...: no data found

Main

  lines......: 73.2% (4285 of 5856 lines)
  functions..: 50.2% (451 of 898 functions)
  branches...: no data found