Reference tests support tool

[lorenzogentile404]

First of all, let's summarise the involved in the tool:

• https://github.com/Consensys/linea-tracer/tree/arith-dev/reference-tests: reference tests we want to run.
• https://github.com/Consensys/linea-constraints: constraints that needs to be satisfied by the traces generated by each references tests
• https://github.com/Consensys/corset: software verifying if the traces satisfy the above constraints.

The following is a pseudocode describing the tool:

# here it is assumed these lists are already populated
list reference_tests # extracted from linea-tracer repo
list modules # extracted from linea-constraints repo
list constraints # extracted from linea-constraints repo

map module_to_failing_tests = {}
map constraint_to_failing_test = {}

# init maps
for module in modules:
    module_to_failing_tests.put(module, [])

for constraint in constraints:
    constraint_to_failing_tests.put(constraint, [])

for reference_test in reference_tests:
    list failing_constraints = reference_test.run()
    for failing_constraint in failing_constraints:
        module_to_failing_tests.get(failing_constraint.module()).add(reference_test)
        constraint_to_failing_tests.get(failing_constraint).add(reference_test) 

# later we want to have the chance to execute again tests based on these maps
module_to_failing_tests("hub").runAll()
constraint_to_failing_test("hub.name-annoying-constraint").runAll()

Informally, what we want is: 1) For each module, show the list of tests that make fail at least one of its constraints and the length of this list; 2) For each constraint, show the list of tests in which it fails and the length of this list; 3) Allow to selectively run only the tests associated to a failing module (case 1) or a failing constraint (case 2);

[OlivierBBB]

I like the idea.

Unimportant question but what is gained by having a map_of_failing_modules when you already have a map_of_failing_constraints ? I would imagine that the constraints are of the form module.name-of-constraint already, no ?

Also shouldn't the present repo https://github.com/Consensys/linea-tracer be mentioned, too ?

I would also be interested in a map_of_tests_to_java_exceptions and a map_of_tests_to_corset_exceptions.

[lorenzogentile404]

@OlivierBBB indeed constraint_to_failing_test is enough due to naming of constraints following module.name-of-constraint. The module_to_failing_tests is just a convenient way to aggregate all constraints of a module.

[OlivierBBB]

What about the other maps, do we agree that we want them @lorenzogentile404 ?

[lorenzogentile404]

The other maps basically just require to store the output of the references tests since there is no aggregation needed to create them. However, there may be an advantage in having these information in a format that is easy to navigate.

[OlivierBBB]

Intellij allows you to convert your test results into a html document. There may be something like that for whatever is output by the EVM testsuite

[lorenzogentile404]

Then probably there is no need for the two additional maps you proposed.

Another thing to consider is that constraint_to_failing_test may be useful in general and not only for reference tests.

[bradbown]

I have so far got a tool which watches the classes for failures and records the testname by module: constraint: List<FailedTestNames>, below is an example of the current output.

[
    {
        "moduleName": "blockdata",
        "constraints": {
            "value-constraints": [
                "callcodecallcallcode_101_SuicideEnd_d0g0v0_London[London]",
                "callcodecallcode_11_d0g0v0_London[London]",
                "callcodecallcode_11_OOGE_d0g0v0_London[London]",
                "callcodecallcode_11_SuicideEnd_d0g0v0_London[London]",
                "callcodecallcallcode_101_SuicideMiddle_d0g0v0_London[London]",
                "callcodecallcodecall_110_d0g0v0_London[London]",
                "..."
            ]
        }
    },
    {
        "moduleName": "txndata",
        "constraints": {
            "txndata-into-blockdata": [
                "callcodecallcallcode_101_SuicideEnd_d0g0v0_London[London]",
                "callcodecallcode_11_d0g0v0_London[London]",
                "callcodecallcode_11_OOGE_d0g0v0_London[London]",
                "callcodecallcode_11_SuicideEnd_d0g0v0_London[London]",
                "callcodecallcallcode_101_SuicideMiddle_d0g0v0_London[London]",
                "callcodecallcodecall_110_d0g0v0_London[London]",
                "callcodecallcodecall_110_OOGE_d0g0v0_London[London]",
                "..."
            ]
        }
    }
]