NethermindEth / blockifier-tester

A testing utility to run juno with our version of the blockifier that uses cairo native, and compare it to results from mainnet or from juno using the unmodified blockifier
1 stars 1 forks source link

`generate_block_comparison` assumes same transaction List #71

Closed rodrigo-pino closed 2 weeks ago

rodrigo-pino commented 4 weeks ago

Problem

Function generate_block_comparison will compare two list of transaction traces. This two list has always matched but it would be good to have the extra safety that we make sure we are comparing the same pair of transactions.

If any of the list has a different transaction, or missing ones it can make results unreadable, something that can be fixed by smartly pairing them.

For example, having these transactions from base:

[
   "tx 1",
   "tx 2",
   "tx 3"
]

and these ones from native:

[
   "tx 1",
   "tx 3",
]

will result in:

[
    "Same",
    "Different": {
         "base": "tx 2",
         "native": "tx 3"
   }
    "Different": {
         "base": "tx 3",
         "native": "Empty"
   }
]

Which doesn't reflect that "tx 3" is present in both blocks. See example 2 for what would be the ideal output.

Solution

Let's use the sorting mechanism to make the outputs a little clearer.

If there is a transaction i both present in native and base, they both should be compared.

If there is a transaction j present in base but not in native and there is a transaction i that is present in both, such that i > j, then the comparison result of j (with Empty) should come before i. This same case applies for a transaction j in native and not in base.

If there is a transaction j present in base but not in native, and it does not exist any transaction i with i > j equal for both native and base. Then you should first compare all transactions >= j from base with Empty. After that, compare all remaining transactions from native with Empty.

Example 1

Base transactions:

[
   "tx 1",
   "tx 2",
   "tx 3"
]

Native transaction:

[
   "tx 1",
   "tx 2",
]

Comparison result:

[
    "Same",
    "Same",
    "Different": {
         "base": "tx 3",
         "native": "Empty"
   }
]

Example 2

Base transactions:

[
   "tx 1",
   "tx 2",
   "tx 3"
]

Native transaction:

[
   "tx 1",
   "tx 3",
]

Comparison result:

[
    "Same",
    "Different": {
         "base": "tx 2",
         "native": "Empty"
   }
    "Same",
]

Example 3

Base transactions:

[
   "tx 1",
   "tx 3",
 ]

Native transaction:

[
   "tx 1",
   "tx 2",
   "tx 3",
]

Comparison result:

[
    "Same",
    "Different": {
         "base": "Empty",
         "native": "tx 2"
   }
    "Same",
]

Example 4

Base transactions:

[
   "tx 1",
   "tx 2",
   "tx 3",
 ]

Native transaction:

[
   "tx 1",
   "tx 4",
   "tx 5",
]

Comparison result:

[
    "Same",
    "Different": {
         "base": "tx 2",
         "native": "Empty"
   }
   "Different": {
         "base": "tx 3",
         "native": "Empty"
   },
    "Different": {
         "base": "Empty",
         "native": "tx 4"
   },
   "Different": {
         "base": "Empty",
         "native": "tx 5"
   },
]
marcuspang commented 4 weeks ago

@rodrigo-pino Just to clarify, there is actually no sorting involved right? The comparison here is on indices to produce a list of comparison results of size union(base, native)

rodrigo-pino commented 4 weeks ago

@marcuspang no sorting involved. The transactions are already "sorted" in the sense that they are presented in order of execution. We take that assumption as true.

Also, please write some unit tests for these as well. Try to find an elegant and simple solution, there is no urgency for this task at the moment.