Consensys / linea-tracer

Part of the Linea stack responsible for extracting data from the execution of an EVM client in order to construct large matrices called execution traces.
https://linea.build
Other
35 stars 25 forks source link

Remove Randomness from `MXP` tests #1536

Closed DavePearce closed 4 days ago

DavePearce commented 4 days ago

Overview

Currently, there is a degree of non-determinism in the MXP tests which use a static field RAND of type java.util.Random. It is a mistake to share a single instance of Random across multiple tests. This is because it can make it very difficult to recreate problems observed on e.g. the CI pipeline.

Details

The fundamental problem with a shared Random instance is that the order in which tests are run will affect the values returned by RAND.nextInt(). Hence, on the CI pipeline, it will run through them one by one --- meaning some value X is returned by RAND.nextInt() for test T. However, if we then run test T on its own then a different value Y will be returned from RAND.nextInt().

Observe the problem is compounded when tests are run in parallel --- since the exact order in which calls to nextInt() are made is inherently non-deterministic.

NOTE: this only applies to static fields. Instance fields are not reused across tests because JUnit always creates a new instance for each @Test being executed.

Solution

The solution is to drop the static modifier so that RAND is an instance field, rather than a static field.

DavePearce commented 4 days ago

Looks like there are a few problematic cases in the tests: