ssvlabs / ssv-spec

GNU General Public License v3.0
25 stars 22 forks source link

Improve performance for test generation and execution #359

Closed MatheusFranco99 closed 7 months ago

MatheusFranco99 commented 7 months ago

Overview

The purpose of this PR is to optimize the performance of test generation and execution with the purpose of improving work quality on future changes.

The ssv module is the one that takes the most time. After running a CPU profiling test, we noticed that the NewTestingKeyManager function consumes 80% of the total time, which is explained by the fact that it's called multiple times and takes 26 milliseconds to run (according to benchmark tests).

Nonetheless, the NewTestingKeyManager function returns a testingKeyManager object that is only used to validate and sign data but does not change state throughout a test execution. Thus, we propose adding a singleton pattern for this object.

Metrics

To illustrate the number of times the NewTestingKeyManager function is called, take a look at the following table

Note: the number after the dash (/) is the number of different instances created, which depends on a slashableDataRoots parameter.

Module Test generation Test execution
QBFT 368/1 390/1
SSV 17058/1 20470/2
Types 48/1 48/1

Performance comparison

With the singleton pattern, if the appropriate object already has been created, the function takes 5 microseconds, rather than 26 milliseconds.

All time values shown below are in seconds (or in minutes:seconds if it has ":" ).

For test generation, we got the following performance results.

QBFT SSV Types
Old 15 7:36 3
New 5 20 3

For test execution, we got the following performance results.

QBFT SSV Types
Old 16 1:22 3
New 4 5 2