Running the workload should require executing only 2 commands:
one command to initialize state
one command to run the workload using that state
State initialized for a given network should be re-usable in subsequent benchmark runs.
Errors and transaction failures should be handled.
The code generating the workload should be easy to understand and extend.
Modular building blocks reduce the effort of generating new workloads.
Building blocks
The following functionality should be generic enough to allow re-using it for other workloads:
[ ] Create an account, enable creating multiple in batches
[ ] Deploy a contract to an account
[ ] Call a contract method
[ ] Get the execution outcome of a transaction
FT workload implementation
Above building blocks should be sufficient to implement the ft workload consisting of:
[ ] Deploy and initialize FT contract(s)
[ ] Create user accounts
[ ] Register them with the FT contract
[ ] Modify the smart contract to enable registering users in batches, i.e. n > 1 users per transaction
[ ] Mint FT for users
[ ] Transfer tokens between users with ft_transfer
Principles
Sequential initialization
Contract and state initialization should be sequential, to dedicate all resources to it.
After its finalization, all transactions sent to the node should be included in tps measurements. The sequential approach avoids tainting tps measurements.
Avoid congestion that currently happens due to the Locust workload sending initialization transactions and ft_transfers at the same time.
Tested workload generation
Executing a command like cargo test should suffice to invoke a test suite for the code that generates the workload.
If some functionality is broken, a test failure should pinpoint the problem. Making the test pass again allows to verify a fix works.
Working on workload generation becomes more convenient when new functionality can be tested automatically, without requiring manual setup and full workload invocations.
Avoid a situation like with the current Locust ft workload that continuously reports errors during "successful" execution.
Use a well maintained RPC library
There are frequent changes to NEAR RPC standards and methods. A well maintained RPC library should handle such changes and minimize the work required to adapt workload generation to RPC changes.
Auto generate tps numbers and further data
After running a workload, a report should be printed containing:
tps numbers
data regarding node health: e.g. skipped chunks, failed transactions, etc.
data regarding node load: e.g. metrics for congestion and factors that limit including transactions or receipts
mooori
commented
1 month ago
Goals
Building blocks
The following functionality should be generic enough to allow re-using it for other workloads:
FT workload implementation
Above building blocks should be sufficient to implement the ft workload consisting of:
n > 1users per transactionft_transferPrinciples
Sequential initialization
Contract and state initialization should be sequential, to dedicate all resources to it.
ft_transfersat the same time.Tested workload generation
Executing a command like
cargo testshould suffice to invoke a test suite for the code that generates the workload.Use a well maintained RPC library
There are frequent changes to NEAR RPC standards and methods. A well maintained RPC library should handle such changes and minimize the work required to adapt workload generation to RPC changes.
Auto generate tps numbers and further data
After running a workload, a report should be printed containing: