[Tracking issue] Measure performance improvements of stateless validation on FT benchmark

near / nearcore

Reference client for NEAR Protocol

https://near.org

GNU General Public License v3.0

2.3k stars 600 forks source link

[Tracking issue] Measure performance improvements of stateless validation on FT benchmark #11680

Open akashin opened 1 week ago

akashin commented 1 week ago

We now have the FT benchmark that measures max number of FT transfers that single NEAR shard can handle. We want to evaluate performance improvements brought by stateless validation on this benchmark.

Some questions that we need to clarify:

[x] How many chunk producers and stateless validators do we need to use in this benchmark?
[x] Do we want to also evaluate effects of enabling memtrie as a part of this experiment or separately?
[x] Which hardware should we use in this experiment?
```
### Tasks
```
[ ] https://github.com/near/nearcore/issues/11459
[ ] https://github.com/near/nearcore/issues/11460
[ ] https://github.com/near/nearcore/issues/11729

bowenwang1996 commented 1 week ago

How many chunk producers and stateless validators do we need to use in this benchmark?

We can start with 1 chunk producer and 1 validator

Do we want to also evaluate effects of enabling memtrie as a part of this experiment or separately?

Yes it should be part of this experiment. Post stateless validation launch memtrie will be required so there is no point in separating them

Which hardware should we use in this experiment?

n2d-highmem-8 for the chunk producer. For the validator much lower memory is required (8G or even 4G should be sufficient)

akashin commented 4 hours ago

The work here builds up on a recently finished continuous FT benchmark.

We now have the large state for FT contract and tooling to use that state in the benchmark. The next step is measuring the impact that this state has on the performance of the node with and without memtrie (https://github.com/near/nearcore/issues/11729). The hope is that this will clearly demonstrate the benefit of using the memtrie.

We are also testing the tooling to automatically adjust the gas limit (https://github.com/near/nearcore/issues/11460), as operating at TPS and Gas limit that the node can't support leads to unstable measurements.

Next up, we will need to find a way to run FT benchmark on multiple nodes https://github.com/near/nearcore/issues/11459 with and without stateless validation enabled. We will be looking into Forknet V2 setup for this.