Metric: number of fungible token transfers

[bowenwang1996]

Set up infrastructure to periodically run a throughput test to see how many fungible token transfers a fork of mainnet can do. This assumes that there is a fungible token contract deployed on every shard. cc @walnut-the-cat

[walnut-the-cat]

Comment from @staffik

Back in the time we used adversenet and grafana metrics: https://near.zulipchat.com/#narrow/stream/295558-core/topic/profiling.20chunk.20processing/near/426159580

In the comment below are instructions from @Longarithm on how to use perf for benchmarking. Though I think Alex run a node in mainnet for that.

If we want to create a fork of mainnet and run locust there, then I think it would be third approach. I would ask @marcelo-gonzalez on what is the current best way to fork mainnet. I've never did that yet.

[Longarithm]

"perf" method is useful only to replay latest txs on chain so it is useless here. Fork of mainnet (forknet?) should be a great long-term approach. At this time I guess it is under active development. Adversenet is the simplest one here and should work in the mid-term. I think I would need 2 weeks to stabilise it to run periodically - implementation should be straightforward, but even our latest setup was very unstable.

[walnut-the-cat]

@Longarithm , how would adversenet set up look like? something like these?

• mainnet binary
• two nodes
• one shard
• no baseline traffic
• contract deployed in node A and bunch of FT transfers are triggered from Node B

[Longarithm]

Yeah, except that we had only one node. Actually setup with N nodes / N shards also must add strong value by checking how much bandwidth is lost due to network and sharding.

[staffik]

We had 1 validator node and 1 RPC node. Transactions were sent to RPC node by a script running at RPC node itself.

[walnut-the-cat]

I think what I would like to know is this. What is LoE to do do the following?

• Set up
  • mainnet binary
  • fork of mainnet (basically have mainnet MPT as baseline state)
  • no baseline traffic
  • 6 shards
  • FT contract deployed in all shards
  • 5 validators per shard
• methodology
  • Generate traffic from all shards (equal amount) to contracts in all shards to fully saturate capacity
  • Run it for meaningful amount of time (e.g. 1 hour)
  • Measure maximum number of FT transfers that can be handled per sec
  • It's completely possible for congestion to happen and that's okay. We should not artificially lift those constraints (e.g. gas limit) as this is to measure what 'current mainnet' can handle

Is it still 2 weeks? Or more since it's somewhat more complicated than what Alex mentioned?

[bowenwang1996]

For this benchmark, let's have at least 5 validators per shard (so 30 total) to make it closer to real-world scenario without making the benchmark too expensive to run

[walnut-the-cat]

@marcelo-gonzalez 's comment

so it looks like what is missing from those scripts is a way to set up the accounts to use for the test. So there are a couple options. The best one would be to allow setting extra accounts when we create a new mocknet test. Another option is to use an existing mainnet account with lots of balance, like in the message I posted a while ago right above this.

But to answer the questions more directly, to get this periodic benchmarking, first I want to finish changing the scripts to use neard fork-network instead of starting from a giant records.json, so that starting a new network wont take like half a day. and then Polina and I are going to work on implementing this periodic thing, where we'll be able to add this locust traffic. Im hopeful we can get something pretty good this month sometime

[aborg-dev]

@MCJOHN974 is looking into this measuring this metric to a first approximation using https://github.com/near/nearcore/blob/master/pytest/tests/loadtest/locust/README.md

[aborg-dev]

I've created a tracking issue for the MVP of this benchmark: https://github.com/near/nearcore/issues/11348

[aborg-dev]

We now publish the metric for number of FT transfers to this dashboard: https://grafana.nearone.org/d/fdo204e4y88owa/continuous-benchmarks?orgId=1

[aborg-dev]

The setup for the FT benchmark is described in https://docs.nearone.org/doc/crt-benchmark-8IpRzRV7C5