Validate FT transfer benchmark performance

akhi3030 commented 2 months ago

There is some confusion as to what performance we should expect to see on mainnet today. @mooori will investigate what results to expect.

mooori commented 2 months ago

As discussed with Akhi, I followed this guide to set up a multinode statelessnet network and ran the locust workloads on it. The neard binary was built off commit ee5f317.

Results

Description	TPS
Not touching state	~112
Touching little state	~121

I was not able to run the workload which touches all state due to the issue described in #12012.

The number of tps has been calculated by querying nodes' /metrics endpoint, similar to how it is done for the continuous benchmark.

Possible reason for low tps

The continuous benchmarks are run with chain_id: benchmarknet, which lifts restrictions for various config parameters:

https://github.com/near/nearcore/blob/4918d6cf9ff352df4de17df2811e356149b18352/core/parameters/src/config_store.rs#L142-L155

In contrast, the multinode network runs on chain_id: test-chain-9hMv4_mocknet. Here the corresponding config values are much more restrictive. This has been confirmed by connecting to a node and querying its protocol config. Presumably the multinode setup uses the values that validators on mainnet would use by default:

"congestion_control_config": {
  "allowed_shard_outgoing_gas": 1000000000000000,
  "max_congestion_incoming_gas": 2e+16,
  "max_congestion_memory_consumption": 1000000000,
  "max_congestion_missed_chunks": 5,
  "max_congestion_outgoing_gas": 1e+16,
  "max_outgoing_gas": 3e+17,
  "max_tx_gas": 500000000000000,
  "min_outgoing_gas": 1000000000000000,
  "min_tx_gas": 20000000000000,
  "outgoing_receipts_big_size_limit": 4718592,
  "outgoing_receipts_usual_size_limit": 102400,
  "reject_tx_congestion_threshold": 0.5
},
...
"witness_config": {
  "combined_transactions_size_limit": 4194304,
  "main_storage_proof_size_soft_limit": 3000000,
  "new_transactions_validation_state_size_soft_limit": 572864
},
...
"wasm_config.limit_config.per_receipt_storage_proof_size_limit": 4000000

mooori commented 2 months ago

The adjustments for benchmarknet were made to remove ceilings on tps, see e.g. here for reference. Which makes the absence of these adjustments in the multinode setup likely a contributing factor to the low tps values.

mooori commented 2 months ago

Repeating above experiment with same setup only changing the number of validator nodes from 5 to 1 yields the following results:

Description	TPS
"touching little state"	~145

neard off the same commit was used.

Interpretation of the result

As described above, a possible reason for the low tps number is restrictive configuration. If factors like congestion control, witness size, and gas limit are the bottleneck, then I wouldn't suspect a big tps difference in single-node vs multi-node setups.

near / nearcore