search

ava-labs / hypersdk

Opinionated Framework for Building Hyper-Scalable Blockchains on Avalanche
Other
204 stars 113 forks source link

RPC server halts under load #1618

Open containerman17 opened 1 month ago

containerman17 commented 1 month ago

I noticed that under a load of a few hundred requests within a few seconds, the RPC server stops responding entirely, even after the requests have completed. This needs further confirmation and investigation.

containerman17 commented 1 month ago

It's easy to reproduce—just launch the spam command with a decent RPS:

  1. Launch a node with ./scripts/run.sh

  2. Launch the spam script:

    go run ./cmd/morpheus-cli/ chain import
go run ./cmd/morpheus-cli/ key import ed25519 demo.pk
go run ./cmd/morpheus-cli/ spam run ed25519
# number of accounts: 1000
# s - default(1.01)
# v - default(2.7)
# txs to try and issue per second: 10000
# minimum txs to issue per second: 1000
# txs to increase per second: 500
# number of clients per node: 50

  3. Start calling any RPC method in another terminal session, for example:

    watch go run ./cmd/morpheus-cli/ key balance

After the node reaches its max TPS (5500 TPS on my Macbook Air), the RPC server stops responding. Stop the spam script, but the RPC API will remain down, even though CPU load went down.