skalenetwork / skaled

Running more than 20 production blockchains, SKALED is Ethereum-compatible, high performance C++ Proof-of-Stake client, tools and libraries. Uses SKALE consensus as a blockchain consensus core. Includes dynamic Oracle. Implements file storage and retrieval as an EVM extension.
https://skale.network
GNU General Public License v3.0
85 stars 39 forks source link

High RAM usage when loaded with JSON-RPC #1629

Open dimalit opened 1 year ago

dimalit commented 1 year ago

1 Have blocks full of 24-kb transactions 2 Enable blockscout. It will request all blocks with their transactions. 3 See RAM > 2.5 GB

### Tasks
- [ ] Investigate how timeout affects performance
- [ ] Deside if we need to limit RAM
dimalit commented 10 months ago

HTTP request flow:

  1. Incoming connection is put in accept backlog or rejected if backlog is full
  2. Connection waits in backlog
  3. Connection is accepted
  4. Request body is read
  5. Response is generated
  6. Response is transferred back to client
  7. Connection is terminated

Problem: When many connections are in the middle of stage 6 - it can consume a lot of RAM.

Proposed solution(s): Count current number (or RAM size?) of requests in stage 6 - and if limit is reached: 1) Don't accept new requests on stage 3 or 2) Block requests from moving from stage 4 to stage 5 or 3) Drop some old connections that are on stage 6 for long time

Need to decide

DmytroNazarenko commented 10 months ago

How much is it - Many connections in the middle of stage 6?

DmytroNazarenko commented 10 months ago

In p.1 clients will receive the message that server is overloaded straightaway and for them it would be clearer the root cause of the issue. That's why p.1 is better than 2 and 3

PolinaKiporenko commented 9 months ago

Moved to Blocked due to lack of testing resources on DevNet, needs to recheck during Regression testing 2.3

For now timeout was decreased to 5 sec from 30 min (default timeout 2 min), still need to check if it not too short timeout and make sure that the amount of memory consumed is sufficient

RAM 1,5 GB

PolinaKiporenko commented 8 months ago

Moved to 2.4 - need more investigate of performance with decreasing limits

PolinaKiporenko commented 6 months ago

Could be postponed to 2.5 release