RPC calls are causing read operation to the kvdb_rocksdb that are increadible slow.
this behaviour might only be observed in larger databases (currently we are around 330 GB)
While block imports are still OK,
HTTP request do not get picked up at all.
stack trace analysis show that it is rather connected with slow mallocs,
or slow disk accesses.
Also the shutdown process of diamond-node is slow on this systems.
Nodes with this problem are unable to handle even simple requests like:
curl --data '{"method":"eth_blockNumber","params":[],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST localhost:8545
The Disk Read in such a scenario is about 10 MB / sec at max,
while disk io benachmarks show much better results, so the disk io raw performance is not the problem.
but maybe a large amount of small reads lead to this problem.
the disk IO shows this Load very constant, without having the CPU involved that much (~6% cpu usage)
A typical consequence of that is that for example blockscout is not able to even to a simple request:
2024-02-08T14:58:55.154 application=ethereum_jsonrpc fetcher=coin_balance count=105 [error] 413 Request Entity Too Large returned from single request batch. Cannot shrink batch further. The actual batched request was [%{id: 0, jsonrpc: "2.0", method: "eth_getBlockByNumber", params: ["0x958BE", true]}]. The actual response of the method was :timeout.
version of libc:
Ubuntu GLIBC 2.35-0ubuntu3.5
todo:
[ ] flamegraph sampling to get a clearer picture what is the problem.
[ ] figure out if the problem only occurs on shared HDD, shared CPU.
SurfingNerd
commented
4 months ago
RPC calls are causing read operation to the kvdb_rocksdb that are increadible slow. this behaviour might only be observed in larger databases (currently we are around 330 GB)
While block imports are still OK, HTTP request do not get picked up at all.
stack trace analysis show that it is rather connected with slow mallocs, or slow disk accesses.
Also the shutdown process of diamond-node is slow on this systems.
Nodes with this problem are unable to handle even simple requests like:
The Disk Read in such a scenario is about 10 MB / sec at max, while disk io benachmarks show much better results, so the disk io raw performance is not the problem. but maybe a large amount of small reads lead to this problem. the disk IO shows this Load very constant, without having the CPU involved that much (~6% cpu usage)
A typical consequence of that is that for example blockscout is not able to even to a simple request:
version of libc:
Ubuntu GLIBC 2.35-0ubuntu3.5todo: