search

openethereum / parity-ethereum

The fast, light, and robust client for Ethereum-like networks.
Other
6.81k stars 1.68k forks source link

JSON-RPC sometimes takes too long to have a result #11820

Closed jesinspirment closed 4 years ago

jesinspirment commented 4 years ago

Before filing a new issue, please provide the following information.

If you think that your issue is an exploitable security vulnerability, please mail your bugreport to security@parity.io instead; your submission might be eligible for our Bug Bounty. You can find mode info on the reporting process in SECURITY.md

I have observed that for certain RPC requests, particularly eth_sendRawTransaction, eth_getTransactionByHash, eth_getTransactionReceipt, sometimes it takes extremely long to have a result, could be even a minute. This is quite disruptive for our production projects.

Is there a way we can minimise this? Initially, we have 2-core CPU (Xeon) and 8 GB RAM with --cache-size=3072. Suspecting that we may be able to overcome this by having more RAM, we upgraded to a 16 GB RAM node with --cache-size=8192. It seems that we still occasionally get this problem. We are using SSD too btw.

If it helps, we are running OpenEthereum with the following options: --jsonrpc-interface="all" --jsonrpc-apis="all" --jsonrpc-cors="" --ws-interface="all" --ws-apis="all" --ws-origins="" --ws-hosts="all" --db-compaction=ssd --cache-size=8192 --min-peers=50 --max-peers=80 --snapshot-peers=50 --mode=active --release-track=stable --tx-queue-size=200000 --tx-queue-per-sender=2000 --tx-queue-mem-limit=128

adria0 commented 4 years ago

Hi @jesinspirment, could you try running it with --no-periodic-snapshot flag activated, please? Your feedback about this change will be very appreciated.

jesinspirment commented 4 years ago

Ok will do for sure. But can I check why? Is there an issue if I allow the periodic snapshot creation? Also, what is the consequence if I do disable it, other than the fact that it will not generate the snapshot?

adria0 commented 4 years ago

@jesinspirment we have been doing some internal checks and snapshot generation takes a lot of CPU and memory. We think that at this moment this makes sense for bootnodes and nodes with 4CPU/32GB RAM at least. You should not have any additional consequences by disabling this. We are going to disable this option by default in the next openethereum release.

jesinspirment commented 4 years ago

Ok I have just restarted my 16 GB server like this:

--jsonrpc-interface="all" --jsonrpc-apis="all" --jsonrpc-cors="" --ws-interface="all" --ws-apis="all" --ws-origins="" --ws-hosts="all" --db-compaction=ssd --cache-size=8192 --min-peers=50 --max-peers=80 --snapshot-peers=50 --mode=active --release-track=stable --tx-queue-size=200000 --tx-queue-per-sender=2000 --tx-queue-mem-limit=128 --no-periodic-snapshot

I will monitor for a while, and report back to you my findings. Hope this will help you guys.

jesinspirment commented 4 years ago

@adria0 I am still monitoring, but so far, disabling periodic snapshot seems to improve things. And it's a major improvement, at least in terms of memory usage.

viaweb3 commented 4 years ago

The same issue, seems very slow when broadcast transactions.

adria0 commented 4 years ago

The same issue, seems very slow when broadcast transactions.

Please, @niulingyun, try with --no-periodic-snapshot flag activated. Your feedback about this change will be very appreciated.

viaweb3 commented 4 years ago

The same issue, seems very slow when broadcast transactions.

Please, @niulingyun, try with --no-periodic-snapshot flag activated. Your feedback about this change will be very appreciated.

not working, I am exchange.

adria0 commented 4 years ago

The same issue, seems very slow when broadcast transactions.

Please, @niulingyun, try with --no-periodic-snapshot flag activated. Your feedback about this change will be very appreciated.

not working, I am exchange.

In which hardware are you running the client? CPU/RAM/Storage(SSD/NVM/HD)

viaweb3 commented 4 years ago

The same issue, seems very slow when broadcast transactions.

Please, @niulingyun, try with --no-periodic-snapshot flag activated. Your feedback about this change will be very appreciated.

not working, I am exchange.

In which hardware are you running the client? CPU/RAM/Storage(SSD/NVM/HD)

AWS tokyo，m4.2xlarge with SSD , in docker

adria0 commented 4 years ago

The same issue, seems very slow when broadcast transactions.

Please, @niulingyun, try with --no-periodic-snapshot flag activated. Your feedback about this change will be very appreciated.

not working, I am exchange.

In which hardware are you running the client? CPU/RAM/Storage(SSD/NVM/HD)

AWS tokyo，m4.2xlarge with SSD , in docker

We need more details to understand where the problem can be, please, can you provide information about: 1) CPU/RAM 2) Which transactions are you running? How much RPCs per second? 3) Are you running a fatdb or archive node?

viaweb3 commented 4 years ago
  1. 16 CORE 64G RAM
  2. transfer and many read rpc calls, like get block, get receipt, trace transaction, get balance
  3. ENTRYPOINT ["/root/openethereum", "--jsonrpc-apis", "safe,parity_set", "--no-ws", "--no-ipc", "--max-peers", "300", "--min-peers","200", "--tracing", "on", "--unsafe-expose", "--no-download", "--tx-queue-mem-limit=64", "--tx-queue-size=200000", "--no-periodic-snapshot"]
viaweb3 commented 4 years ago

I change my send function to geth and keep read functions on parity, and works well.

adria0 commented 4 years ago

I change my send function to geth and keep read functions on parity, and works well.

@niulingyun, we do not know what "works well" in terms of operations per second means for you, openethereum and geth have different database models and for some RPC operations, the same query has different costs. If you dou think that this information is confidential you can contact me in discoord https://discord.io/openethereum

rakita commented 4 years ago

Initial problem seems resolved