search

paritytech / substrate-api-sidecar

REST service that makes it easy to interact with blockchain nodes built using Substrate's FRAME framework.
https://paritytech.github.io/substrate-api-sidecar/dist/
GNU General Public License v3.0
246 stars 153 forks source link

Performance issue with kusama #1554

Open jun0tpyrc opened 3 days ago

jun0tpyrc commented 3 days ago

Description Performance issue with kusama private nodes running behind sidecar hang & stuck => rpc timeout etc , also maybe the cause of explorer slowdown https://kusama.subscan.io/

Steps to Reproduce

run sidecar (v19.2.0/v19.3.1) on your rpc nodes ,tested on polkadot 1.16.2-dba2dd59101, run some calls like /blocks/head etc

Imod7 commented 3 days ago

Thank you for the issue @jun0tpyrc I just tested Sidecar (v19.3.1) with some public RPC endpoints connected to Kusama and I either get almost instant results from /blocks/head or with some delay. However, I cannot reproduce the timeout you are experiencing. On Sidecar's side, there are no significant changes in the code of the last releases that could result in delays in endpoints like /blocks/head. Have you changed something in your nodes ? I am checking now to see if there is something related to the Kusama network in general that could justify a delay.

nfekunprdtiunnkge commented 3 days ago

We are experiencing the same, sidecar using more than 4gbs of ram. Try with blocks around 25875378.

filvecchiato commented 2 days ago

I'm currently looking into this issue. I have seen some degradation for certain blocks in Kusama due to the spammening reported here. Specifically certain blocks in the range: 25875509-25875808

Those blocks have about 4/5k extrinsics per block (compared to 40/50 average) every 6s. So the endpoints in sidecar /blocks/:number or /blocks/head (when those blocks were the head) would struggle to get all extrinsics in order to respond in a timely manner. We are currently looking into a solution to resolve the extrinsics decoding overhead once all the blocks will be averaging that level of traffic. It would be quite helpful to understand your use cases for sidecar specifically to /blocks/head and /blocks/:number so we can work on a more performant solution, thanks!!