Getting "connection dropped by remote peer" when querying too fast

[anhcao142]

Description

This usually happens when I try to call multiple requests get block at the same time and after a while, this error appears in the substrate-api-sidecar server. Here is the error log:

2020-10-13 11:01:35 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: connection failed
2020-10-13 11:01:36 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: connection failed
2020-10-13 11:01:37 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: connection failed
2020-10-13 11:01:39 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: connection failed
2020-10-13 11:01:40 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: connection failed
2020-10-13 11:01:41 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: connection failed
2020-10-13 11:04:48 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: Connection dropped by remote peer.
2020-10-13 11:06:48 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: Connection dropped by remote peer.
2020-10-13 11:08:30 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: Connection dropped by remote peer.
2020-10-13 11:10:15 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: Connection dropped by remote peer.
2020-10-13 11:12:27 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: Connection dropped by remote peer.
2020-10-13 11:13:57 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: Connection dropped by remote peer.
2020-10-13 11:16:41 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: Connection dropped by remote peer.
2020-10-13 11:17:55 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: Connection dropped by remote peer.
2020-10-13 11:18:57 error: API-WS: disconnected from wss://rpc.polkadot.io: 1006:: Connection dropped by remote peer.

Steps to Reproduce

const Promise = require('bluebird');
const fetch = require('node-fetch');

async function main() {
  // Generate array from 2004125 to 2005124
  const range = Array.from(new Array(1000), (x, i) => i + 2004125);

  await Promise.map(range, async (blockHeight) => {
    const block = await get(`blocks/${blockHeight}`);
    console.log(block.number);
  }, { concurrency: 10 }); // Try to increase this number if you don't see the error log after a while
}

async function get(path) {
  const url = 'http://localhost:8080';
  const res = await fetch(`${url}/${path}`);
  return res.json();
}

main().catch(console.log);

Expected vs. Actual Behavior I don't know if the connection is auto-renewal after the log or it just stays disconnected.

[emostov]

I can't reproduce this issue. I can do 1000 concurrent requests without any issue and I have not seen this issue when doing load testing in the past.

How are you running the node and sidecar? Are you using a public node?

[anhcao142]

Yeah, I'm currently using with public node wss://rpc.polkadot.io. Haven't tried it with local node yet

[emostov]

I had no issue while using the public node

[anhcao142]

I still unable to provide a reproduce source code, but I have a service that repeatedly checks if there is a new block and fetches it, and after like half a day this error appears. I have to restart the substrate-api-sidecar server to fix it temporarily. I think it has something to do with the WebSocket server after connecting for a while.

Currently, I have a version of substrate-api-sidecar that I modified to use JSON-RPC instead of WebSocket, and it seems stable. The server has been running for more than a day without any connection issues.

[schadokar]

I am also facing this issue. Don't know the exact solution. For now I am using the nginx load balancer and whenever this error occurs, I am restarting the instances.

[TarikGul]

@anhcao142 Has this still been an issue for you? polkadot-js's ws provider has a built in reconnect which has proven to work for me when a connection drops. In terms of bottlenecks I havent been able to reproduce the issue above.

It's not recommended to be scraping from a public node, as unexpected error may occur and are way more common.

Closing this issue for now, please reopen if it reoccurs.

[anhcao142]

Hi @TarikGul, thanks for asking, I haven't encountered this anymore.

[TarikGul]

@anhcao142 Sweet thanks for letting me know! :)