ws disconnected occasionally

[shunjizhan]

sometimes after running chopsticks node for a while, it will disconnect from ws, and cannot recover. When making a request to it, it throws ERROR (ws): Error handling request: 'Error: WebSocket is not connected.

In bodhi tests, sometimes it throws such error in the middle of test running, and fails the tests. Rerun tests usually works.

example logs

node          | [06:58:10.116] ERROR (ws): Error handling request: 'Error: WebSocket is not connected
node          |     at <anonymous> (/tmp/bunx-0-@acala-network/chopsticks@0.15.0/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:277:31)
node          |     at new Promise (:1:11)
node          |     at <anonymous> (/tmp/bunx-0-@acala-network/chopsticks@0.15.0/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:274:20)
node          |     at __internal__send (/tmp/bunx-0-@acala-network/chopsticks@0.15.0/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:273:32)
node          |     at send (/tmp/bunx-0-@acala-network/chopsticks@0.15.0/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:262:34)
node          |     at send (/tmp/bunx-0-@acala-network/chopsticks@0.15.0/node_modules/@acala-network/chopsticks-core/dist/cjs/api.js:109:16)
node          |     at getStorage (/tmp/bunx-0-@acala-network/chopsticks@0.15.0/node_modules/@acala-network/chopsticks-core/dist/cjs/api.js:137:22)
node          |     at <anonymous> (/tmp/bunx-0-@acala-network/chopsticks@0.15.0/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/storage-layer.js:103:65)
node          |     at get (/tmp/bunx-0-@acala-network/chopsticks@0.15.0/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/storage-layer.js:92:15)
node          |     at <anonymous> (/tmp/bunx-0-@acala-network/chopsticks@0.15.0/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/storage-layer.js:205:65)'
node          |     app: "chopsticks"
node          | 2024-10-08 06:58:30          API-WS: disconnected from wss://acala-rpc.aca-api.network: 1006:: Connection ended
node          | 2024-10-08 06:59:34          API-WS: disconnected from wss://acala-rpc.aca-api.network: 1006:: Connection ended
node          | 2024-10-08 07:00:37          API-WS: disconnected from wss://acala-rpc.aca-api.network: 1006:: Connection ended

Maybe the ws reconnecting mechanism failed to work? Expected behavior:

receive request
=> if ws is connected: handle request normally
=> if ws has disconnected: first reconnect, then handle request normally

[shunjizhan]

reproduce

use this docker compose file, and docker compose up.

Then just wait for a while, until seeing bunch of these logs. Now sending tx will throw ERROR (ws): Error handling request: 'Error: WebSocket is not connected

node          | 2024-10-08 07:18:05          API-WS: disconnected from wss://acala-rpc.aca-api.network: 1006:: Connection ended
node          | 2024-10-08 07:19:08          API-WS: disconnected from wss://acala-rpc.aca-api.network: 1006:: Connection ended
node          | 2024-10-08 07:20:12          API-WS: disconnected from wss://acala-rpc.aca-api.network: 1006:: Connection ended
node          | 2024-10-08 07:21:18          API-WS: disconnected from wss://acala-rpc.aca-api.network: 1006:: Connection ended
node          | 2024-10-08 07:22:21          API-WS: disconnected from wss://acala-rpc.aca-api.network: 1006:: Connection ended

[xlc]

this could be related to wss://acala-rpc.aca-api.network which I have observed some disconnects but not yet got time to investigate. can you try RPC hosted by other infra provider and see if it is still happening?

[shunjizhan]

yeah seems to be wss://acala-rpc.aca-api.network's issue, tried wss://acala-rpc.dwellir.com and it's been very stable so far.

Closing it since it's not related to chopsticks

[shunjizhan]

might be subway's issue because that's where wss://acala-rpc.aca-api.network is different

[shunjizhan]

sometimes also get this error Error: disconnected from wss://acala-rpc.aca-api.network: 1000:: Normal Closure

https://github.com/AcalaNetwork/bodhi.js/actions/runs/11236784903/job/31237676355?pr=1032