Kusama: websockets.exceptions.PayloadTooBig

[alanchchen]

I got this error when syncing Kusama data

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/usr/local/lib/python3.6/site-packages/websockets/protocol.py", line 950, in read_message
    frame = await self.read_data_frame(max_size=max_size)
  File "/usr/local/lib/python3.6/site-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/usr/local/lib/python3.6/site-packages/websockets/protocol.py", line 1051, in read_frame
    extensions=self.extensions,
  File "/usr/local/lib/python3.6/site-packages/websockets/framing.py", line 127, in read
    f"payload length exceeds size limit ({length} > {max_size} bytes)"
websockets.exceptions.PayloadTooBig: payload length exceeds size limit (65535 > 16 bytes)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/app/app/tasks.py", line 114, in accumulate_block_recursive
    block = harvester.add_block(block_hash)
  File "/usr/src/app/app/processors/converters.py", line 366, in add_block
    json_block = self.substrate.get_chain_block(block_hash)
  File "/usr/local/lib/python3.6/site-packages/substrateinterface/__init__.py", line 211, in get_chain_block
    response = self.rpc_request("chain_getBlock", [block_hash]).get('result')
  File "/usr/local/lib/python3.6/site-packages/substrateinterface/__init__.py", line 134, in rpc_request
    asyncio.get_event_loop().run_until_complete(self.ws_request(payload))
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 467, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/site-packages/substrateinterface/__init__.py", line 108, in ws_request
    self._ws_result = json.loads(await websocket.recv())
  File "/usr/local/lib/python3.6/site-packages/websockets/protocol.py", line 509, in recv
    await self.ensure_open()
  File "/usr/local/lib/python3.6/site-packages/websockets/protocol.py", line 812, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 375, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/src/app/app/tasks.py", line 74, in __call__
    return super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 632, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/src/app/app/tasks.py", line 141, in accumulate_block_recursive
    raise HarvesterCouldNotAddBlock(block_hash) from exc
app.processors.converters.HarvesterCouldNotAddBlock: 0x2e1c614587197fded1d5d79c7bc712189f7a74f62d6782c07127604301536fb5

I can simply reproduce it by

>>> from substrateinterface import SubstrateInterface
>>> substrate = SubstrateInterface('wss://kusama-rpc.polkadot.io', type_registry_preset='kusama')
>>> block = substrate.get_chain_block('0x2e1c614587197fded1d5d79c7bc712189f7a74f62d6782c07127604301536fb5')

The block 0x2e1c614587197fded1d5d79c7bc712189f7a74f62d6782c07127604301536fb5 has a really big extrinsic (4th).

I think we should enlarge max_size when using websocket client. https://github.com/polkascan/py-substrate-interface/blob/1d15aafb581c606b717517bdab14bfd8648d1888/substrateinterface/__init__.py#L104 https://websockets.readthedocs.io/en/stable/api.html#module-websockets.client

Just wonder if anyone got this problem as well?