search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.82k stars 2.03k forks source link

websockets.exceptions.PayloadTooBig: payload length exceeds size limit (58173039 > 50000000 bytes) #8061

Closed koolfur closed 3 years ago

koolfur commented 3 years ago

Hi All,

It seems that my payload size has been reached as showed below:

2021-08-14T12:02:25.246 daemon main : INFO Websocket exception. Closing websocket with chia_farmer code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last): File "websockets\protocol.py", line 827, in transfer_data File "websockets\protocol.py", line 895, in read_message File "websockets\protocol.py", line 971, in read_data_frame File "websockets\protocol.py", line 1047, in read_frame File "websockets\framing.py", line 126, in read websockets.exceptions.PayloadTooBig: payload length exceeds size limit (58173039 > 50000000 bytes)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "D:\a\chia-blockchain\chia-blockchain\venv\Lib\site-packages\chia\daemon\server.py", line 172, in safe_handle File "websockets\protocol.py", line 439, in aiter File "websockets\protocol.py", line 509, in recv File "websockets\protocol.py", line 812, in ensure_open websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

Any help would be great. Thanks.

loppefaaret commented 3 years ago

Could expand a bit on your setup ? do you have alot of harvesters or a big harvester ? are you pool farming on a pool that runs low diff generating additional traffic ? OS ?

nicezic commented 3 years ago

I got the same situation.

The farming status in GUI does not work.

2021-08-20T00:07:47.587 daemon chia.daemon.server         : INFO     Websocket exception. Closing websocket with chia_farmer code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
    frame = await Frame.read(
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/framing.py", line 126, in read
    raise PayloadTooBig(
websockets.exceptions.PayloadTooBig: payload length exceeds size limit (106911944 > 50000000 bytes)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/chia/chia-blockchain/chia/daemon/server.py", line 172, in safe_handle
    async for message in websocket:
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 439, in __aiter__
    yield await self.recv()
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 509, in recv
    await self.ensure_open()
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 812, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-08-20T00:07:47.589 farmer chia.rpc.rpc_server        : WARNING  Sending data failed. Exception Traceback (most recent call last):
  File "/home/chia/chia-blockchain/chia/rpc/rpc_server.py", line 67, in _state_changed
    await self.websocket.send_str(dict_to_json_str(payload))
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/client_ws.py", line 150, in send_str
    await self._writer.send(data, binary=False, compress=compress)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 687, in send
    await self._send_frame(message, WSMsgType.TEXT, compress)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 656, in _send_frame
    await self.protocol._drain_helper()
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/base_protocol.py", line 87, in _drain_helper
    await waiter
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
nicezic commented 3 years ago

Could expand a bit on your setup ? do you have alot of harvesters or a big harvester ? are you pool farming on a pool that runs low diff generating additional traffic ? OS ?

Hi~ @loppefaaret

OS : Ubuntu 20.04 server. Harvester connections : about 40 storage servers ( many plots ) + 70 servers. Traffic : only for framing.

nicezic commented 3 years ago

@emlowe Does this affect farming? How is going on the progress?

nicezic commented 3 years ago

Hi~ @emlowe updated chia to 1.2.6, but still got the problem.

2021-09-14T15:28:43.525 daemon chia.daemon.server         : INFO     Websocket exception. Closing websocket with chia_farmer code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
    frame = await Frame.read(
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/framing.py", line 126, in read
    raise PayloadTooBig(
websockets.exceptions.PayloadTooBig: payload length exceeds size limit (135800918 > 50000000 bytes)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/chia/chia-blockchain/chia/daemon/server.py", line 190, in safe_handle
    async for message in websocket:
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 439, in __aiter__
    yield await self.recv()
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 509, in recv
    await self.ensure_open()
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 812, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-09-14T15:28:43.803 full_node chia.full_node.mempool_manager: INFO     It took 0.0018131732940673828 to pre validate transaction
2021-09-14T15:28:43.810 full_node chia.full_node.mempool_manager: INFO     add_spendbundle took 0.007304668426513672 seconds, cost 14762983 (0.134%)
2021-09-14T15:28:43.940 farmer chia.rpc.rpc_server        : WARNING  Sending data failed. Exception Traceback (most recent call last):
  File "/home/chia/chia-blockchain/chia/rpc/rpc_server.py", line 69, in _state_changed
    await self.websocket.send_str(dict_to_json_str(payload))
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/client_ws.py", line 150, in send_str
    await self._writer.send(data, binary=False, compress=compress)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 687, in send
    await self._send_frame(message, WSMsgType.TEXT, compress)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 656, in _send_frame
    await self.protocol._drain_helper()
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/base_protocol.py", line 87, in _drain_helper
    await waiter
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
.
zhangcloud commented 3 years ago

daemon->server.py websocket max_size may be increase; Can compile a new window version ;I do not known how to compile window chia self.websocket_server = await serve( self.safe_handle, self.self_hostname, self.daemon_port, max_size=50 1000 1000, ping_interval=500, ping_timeout=300, ssl=self.ssl_context, )

zhangcloud commented 3 years ago

daemon->server.py websocket max_size may be increase; Can compile a new window version ;I do not known how to compile window chia self.websocket_server = await serve( self.safe_handle, self.self_hostname, self.daemon_port, max_size=50 1000 1000, ping_interval=500, ping_timeout=300, ssl=self.ssl_context, )

zhangcloud commented 3 years ago

Hi @emlowe Some GUI functions are affected,Can you help us solve this problem

emlowe commented 3 years ago

The current limit is 50MB, but one report above was needing 135MB. This seems like an extremely large amount of data, so I'm concerned something else may be going on and that raising the limit, while straightforward, may not be the best approach here.

emlowe commented 3 years ago

Also, can you put your log into DEBUG and provide more of the log - I'd like to find out which specific message is requiring so much data. I have an idea, but I'd like some confirmation

mariano54 commented 3 years ago

It might be the list of plots that is exceeding 50MiB. You must have many pebibytes of plots. Can you say approximately how much? eg. 5-10, 10-20, etc

emlowe commented 3 years ago

It has been reported as well that with very large farms (8 PB) the UI has other problems (see https://github.com/Chia-Network/chia-blockchain-gui/issues/528). So although increasing for now this limit may get past this issue, you may experience other issues using the UI

zhangcloud commented 3 years ago

Hi @emlowe @mariano54 My farm resources as follows,chia version 1.2.6; Local Harvester 366 plots of size: 36.228 TiB Remote Harvester for IP: 192.168.8.16 2057 plots of size: 203.612 TiB Remote Harvester for IP: 192.168.8.28 2073 plots of size: 205.196 TiB Remote Harvester for IP: 192.168.8.33 1087 plots of size: 107.596 TiB Remote Harvester for IP: 192.168.8.26 1010 plots of size: 114.459 TiB Remote Harvester for IP: 192.168.8.41 4296 plots of size: 444.237 TiB Remote Harvester for IP: 192.168.8.24 2043 plots of size: 202.226 TiB Remote Harvester for IP: 192.168.8.14 2000 plots of size: 197.969 TiB Remote Harvester for IP: 192.168.8.39 5242 plots of size: 518.881 TiB Remote Harvester for IP: 192.168.8.42 5126 plots of size: 507.397 TiB Remote Harvester for IP: 192.168.8.35 8253 plots of size: 821.544 TiB Remote Harvester for IP: 192.168.8.25 1917 plots of size: 189.753 TiB Remote Harvester for IP: 192.168.8.30 2097 plots of size: 207.572 TiB Remote Harvester for IP: 192.168.8.11 1999 plots of size: 197.871 TiB Remote Harvester for IP: 192.168.8.40 1841 plots of size: 213.722 TiB Remote Harvester for IP: 192.168.8.18 2027 plots of size: 200.644 TiB Remote Harvester for IP: 192.168.8.12 2092 plots of size: 207.078 TiB Remote Harvester for IP: 192.168.8.29 2093 plots of size: 207.177 TiB Remote Harvester for IP: 192.168.8.21 1179 plots of size: 116.703 TiB Remote Harvester for IP: 192.168.8.34 7358 plots of size: 744.499 TiB Remote Harvester for IP: 192.168.8.19 1985 plots of size: 196.486 TiB Remote Harvester for IP: 192.168.8.23 2102 plots of size: 208.064 TiB Remote Harvester for IP: 192.168.8.17 1823 plots of size: 180.450 TiB Remote Harvester for IP: 192.168.8.36 2983 plots of size: 377.454 TiB Remote Harvester for IP: 192.168.8.31 1853 plots of size: 207.665 TiB Remote Harvester for IP: 192.168.8.27 1921 plots of size: 190.151 TiB Remote Harvester for IP: 192.168.8.13 2022 plots of size: 200.150 TiB Remote Harvester for IP: 192.168.8.15 1867 plots of size: 184.806 TiB Remote Harvester for IP: 192.168.8.22 1883 plots of size: 206.331 TiB Remote Harvester for IP: 192.168.8.32 991 plots of size: 98.095 TiB Remote Harvester for IP: 192.168.8.10 1838 plots of size: 181.934 TiB Remote Harvester for IP: 192.168.8.37 2860 plots of size: 309.547 TiB Remote Harvester for IP: 192.168.8.38 8209 plots of size: 850.251 TiB Plot count for all harvesters: 88493 Total size of plots: 8.824 PiB Estimated network space: 34.769 EiB not only gui about "plots,farms" part can not show ,but also find same harvester can not connnect to farm node last night;harvester reconnect to farm node again and again;
2021-09-16T04:27:13.761 harvester harvester : INFO Reconnecting to peer {'host': '192.168.8.128', 'port': 8447} 2021-09-16T04:27:47.733 harvester harvester : INFO Reconnecting to peer {'host': '192.168.8.128', 'port': 8447} 2021-09-16T04:27:56.032 harvester chia.harvester.harvester: INFO refresh_batch: loaded_plots 0, loaded_size 0.00 TiB, removed_plots 0, processed_plots 0, remaining_plots 0, duration: 0.97 seconds 2021-09-16T04:28:21.760 harvester harvester : INFO Reconnecting to peer {'host': '192.168.8.128', 'port': 8447} 2021-09-16T04:28:55.739 harvester harvester : INFO Reconnecting to peer {'host': '192.168.8.128', 'port': 8447} 2021-09-16T04:29:29.760 harvester harvester : INFO Reconnecting to peer {'host': '192.168.8.128', 'port': 8447}

zhangcloud commented 3 years ago

Hi @emlowe I do not use gui ,just chia.exe start node ,farm,harverster,but still get this error; Will this error affect the reward? can you tell me the step ,how to complile chia in windows;

2021-09-16T14:48:07.716 full_node chia.full_node.mempool_manager: INFO add_spendbundle took 0.0 seconds, cost 40975063 (0.373%) 2021-09-16T14:48:13.235 daemon chia.daemon.server : INFO Register service {'service': 'chia_farmer'} 2021-09-16T14:48:13.235 daemon chia.daemon.server : INFO registered for service chia_farmer 2021-09-16T14:48:13.235 daemon chia.daemon.server : INFO {'success': True} 2021-09-16T14:48:13.391 full_node chia.full_node.full_node: INFO Finished signage point 37/64: CC: df0fd3a8116f5be28a685b5a6570128d66e17bf66b64e71aaabd5c8f63a4b85a RC: b9aff39194a551ab028a4852d1fee1478e133a62ffca324b7b49a9c3045d88fd 2021-09-16T14:48:14.391 harvester chia.harvester.harvester: INFO 0 plots were eligible for farming 3a95277ff2... Found 0 proofs. Time: 0.01563 s. Total 366 plots 2021-09-16T14:48:15.063 full_node chia.full_node.mempool_manager: INFO It took 0.0 to pre validate transaction 2021-09-16T14:48:15.079 full_node chia.full_node.mempool_manager: INFO add_spendbundle took 0.01566767692565918 seconds, cost 247998691 (2.255%) 2021-09-16T14:48:16.970 full_node chia.full_node.full_node: INFO Updated peak to height 866046, weight 1247459280, hh 7d84f8b4e75cdccf278bd5a3aa3a5cacbb986f3bbd153aa3469eef5aa2ed1c45, forked at 866045, rh: 1f52f3133e26c1e893e5555e7aa121eac2bf4460943a204fc88fe4e14f5e24c7, total iters: 3123725404737, overflow: False, deficit: 0, difficulty: 2960, sub slot iters: 136314880, Generator size: 82865, Generator ref list size: 0 2021-09-16T14:48:16.970 full_node chia.full_node.mempool_manager: INFO add_spendbundle took 0.0 seconds, cost 61829 (0.001%) 2021-09-16T14:48:17.032 full_node chia.full_node.mempool_manager: INFO add_spendbundle took 0.0 seconds, cost 31381345 (0.285%) 2021-09-16T14:48:17.032 full_node chia.full_node.mempool_manager: INFO add_spendbundle took 0.0 seconds, cost 10844856 (0.099%) 2021-09-16T14:48:17.032 full_node chia.full_node.mempool_manager: INFO add_spendbundle took 0.0 seconds, cost 40975063 (0.373%) 2021-09-16T14:48:17.032 full_node chia.full_node.mempool_manager: INFO add_spendbundle took 0.0 seconds, cost 247998691 (2.255%) 2021-09-16T14:48:17.032 full_node chia.full_node.mempool_manager: INFO Size of mempool: 5 spends, cost: 331261784 minimum fee to get in: 0 2021-09-16T14:48:17.032 full_node chia.full_node.full_node: INFO Block validation time: 0.5312914848327637, cost: 1350962725, percent full: 12.281% 2021-09-16T14:48:17.157 wallet chia.wallet.wallet_blockchain: INFO Updated wallet peak to height 866046, weight 1247459280, 2021-09-16T14:48:18.189 daemon chia.daemon.server : INFO Websocket exception. Closing websocket with chia_farmer code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last): File "websockets\protocol.py", line 827, in transfer_data File "websockets\protocol.py", line 895, in read_message File "websockets\protocol.py", line 971, in read_data_frame File "websockets\protocol.py", line 1047, in read_frame File "websockets\framing.py", line 126, in read websockets.exceptions.PayloadTooBig: payload length exceeds size limit (52772649 > 50000000 bytes)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "chia\daemon\server.py", line 190, in safe_handle File "websockets\protocol.py", line 439, in aiter File "websockets\protocol.py", line 509, in recv File "websockets\protocol.py", line 812, in ensure_open websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason 2021-09-16T14:48:22.986 full_node chia.full_node.full_node: INFO Finished signage point 38/64: CC: 8ec5017f5469c75

lidangqi commented 3 years ago

When the harvester detects a new Plot file, FULLNODE will have this error.

emlowe commented 3 years ago

It probably partially affects farming, because this causes the connection to drop to the farmer, so I can only guess this has an impact.

I'll look into making this size configurable in config.yaml as a short-term workaround.

Building the windows exes/installer is not that easy - we use pyinstaller. You may find some steps here: https://github.com/Chia-Network/chia-blockchain/blob/main/build_scripts/build_windows.ps1

emlowe commented 3 years ago

You can try this https://github.com/Chia-Network/chia-blockchain/actions/runs/1242170382

You will need to add to config.yaml something like this

daemon_max_message_size: 60000000

or whatever size in bytes you want. I would add that near the top of the file where you see daemon_port

emlowe commented 3 years ago

I do NOT recommend trying the UI with that however.

nicezic commented 3 years ago

@emlowe Is there Ubuntu 20.04 verison?

emlowe commented 3 years ago

For Ubuntu you can with just pull the revision git checkout 393535006a35557f406e77f55c412bd1c994f836. Then perform the upgrade steps as normal.

If you want the deb installer you can get it here: https://github.com/Chia-Network/chia-blockchain/actions/runs/1242170406

emlowe commented 3 years ago

This was merged and is in 1.2.8 - still wouldn't mind hearing if this works - but closing issue

nicezic commented 3 years ago

This was merged and is in 1.2.8 - still wouldn't mind hearing if this works - but closing issue

Yes I updated to 1.2.8. thanks