[BUG] ConnectionResetError: Cannot write to closing transport

[Drakanus]

Describe the bug After some hours of working, it seems that something gets stuck (+-6-8 hrs) - more logs below. First, I always see that Last Attempted Proof is not updated, then in pool section I can see that the line is going down (till 0), everything seems to be synched, space pool partial section is also not upgrated till the chia is not restarted.. After the chia is restarted - everything is ok for next 6-8 hours. All instances are on the same version.

To Reproduce

Steps to reproduce the behavior:

just leave chia for a few hours

Expected behavior

Screenshots

Desktop

I use: 1 Win 10 which is set to be the farmer + of course it plots 1 Win 10, which is set to be the harvester + of course it plots 1 Ubuntu machine, which is set to be the harvester + of course it plots 1 TNAS machine, which receives the plots from 2x Win 10 machines

Additional context 2021-08-09T00:37:15.775 harvester chia.harvester.harvester: WARNING Looking up qualities on \TNAS-DB24\public\pool\plot-k32-2021-07-08-19-29.plot took: 7.176719903945923. This should be below 5 seconds to minimize risk of losing rewards. 2021-08-09T00:39:17.679 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T00:40:48.701 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T00:42:19.713 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T00:43:50.722 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T00:45:21.725 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T00:46:52.740 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T00:48:23.761 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T00:49:54.777 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T00:51:25.805 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T00:51:44.522 daemon main : ERROR Unexpected exception trying to send to websocket: code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last): File "D:\a\chia-blockchain\chia-blockchain\venv\Lib\site-packages\chia\daemon\server.py", line 187, in safe_handle File "websockets\protocol.py", line 555, in send File "websockets\protocol.py", line 803, in ensure_open websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-08-09T00:52:56.838 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T00:54:27.862 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T00:54:41.919 daemon main : ERROR Unexpected exception trying to send to websocket: code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last): File "D:\a\chia-blockchain\chia-blockchain\venv\Lib\site-packages\chia\daemon\server.py", line 187, in safe_handle File "websockets\protocol.py", line 555, in send File "websockets\protocol.py", line 803, in ensure_open websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-08-09T00:55:58.890 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T00:57:29.922 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T00:57:43.352 daemon websockets.protocol : ERROR Error in data transfer Traceback (most recent call last): File "websockets\protocol.py", line 827, in transfer_data File "websockets\protocol.py", line 895, in read_message File "websockets\protocol.py", line 995, in read_data_frame File "websockets\protocol.py", line 766, in pong File "websockets\protocol.py", line 803, in ensure_open websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason 2021-08-09T00:59:00.929 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T01:00:18.185 daemon main : ERROR Unexpected exception trying to send to websocket: code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last): File "websockets\protocol.py", line 827, in transfer_data File "websockets\protocol.py", line 895, in read_message File "websockets\protocol.py", line 995, in read_data_frame File "websockets\protocol.py", line 766, in pong File "websockets\protocol.py", line 803, in ensure_open websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "D:\a\chia-blockchain\chia-blockchain\venv\Lib\site-packages\chia\daemon\server.py", line 187, in safe_handle File "websockets\protocol.py", line 555, in send File "websockets\protocol.py", line 803, in ensure_open websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-08-09T01:00:31.957 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T01:01:10.041 full_node chia.rpc.rpc_server : WARNING Sending data failed. Exception Traceback (most recent call last): File "chia\rpc\rpc_server.py", line 67, in _state_changed File "aiohttp\client_ws.py", line 150, in send_str File "aiohttp\http_websocket.py", line 687, in send File "aiohttp\http_websocket.py", line 643, in _send_frame File "aiohttp\http_websocket.py", line 660, in _write ConnectionResetError: Cannot write to closing transport . 2021-08-09T01:01:10.530 full_node chia.rpc.rpc_server : WARNING Sending data failed. Exception Traceback (most recent call last): File "chia\rpc\rpc_server.py", line 67, in _state_changed File "aiohttp\client_ws.py", line 150, in send_str File "aiohttp\http_websocket.py", line 687, in send File "aiohttp\http_websocket.py", line 643, in _send_frame File "aiohttp\http_websocket.py", line 660, in _write ConnectionResetError: Cannot write to closing transport . 2021-08-09T01:01:16.305 full_node chia.rpc.rpc_server : WARNING Sending data failed. Exception Traceback (most recent call last): File "chia\rpc\rpc_server.py", line 67, in _state_changed File "aiohttp\client_ws.py", line 150, in send_str File "aiohttp\http_websocket.py", line 687, in send File "aiohttp\http_websocket.py", line 643, in _send_frame File "aiohttp\http_websocket.py", line 660, in _write ConnectionResetError: Cannot write to closing transport . 2021-08-09T01:01:17.122 full_node chia.rpc.rpc_server : WARNING Sending data failed. Exception Traceback (most recent call last): File "chia\rpc\rpc_server.py", line 67, in _state_changed File "aiohttp\client_ws.py", line 150, in send_str File "aiohttp\http_websocket.py", line 687, in send File "aiohttp\http_websocket.py", line 643, in _send_frame File "aiohttp\http_websocket.py", line 660, in _write ConnectionResetError: Cannot write to closing transport . 2021-08-09T01:02:02.960 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T01:02:16.003 daemon main : ERROR Unexpected exception trying to send to websocket: code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last): File "D:\a\chia-blockchain\chia-blockchain\venv\Lib\site-packages\chia\daemon\server.py", line 187, in safe_handle File "websockets\protocol.py", line 555, in send File "websockets\protocol.py", line 803, in ensure_open websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-08-09T01:03:33.989 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T01:03:52.659 harvester chia.plotting.plot_tools: ERROR Failed to open file \TNAS-DB24\public\pool\plot-k32-2021-07-09-01-29-zzz.plot. Invalid plot header magic Traceback (most recent call last): File "chia\plotting\plot_tools.py", line 189, in process_file ValueError: Invalid plot header magic

2021-08-09T01:03:56.754 harvester chia.plotting.plot_tools: ERROR Failed to open file \TNAS-DB24\public\pool\plot-k32-2021-07-07-03-14-zzz.plot. badbit or failbit after reading size 4 at position 107536099631 Traceback (most recent call last): File "chia\plotting\plot_tools.py", line 189, in process_file RuntimeError: badbit or failbit after reading size 4 at position 107536099631

2021-08-09T01:04:00.049 harvester chia.plotting.plot_tools: ERROR Failed to open file \TNAS-DB24\public\pool\plot-k32-2021-07-07-12-21-zzz.plot. Invalid plot header magic Traceback (most recent call last): File "chia\plotting\plot_tools.py", line 189, in process_file ValueError: Invalid plot header magic

2021-08-09T01:04:07.139 harvester chia.plotting.plot_tools: ERROR Failed to open file \TNAS-DB24\public\pool\plot-k32-2021-07-23-11-49-zzz.plot. Invalid plot header magic Traceback (most recent call last): File "chia\plotting\plot_tools.py", line 189, in process_file ValueError: Invalid plot header magic

2021-08-09T01:04:11.288 harvester chia.plotting.plot_tools: ERROR Failed to open file \TNAS-DB24\public\pool\plot-k32-2021-07-08-01-47-zzz.plot. badbit or failbit after reading size 4 at position 107580944971 Traceback (most recent call last): File "chia\plotting\plot_tools.py", line 189, in process_file RuntimeError: badbit or failbit after reading size 4 at position 107580944971

2021-08-09T01:04:16.440 harvester chia.plotting.plot_tools: ERROR Failed to open file \TNAS-DB24\public\pool\plot-k32-2021-07-07-03-14-zzz.plot. badbit or failbit after reading size 4 at position 107514987023 Traceback (most recent call last): File "chia\plotting\plot_tools.py", line 189, in process_file RuntimeError: badbit or failbit after reading size 4 at position 107514987023

2021-08-09T01:04:18.945 harvester chia.plotting.plot_tools: ERROR Failed to open file \TNAS-DB24\public\pool\plot-k32-2021-07-09-01-29-zzz.plot. Invalid plot header magic Traceback (most recent call last): File "chia\plotting\plot_tools.py", line 189, in process_file ValueError: Invalid plot header magic

2021-08-09T01:04:25.549 harvester chia.plotting.plot_tools: ERROR Failed to open file \TNAS-DB24\public\pool\plot-k32-2021-07-07-03-14-zzz.plot. badbit or failbit after reading size 4 at position 107554495250 Traceback (most recent call last): File "chia\plotting\plot_tools.py", line 189, in process_file RuntimeError: badbit or failbit after reading size 4 at position 107554495250

2021-08-09T01:04:25.549 harvester chia.plotting.plot_tools: ERROR Failed to open file \TNAS-DB24\public\pool\plot-k32-2021-07-23-11-49-zzz.plot. Invalid plot header magic Traceback (most recent call last): File "chia\plotting\plot_tools.py", line 189, in process_file ValueError: Invalid plot header magic

2021-08-09T01:04:29.606 harvester chia.plotting.plot_tools: ERROR Failed to open file \TNAS-DB24\public\pool\plot-k32-2021-07-07-03-14-zzz.plot. badbit or failbit after reading size 4 at position 107473917291 Traceback (most recent call last): File "chia\plotting\plot_tools.py", line 189, in process_file RuntimeError: badbit or failbit after reading size 4 at position 107473917291

2021-08-09T01:05:05.001 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T01:05:13.641 daemon main : ERROR Unexpected exception trying to send to websocket: code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last): File "websockets\protocol.py", line 838, in transfer_data asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "D:\a\chia-blockchain\chia-blockchain\venv\Lib\site-packages\chia\daemon\server.py", line 187, in safe_handle File "websockets\protocol.py", line 555, in send File "websockets\protocol.py", line 812, in ensure_open websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-08-09T01:06:06.506 full_node chia.rpc.rpc_server : WARNING Sending data failed. Exception Traceback (most recent call last): File "chia\rpc\rpc_server.py", line 67, in _state_changed File "aiohttp\client_ws.py", line 150, in send_str File "aiohttp\http_websocket.py", line 687, in send File "aiohttp\http_websocket.py", line 643, in _send_frame File "aiohttp\http_websocket.py", line 660, in _write ConnectionResetError: Cannot write to closing transport . 2021-08-09T01:06:06.522 full_node chia.rpc.rpc_server : WARNING Sending data failed. Exception Traceback (most recent call last): File "chia\rpc\rpc_server.py", line 67, in _state_changed File "aiohttp\client_ws.py", line 150, in send_str File "aiohttp\http_websocket.py", line 687, in send File "aiohttp\http_websocket.py", line 643, in _send_frame File "aiohttp\http_websocket.py", line 660, in _write ConnectionResetError: Cannot write to closing transport . 2021-08-09T01:06:06.527 full_node chia.rpc.rpc_server : WARNING Sending data failed. Exception Traceback (most recent call last): File "chia\rpc\rpc_server.py", line 67, in _state_changed File "aiohttp\client_ws.py", line 150, in send_str File "aiohttp\http_websocket.py", line 687, in send File "aiohttp\http_websocket.py", line 643, in _send_frame File "aiohttp\http_websocket.py", line 660, in _write ConnectionResetError: Cannot write to closing transport . 2021-08-09T01:06:06.530 full_node chia.rpc.rpc_server : WARNING Sending data failed. Exception Traceback (most recent call last): File "chia\rpc\rpc_server.py", line 67, in _state_changed File "aiohttp\client_ws.py", line 150, in send_str File "aiohttp\http_websocket.py", line 687, in send File "aiohttp\http_websocket.py", line 643, in _send_frame File "aiohttp\http_websocket.py", line 660, in _write ConnectionResetError: Cannot write to closing transport . 2021-08-09T01:06:07.184 full_node chia.rpc.rpc_server : WARNING Sending data failed. Exception Traceback (most recent call last): File "chia\rpc\rpc_server.py", line 67, in _state_changed File "aiohttp\client_ws.py", line 150, in send_str File "aiohttp\http_websocket.py", line 687, in send File "aiohttp\http_websocket.py", line 643, in _send_frame File "aiohttp\http_websocket.py", line 660, in _write ConnectionResetError: Cannot write to closing transport . 2021-08-09T01:06:07.773 full_node chia.rpc.rpc_server : WARNING Sending data failed. Exception Traceback (most recent call last): File "chia\rpc\rpc_server.py", line 67, in _state_changed File "aiohttp\client_ws.py", line 150, in send_str File "aiohttp\http_websocket.py", line 687, in send File "aiohttp\http_websocket.py", line 643, in _send_frame File "aiohttp\http_websocket.py", line 660, in _write ConnectionResetError: Cannot write to closing transport . 2021-08-09T01:06:26.283 daemon main : ERROR Unexpected exception trying to send to websocket: code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last): File "D:\a\chia-blockchain\chia-blockchain\venv\Lib\site-packages\chia\daemon\server.py", line 187, in safe_handle File "websockets\protocol.py", line 555, in send File "websockets\protocol.py", line 803, in ensure_open websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-08-09T01:06:36.024 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T01:08:07.041 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T01:09:38.063 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T01:09:45.982 daemon main : ERROR Unexpected exception trying to send to websocket: code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last): File "websockets\protocol.py", line 838, in transfer_data asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "D:\a\chia-blockchain\chia-blockchain\venv\Lib\site-packages\chia\daemon\server.py", line 187, in safe_handle File "websockets\protocol.py", line 555, in send File "websockets\protocol.py", line 812, in ensure_open websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-08-09T01:11:09.080 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version 2021-08-09T01:12:40.098 farmer chia.farmer.farmer : ERROR Harvester did not respond. You might need to update harvester to the latest version

[ZZip3]

You might need to update harvester to the latest version

[Drakanus]

You might need to update harvester to the latest version

all are on the 1.2.3 version

[Drakanus]

debug.log

[sobertram]

Just a thought but did you make sure all your servers have the time synched to the same time server?

[github-actions[bot]]

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

[github-actions[bot]]

This issue was automatically closed because it has been flagged as stale and subsequently passed 7 days with no further activity.