search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.82k stars 2.03k forks source link

[BUG]full_node asyncio : WARNING socket.send() raised exception #9104

Closed electroboyz closed 2 years ago

electroboyz commented 3 years ago 
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "websockets\protocol.py", line 827, in transfer_data
  File "websockets\protocol.py", line 895, in read_message
  File "websockets\protocol.py", line 971, in read_data_frame
  File "websockets\protocol.py", line 1047, in read_frame
  File "websockets\framing.py", line 105, in read
  File "asyncio\streams.py", line 723, in readexactly
  File "asyncio\streams.py", line 517, in _wait_for_data
  File "asyncio\proactor_events.py", line 280, in _loop_reading
  File "asyncio\windows_events.py", line 812, in _poll
  File "asyncio\windows_events.py", line 461, in finish_recv
ConnectionResetError: [WinError 64] The specified network name is no longer available

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "Lib\site-packages\chia\daemon\server.py", line 208, in safe_handle
  File "websockets\protocol.py", line 555, in send
  File "websockets\protocol.py", line 803, in ensure_open
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-11-03T21:55:10.385 full_node asyncio                 : WARNING  socket.send() raised exception.
2021-11-03T21:55:10.417 full_node asyncio                 : WARNING  socket.send() raised exception.
2021-11-03T21:55:10.591 full_node asyncio                 : WARNING  socket.send() raised exception.
2021-11-03T21:55:12.432 full_node asyncio                 : WARNING  socket.send() raised exception.
2021-11-03T21:55:13.472 full_node asyncio                 : WARNING  socket.send() raised exception.
2021-11-03T21:55:13.629 full_node asyncio                 : WARNING  socket.send() raised exception.

Been getting this error repeatedly. Running Chia 1.2.10 on Windows 10

paulhainsworth-chia commented 3 years ago

Possibly upgrade Chia to latest websockets. Get better logging so we can determine why this is happening

brause commented 3 years ago

I am on windows with 1.2.11 . Just found this line this morning repeatedly in the logs. full_node asyncio : WARNING socket.send() raised exception. I was still synced and everything seemed to be working. Usually it just works fine. This is the first time. Tried to shut everything down from the client interface but it just sat there for more than 5 minutes... i am usually very patient with it. I went to the command line for another shutdown method. Did not work either. The logs did not record any activity except the waiting for shutdown usual. I killed all tasks with process explorer. After the client start it works again and i am synced.

I checked the logs. it started at: 2021-11-15T02:23:36.735 there were no errors in 2:20-2:23 before it started.

github-actions[bot] commented 2 years ago

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

itzando commented 2 years ago

Hi team, having the same issue here. My log file is being spammed but I cant work out what its telling me.

This is running on Windows 10, Chia client 1.2.11

It appears to start for no reason and happens several times a second.

2021-12-07T20:13:26.730 full_node asyncio : WARNING socket.send() raised exception. 2021-12-07T20:13:27.106 full_node asyncio : WARNING socket.send() raised exception. 2021-12-07T20:13:27.113 full_node asyncio : WARNING socket.send() raised exception. 2021-12-07T20:13:27.128 full_node asyncio : WARNING socket.send() raised exception. 2021-12-07T20:13:28.460 full_node asyncio : WARNING socket.send() raised exception. 2021-12-07T20:13:29.807 full_node asyncio : WARNING socket.send() raised exception.

Artra5253 commented 2 years ago 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "websockets\protocol.py", line 827, in transfer_data

  File "websockets\protocol.py", line 895, in read_message

  File "websockets\protocol.py", line 971, in read_data_frame

  File "websockets\protocol.py", line 1047, in read_frame

  File "websockets\framing.py", line 105, in read

  File "asyncio\streams.py", line 723, in readexactly

  File "asyncio\streams.py", line 517, in _wait_for_data

  File "asyncio\proactor_events.py", line 280, in _loop_reading

  File "asyncio\windows_events.py", line 812, in _poll

  File "asyncio\windows_events.py", line 461, in finish_recv

ConnectionResetError: [WinError 64] The specified network name is no longer available

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "Lib\site-packages\chia\daemon\server.py", line 208, in safe_handle

  File "websockets\protocol.py", line 555, in send

  File "websockets\protocol.py", line 803, in ensure_open

websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-11-03T21:55:10.385 full_node asyncio                 : WARNING  socket.send() raised exception.

2021-11-03T21:55:10.417 full_node asyncio                 : WARNING  socket.send() raised exception.

2021-11-03T21:55:10.591 full_node asyncio                 : WARNING  socket.send() raised exception.

2021-11-03T21:55:12.432 full_node asyncio                 : WARNING  socket.send() raised exception.

2021-11-03T21:55:13.472 full_node asyncio                 : WARNING  socket.send() raised exception.

2021-11-03T21:55:13.629 full_node asyncio                 : WARNING  socket.send() raised exception.

Been getting this error repeatedly. Running Chia 1.2.10 on Windows 10

itzando commented 2 years ago

Possibly upgrade Chia to latest websockets. Get better logging so we can determine why this is happening

Hey mate, How do I do this on Windows?

karstenbruch commented 2 years ago

Possibly upgrade Chia to latest websockets. Get better logging so we can determine why this is happening

Hey mate, How do I do this on Windows?

which one ? Upgrade or more logging ?

itzando commented 2 years ago

Possibly upgrade Chia to latest websockets. Get better logging so we can determine why this is happening

Hey mate, How do I do this on Windows?

which one ? Upgrade or more logging ?

Im on windows 10 How much logging to you want? This is it.. all the time.. and it just suddenly starts..

2021-12-17T22:02:16.464 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:16.807 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:16.808 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:17.228 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:17.230 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:17.230 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:19.029 full_node chia.full_node.full_node: INFO Finished signage point 5/64: CC: c2b564f47c3cafdbf3300c0b160f9c344bb4e990ce85cd40b581e529cbf2a7a3 RC: 4cff9db740dfed099d1988f687c8529c75f44238f8e92c06e95dd8ec12693e8c 2021-12-17T22:02:19.034 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:19.035 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:19.036 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:19.616 harvester chia.harvester.harvester: INFO 12 plots were eligible for farming 11fb4315ba... Found 0 proofs. Time: 0.57835 s. Total 7127 plots 2021-12-17T22:02:21.117 full_node chia.full_node.full_node: INFO Updated peak to height 1291478, weight 2505148848, hh f21dc0d156944ebb18e92c1351f1d5f43f7a72786a01c829b5dcf038522cdb38, forked at 1291477, rh: 3320530e10a1c538dc23d2532edaf76bafa900506f0091ec3c8a1ecbedcbde21, total iters: 4925899983893, overflow: False, deficit: 15, difficulty: 2768, sub slot iters: 135266304, Generator size: 43115, Generator ref list size: 0 2021-12-17T22:02:21.132 full_node chia.full_node.mempool_manager: INFO Size of mempool: 8 spends, cost: 1087341709 minimum fee to get in: 0 2021-12-17T22:02:21.132 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:21.132 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:21.132 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:21.195 wallet chia.wallet.wallet_blockchain: INFO Updated wallet peak to height 1291478, weight 2505148848, 2021-12-17T22:02:21.367 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:21.367 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:21.367 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:22.487 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:22.488 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:22.489 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:23.701 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:23.704 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:23.704 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:26.473 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:26.474 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:26.475 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:27.192 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:27.192 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:27.192 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:28.769 full_node chia.full_node.full_node: INFO Finished signage point 6/64: CC: 68119f96e067b433ce54d059ac51f14856004949d121019e4791ac07d25d24b5 RC: 0e088afa1c877039b2567ec7fc1d75dd16be46e953e69920c88cda1222941111 2021-12-17T22:02:28.769 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:28.769 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:28.769 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:29.262 harvester chia.harvester.harvester: INFO 22 plots were eligible for farming 11fb4315ba... Found 0 proofs. Time: 0.49294 s. Total 7127 plots 2021-12-17T22:02:31.437 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:31.437 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:31.437 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:32.132 full_node chia.full_node.full_node: INFO Updated peak to height 1291479, weight 2505151616, hh 6f61e457c39d9b813014f2d510d643da9ac42abdb735c411aa77929fb3cdaf26, forked at 1291478, rh: 1b56c94711e21ab0df58289d8bd59d058630f8129ce34965884088227f50f00e, total iters: 4925902568318, overflow: False, deficit: 14, difficulty: 2768, sub slot iters: 135266304, Generator size: No tx, Generator ref list size: No tx 2021-12-17T22:02:32.139 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:32.141 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:32.141 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:32.163 wallet chia.wallet.wallet_blockchain: INFO Updated wallet peak to height 1291479, weight 2505151616, 2021-12-17T22:02:32.457 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:32.473 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:32.473 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:33.602 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:33.602 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:33.602 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:36.296 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:36.298 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:36.299 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:37.141 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:37.144 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:37.144 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:38.177 full_node chia.full_node.full_node: INFO Finished signage point 7/64: CC: 06fbcf7a6ef50554a67324fda88016649086135c37cf0b735630df34b5223559 RC: e9737c522977db4b917f53649672801b56f5b456cae9e204a4c3295931faa42a 2021-12-17T22:02:38.193 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:38.193 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:38.193 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:38.538 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:38.538 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:38.538 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:38.721 harvester chia.harvester.harvester: INFO 14 plots were eligible for farming 11fb4315ba... Found 0 proofs. Time: 0.52830 s. Total 7127 plots 2021-12-17T22:02:40.057 full_node full_node_server : INFO Not accepting inbound connection: {'host': '14.207.53.127', 'port': 8444}.Inbound limit reached. 2021-12-17T22:02:40.244 full_node full_node_server : INFO Connection closed: 14.207.53.127, node id: ce9b29c4c90cffb625ba6886c57714189f23d8baee6ccdac82ff98cf1032c6b8 2021-12-17T22:02:40.244 full_node chia.full_node.full_node: INFO peer disconnected {'host': '14.207.53.127', 'port': 8444} 2021-12-17T22:02:41.587 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:41.589 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:41.589 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:42.545 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:42.545 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:42.545 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:46.462 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:46.462 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:46.462 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:46.732 full_node chia.full_node.full_node: INFO Finished signage point 8/64: CC: 91b4af2b662e0eacbd38c33188750dc3ed11595f8b566d04028e4c5b6253167a RC: 656b9c1f5973467f86902d5f836412b3696cf416186095b0062e0825bd2d3bed 2021-12-17T22:02:46.737 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:46.738 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:46.739 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:47.102 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:47.102 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:47.102 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:47.430 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:47.430 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:47.430 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:47.493 harvester chia.harvester.harvester: INFO 18 plots were eligible for farming 11fb4315ba... Found 0 proofs. Time: 0.75290 s. Total 7127 plots 2021-12-17T22:02:47.696 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:47.696 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:47.696 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:51.070 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:51.070 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:51.086 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:51.478 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:51.478 full_node asyncio : WARNING socket.send() raised exception. 2021-12-17T22:02:51.478 full_node asyncio : WARNING socket.send() raised exception.

karstenbruch commented 2 years ago

close the chia app and patiently wait for it to finish. And start again. That fixed it for me. If it persists restart your system after closing chia app first. Always close the chia app first before you restart. If windows is loosing patience with the chia app closing the db might be broken afterwards.

myCryptoUK commented 2 years ago

close the chia app and patiently wait for it to finish. And start again. That fixed it for me. If it persists restart your system after closing chia app first. Always close the chia app first before you restart. If windows is loosing patience with the chia app closing the db might be broken afterwards.

I did the same and it worked too. Thanks for the advice.

sbstockmeup commented 2 years ago

I'm on 1.2.11 and this warning is spamming my logs during the Dust storm right now. Anything to worry about?

itzando commented 2 years ago

close the chia app and patiently wait for it to finish. And start again. That fixed it for me. If it persists restart your system after closing chia app first. Always close the chia app first before you restart. If windows is loosing patience with the chia app closing the db might be broken afterwards.

I've done a complete resync of the DB and this still happens. When I reset it works for an hour or 2 before reappearing.. I am able to pool but like the comment above I don't know of this is something to worry about or can just be ignored.

itzando commented 2 years ago

After much testing I believe my issue is because I run a PFSense router and I managed to resolve my problem by shutting down Suricata. Hopefully this sticks. No sure what's happening or what but seems to be creating the error.

sbstockmeup commented 2 years ago

I think I found that clearing out either Wallet Nodes, Nodes way behind or Nodes at are not providing their block count stops this Warning/Error from appearing and spamming my logs. I didn't test yet one one of those three types are causing it, as I just went through and removed all those ones tonight. Give that a shot for future testers.

brause commented 2 years ago

clearing out nodes way behind will kill the ecosystem. so no, bad idea. If you can not sync up another time after downtime you will know what i mean.

Clearing out unresponding nodes(not sending block count) after a timeframe seems ok to me at least.

github-actions[bot] commented 2 years ago

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

github-actions[bot] commented 2 years ago

This issue was automatically closed because it has been flagged as stale, and subsequently passed 7 days with no further activity from the submitter or watchers.