search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.82k stars 2.03k forks source link

1.1.1 Ubuntu: Not Synced #2408

Closed salamanders closed 3 years ago

salamanders commented 3 years ago
  1. Woke up to the "this will stop working at height xxx" message. Ok, no worries...
  2. Noticed that I wasn't synced since last night. Stopped around height 166101
  3. Killed my plotting.
  4. Updated to latest 1.1.1 via git.
  5. installed latest using .sh
  6. Rebooted machine
  7. Saw it go from "Not Synced" to "Synchronizing..." then back to "Not Synced"
  8. While UI was running, tried a few commands in the same . /activate terminal:

I am connected to 2 nodes (1 farmer, 1 wallet)

Questions

  1. Is this broken or just "dude, wait at least 6 hours before worrying."
  2. Is trying to connect via command line when the GUI is running a "bad thing"?
  3. If 1=Broken, Anything I should be doing to help debug?
Traceback (most recent call last):
  File "../chia/daemon/server.py", line 172, in safe_handle
    response, sockets_to_use = await self.handle_message(websocket, decoded)
  File "../chia/daemon/server.py", line 275, in handle_message
    response = await self.stop_plotting(cast(Dict[str, Any], data))
RuntimeError: coroutine raised StopIteration

2021-04-22T11:06:14.747 daemon __main__                   : ERROR    Error while handling message: Traceback (most recent call last):
  File "../chia/daemon/server.py", line 550, in stop_plotting
    config = self._get_plots_queue_item(id)
  File "../chia/daemon/server.py", line 432, in _get_plots_queue_item
    config = next(item for item in self.plots_queue if item["id"] == id)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "../chia/daemon/server.py", line 172, in safe_handle
    response, sockets_to_use = await self.handle_message(websocket, decoded)
  File "../chia/daemon/server.py", line 275, in handle_message
    response = await self.stop_plotting(cast(Dict[str, Any], data))
RuntimeError: coroutine raised StopIteration

2021-04-22T11:06:22.349 daemon __main__                   : ERROR    Error while handling message: Traceback (most recent call last):
  File "../chia/daemon/server.py", line 550, in stop_plotting
    config = self._get_plots_queue_item(id)
  File "../chia/daemon/server.py", line 432, in _get_plots_queue_item
    config = next(item for item in self.plots_queue if item["id"] == id)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "../chia/daemon/server.py", line 172, in safe_handle
    response, sockets_to_use = await self.handle_message(websocket, decoded)
  File "../chia/daemon/server.py", line 275, in handle_message
    response = await self.stop_plotting(cast(Dict[str, Any], data))
RuntimeError: coroutine raised StopIteration

2021-04-22T11:06:44.434 full_node asyncio                 : ERROR    Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f2064ad61c0>
2021-04-22T11:06:47.860 daemon asyncio                    : ERROR    Task exception was never retrieved
future: <Task finished name='Task-11241' coro=<kill_service() done, defined at ../chia/daemon/server.py:813> exception=AttributeError("'list' object has no attribute 'terminate'")>
Traceback (most recent call last):
  File "../chia/daemon/server.py", line 821, in kill_service
    result = await kill_process(process, root_path, service_name, "", delay_before_kill)
  File "../chia/daemon/server.py", line 789, in kill_process
    process.terminate()
AttributeError: 'list' object has no attribute 'terminate'
2021-04-22T11:27:30.437 full_node chia.rpc.rpc_server     : WARNING  Error while handling message: Traceback (most recent call last):
  File "/home/ME/Documents/chia-blockchain/chia/rpc/rpc_server.py", line 81, in inner
    res_object = await f(request_data)
  File "/home/ME/Documents/chia-blockchain/chia/rpc/rpc_server.py", line 161, in open_connection
    raise ValueError("Start client failed, or server is not set")
ValueError: Start client failed, or server is not set

2021-04-22T11:28:16.426 full_node chia.rpc.rpc_server     : WARNING  Error while handling message: Traceback (most recent call last):
  File "/home/ME/Documents/chia-blockchain/chia/rpc/rpc_server.py", line 81, in inner
    res_object = await f(request_data)
  File "/home/ME/Documents/chia-blockchain/chia/rpc/rpc_server.py", line 161, in open_connection
    raise ValueError("Start client failed, or server is not set")
ValueError: Start client failed, or server is not set

2021-04-22T11:29:10.427 full_node chia.rpc.rpc_server     : WARNING  Error while handling message: Traceback (most recent call last):
  File "/home/ME/Documents/chia-blockchain/chia/rpc/rpc_server.py", line 81, in inner
    res_object = await f(request_data)
  File "/home/ME/Documents/chia-blockchain/chia/rpc/rpc_server.py", line 161, in open_connection
    raise ValueError("Start client failed, or server is not set")
ValueError: Start client failed, or server is not set
dorkmo commented 3 years ago

make sure your port 8444 is forwarded to your node on your network router

salamanders commented 3 years ago

make sure your port 8444 is forwarded to your node on your network router

Just confirmed, is forwarded in wifi app config.

nowickj commented 3 years ago

I had a similar problem. I have installed a new Ubuntu 20.04 LTS, configured everything. Status was Not Synced. After few minutes it has changed for Syncing 1120/169461, and again it has changing to Not Synced. I have another server on Ubuntu too, but it has older version and it is Synced. Port 8444 is open on My router. after making chia show -a node.chia.net:8444 chia show -a 35.80.8.179:8444

syncing moovs forward.

megablue commented 3 years ago

After updating to v1.1.1 none of my windows machines are able to sync as well...

salamanders commented 3 years ago

chia show -a node.chia.net:8444 - no error chia show -a 35.80.8.179:8444 - error

"Status: Syncing 166101/169762" Then a while later goes back to "Not Synced"

salamanders commented 3 years ago

Oh freaky, chia version says 1.1.2.dev0

hoffmang9 commented 3 years ago

The 1.1.2.dev0 is a common git related issue you can ignore.

salamanders commented 3 years ago

common git related issue you can ignore -- Cool, ignoring! :)

Any thoughts on how to debug?

hoffmang9 commented 3 years ago

You need to get your router routing 8444 to your node.

salamanders commented 3 years ago

Hopefully red herring:

$ npm run electron &
[1] 9324
(venv) ME@YELLOW:~/Documents/chia-blockchain/chia-blockchain-gui$ 
> electron
> electron .

/home/ME/Documents/chia-blockchain/app.asar.unpacked/daemon
false
Error loading config - using defaults
salamanders commented 3 years ago

You need to get your router routing 8444 to your node.

Is there anything that might have increased the chances of more rare issues (like this) in the last 2 days? Over in /r/chia it seems like lots of ppl are suddenly having connection or sync issues. Or did we reach some threshold where not having 8444 routed properly suddenly has more chances of causing a glitch?

salamanders commented 3 years ago

Upped the logs to INFO:

2021-04-22T15:17:56.204 full_node full_node_server        : INFO     -> handshake to peer 90.152.200.12 4ccc801aa64da04bd4a200ff3f49e5e398062ec1d56b876787320ff7029d9305
2021-04-22T15:17:56.450 full_node full_node_server        : INFO     Connected with full_node {'host': '90.152.200.12', 'port': 8444}
2021-04-22T15:17:56.451 full_node full_node_server        : INFO     -> new_peak to peer 90.152.200.12 4ccc801aa64da04bd4a200ff3f49e5e398062ec1d56b876787320ff7029d9305
2021-04-22T15:17:56.452 full_node full_node_server        : INFO     -> request_peers to peer 90.152.200.12 4ccc801aa64da04bd4a200ff3f49e5e398062ec1d56b876787320ff7029d9305
2021-04-22T15:17:56.453 full_node full_node_server        : INFO     Connection closed: 90.152.200.12, node id: 4ccc801aa64da04bd4a200ff3f49e5e398062ec1d56b876787320ff7029d9305
2021-04-22T15:17:56.454 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '90.152.200.12', 'port': 8444}

2021-04-22T15:19:07.600 full_node full_node_server        : INFO     Cannot connect to host 85.144.8.197:8444 ssl:<ssl.SSLContext object at 0x7f2b2e46eec0> [Connect call failed ('85.144.8.197', 8444)]
2021-04-22T15:19:42.432 full_node full_node_server        : INFO     Cannot connect to host 2a01:e34:ecc3:6730:e1c0:a2dc:9c48:735d:8444 ssl:<ssl.SSLContext object at 0x7f2b351fb140> [Network is unreachable]
2021-04-22T15:21:59.387 full_node full_node_server        : INFO     Cannot connect to host 67.180.62.156:8444 ssl:<ssl.SSLContext object at 0x7f2b3457a140> [Connect call failed ('67.180.62.156', 8444)]
2021-04-22T15:22:03.155 full_node full_node_server        : INFO     Cannot connect to host 58.173.47.117:8444 ssl:<ssl.SSLContext object at 0x7f2b346a2d40> [Connect call failed ('58.173.47.117', 8444)]
2021-04-22T15:22:14.886 full_node full_node_server        : INFO     -> handshake to peer 5.135.153.16 fcf6bbaf059eeaaa9ae927d74dd867d07c19cc219c4f010327f143d737c97682
2021-04-22T15:22:15.420 full_node full_node_server        : INFO     Connected with full_node {'host': '5.135.153.16', 'port': 8444}
2021-04-22T15:22:15.421 full_node full_node_server        : INFO     -> new_peak to peer 5.135.153.16 fcf6bbaf059eeaaa9ae927d74dd867d07c19cc219c4f010327f143d737c97682
2021-04-22T15:22:15.423 full_node full_node_server        : INFO     Connection closed: 5.135.153.16, node id: fcf6bbaf059eeaaa9ae927d74dd867d07c19cc219c4f010327f143d737c97682
2021-04-22T15:22:15.423 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '5.135.153.16', 'port': 8444}
salamanders commented 3 years ago

Ok so I HAVE FIXED IT FOR JUST ME. (using @hoffmang9 's advice because of course they were right)

  1. I had previously set up forwarding in my mesh Wifi. This was working fine.
  2. I had also checked that the firewall was behaving. This was ALSO working fine.
  3. I had hoped that UPnP in my cable modem was working. this was NOT FINE.
  4. So I tried to do port forwarding from the modem to my mesh. xfinity's port forwarding is (AFAIK) very broken. I couldn't forward from the modem to my Mesh Wifi. They are trying to figure out why it is so broken.
  5. In the meantime, I moved my mesh network's router into the xFinity DMZ. This instantly fixed the issue.

I am still curious why this didn't previously hurt me, it feels like something around the same time as the 1.x release made it MUCH more important to have that port set up right.

dorkmo commented 3 years ago

i believe they said it might have to do with the huge increase in new nodes coming online. if your ports were not set up correctly before, the nodes with ports open could support a good number of others, but it became too overwhelming.

salamanders commented 3 years ago

Huh. If the port needs to be routed right, then the app should check for that and give you big blinky errors if it isn't... right?

dorkmo commented 3 years ago

that would be nice! but unfortunately it doesnt :( node communication seemed to be pretty stable a week or two ago, but there are thousands of nodes coming online everyday now. i think they were depending on upnp to keep most people connected to the network but it seems to not be reliable enough for the current reality.

hoffmang9 commented 3 years ago

The largest blockchain in the world has 85,000 nodes. We passed that a few days ago and are adding about 10,000 nodes a day. That is... ahead of expectations.

salamanders commented 3 years ago

Daaaang.

On Fri, Apr 23, 2021, 6:23 PM Gene Hoffman @.***> wrote:

The largest blockchain in the world has 85,000 nodes. We passed that a few days ago and are adding about 10,000 nodes a day. That is... ahead of expectations.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Chia-Network/chia-blockchain/issues/2408#issuecomment-826012233, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAU25WJZU2JHZ3SB6CKGJTTKIMSLANCNFSM43NCDIQQ .

andre-bender commented 3 years ago

To the moooooooon

github-actions[bot] commented 3 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.