search

parallel-finance / parallel

A decentralized lending & staking protocol built on top of the Polkadot ecosystem.
https://parallel.fi
GNU General Public License v3.0
119 stars 35 forks source link

Full Node does not connect to peers. #1972

Closed travnado300 closed 6 months ago

travnado300 commented 6 months ago

Full node will not connect to parachain peers. Relay chain continues to be able to connect and seemingly maintain normal functionality. Node had been running for ~4 months seamlessly until 3 days ago it needed to be restarted and has not been able to reconnect to peers.

System has been restarted, network and firewall settings have been altered, to no avail.

Verified by starting from a brand new database, different network, different device, using systemd, using docker, resetting full-node.sh script, and all yield the exact same result of connecting to relay chain peers but not parachain peers.

SixoneKui commented 6 months ago

I am experiencing the same issue.

dcolley commented 6 months ago

I can't reach any official boot nodes: https://github.com/parallel-finance/specs/blob/55e3bac2f413ca0a44227b7c6b5c4c4ae0019b4e/parallel.json

"bootNodes": [
    "/dns/bootnode-1.parallel.fi/tcp/30333/p2p/12D3KooWA3ngBk5UuAvtvANXRJSTjdCT9bz7AcdBqmnR64m3tRWb",
    "/dns/bootnode-5.parallel.fi/tcp/30333/p2p/12D3KooWGxnMnE1AFriohESDGQnpYBzbxBpRDf3MFNu8yuobJBbF"
  ],

Anyone else have the same problem, please try adding my node as follows:

--bootnodes /dns/boot.metaspan.io/tcp/16092/p2p/12D3KooWAWknffkU5uJPYSNcJrwh5GN1ofWhFCUUY5uDTaB4yLAN
travnado300 commented 6 months ago

Thank you @dcolley for resolving the issue. Issue resolved using this bootnode in chain specs and compiling from source. Devs should consider:

A) Verify security and sustainability of @dcolley private bootnode and update chain specs to include it in official bootnodes. B) Determine issue with current official bootnodes and restart.

Leaving issue open until devs provide update for resolution.

SixoneKui commented 6 months ago

Thank you @dcolley for resolving the issue. Issue resolved using this bootnode in chain specs and compiling from source. Devs should consider:

A) Verify security and sustainability of @dcolley private bootnode and update chain specs to include it in official bootnodes. B) Determine issue with current official bootnodes and restart.

Leaving issue open until devs provide update for resolution.

I tried to run with: sudo docker run -it --privileged=true -v /XXXXX:/parallel/data parallelfinance/parallel:latest --base-path=/parallel/data --chain=parallel --bootnodes="/dns/boot.metaspan.io/tcp/16092/p2p/12D3KooWAWknffkU5uJPYSNcJrwh5GN1ofWhFCUUY5uDTaB4yLAN" --name=XXXX --rpc-cors all --rpc-external --ws-port=9944 --ws-external

It seems the node is still not syncing .

Do I use it the right way.

Many thanks

nfekunprdtiunnkge commented 6 months ago

I also tried using bootnodes @dcolley provided. Sometimes I get a peer or even two peers (without bootnodes param it is always at 0) but doesn't seem to be syncing.

dcolley commented 6 months ago

My existing node is able to stop & start, and obtains ±10 peers.

However, when I start a new node it won't sync. I wonder if there is an issue with early blocks?

./parallel --chain parallel \
  --tmp \
  --database paritydb \
  -lsync=debug \
  --no-private-ip --no-mdns \
  --collator \
  --no-prometheus \
  --no-telemetry \
  --bootnodes /dns/boot.metaspan.io/tcp/16092/p2p/12D3KooWAWknffkU5uJPYSNcJrwh5GN1ofWhFCUUY5uDTaB4yLAN \
  --bootnodes /dns/boot.gatotech.network/tcp/33152/p2p/12D3KooWFnXeatXN73ytjDfcUNsoyRxLLK3tqmmzEjhbLhoGCDKt \
  --listen-addr /ip4/0.0.0.0/tcp/30339 \
  --public-addr /ip4/195.144.22.130/tcp/16095 \
  --relay-chain-rpc-urls ws://192.168.nn.nn:13015 # this is my internal polkadot rpc node

This log might be useful:

2024-05-08 16:55:07.600 DEBUG tokio-runtime-worker sync::import-queue: [Parachain] Error importing block 1: 0x51c5ee0668b3c2aa257ccf19f4e48ba8d854a45e59f654761621550bcbb5ec12: Import failed: Runtime Ethereum block not found, rejecting!

Could be related to : https://github.com/polkadot-evm/frontier/issues/223

dcolley commented 6 months ago

testing with parallel-v1.9.7 syncing starts, v2.0.2 does not sync

SixoneKui commented 6 months ago

My existing node is able to stop & start, and obtains ±10 peers.

However, when I start a new node it won't sync. I wonder if there is an issue with early blocks?

./parallel --chain parallel \
  --tmp \
  --database paritydb \
  -lsync=debug \
  --no-private-ip --no-mdns \
  --collator \
  --no-prometheus \
  --no-telemetry \
  --bootnodes /dns/boot.metaspan.io/tcp/16092/p2p/12D3KooWAWknffkU5uJPYSNcJrwh5GN1ofWhFCUUY5uDTaB4yLAN \
  --bootnodes /dns/boot.gatotech.network/tcp/33152/p2p/12D3KooWFnXeatXN73ytjDfcUNsoyRxLLK3tqmmzEjhbLhoGCDKt \
  --listen-addr /ip4/0.0.0.0/tcp/30339 \
  --public-addr /ip4/195.144.22.130/tcp/16095 \
  --relay-chain-rpc-urls ws://192.168.nn.nn:13015 # this is my internal polkadot rpc node

This log might be useful:

2024-05-08 16:55:07.600 DEBUG tokio-runtime-worker sync::import-queue: [Parachain] Error importing block 1: 0x51c5ee0668b3c2aa257ccf19f4e48ba8d854a45e59f654761621550bcbb5ec12: Import failed: Runtime Ethereum block not found, rejecting!

Could be related to : polkadot-evm/frontier#223

I have a node that was synced before, but lost sync a few days ago Then I restart this node several times, but still cannot sync, (0 peers problem occurs)

nfekunprdtiunnkge commented 6 months ago

Using both bootnodes I was able to sync again, thanks!

--bootnodes /dns/boot.metaspan.io/tcp/16092/p2p/12D3KooWAWknffkU5uJPYSNcJrwh5GN1ofWhFCUUY5uDTaB4yLAN 
--bootnodes /dns/boot.gatotech.network/tcp/33152/p2p/12D3KooWFnXeatXN73ytjDfcUNsoyRxLLK3tqmmzEjhbLhoGCDKt
dcolley commented 6 months ago

Using both bootnodes I was able to sync again, thanks!

--bootnodes /dns/boot.metaspan.io/tcp/16092/p2p/12D3KooWAWknffkU5uJPYSNcJrwh5GN1ofWhFCUUY5uDTaB4yLAN 
--bootnodes /dns/boot.gatotech.network/tcp/33152/p2p/12D3KooWFnXeatXN73ytjDfcUNsoyRxLLK3tqmmzEjhbLhoGCDKt

What version client are you using?

nfekunprdtiunnkge commented 6 months ago

Using both bootnodes I was able to sync again, thanks!

--bootnodes /dns/boot.metaspan.io/tcp/16092/p2p/12D3KooWAWknffkU5uJPYSNcJrwh5GN1ofWhFCUUY5uDTaB4yLAN 
--bootnodes /dns/boot.gatotech.network/tcp/33152/p2p/12D3KooWFnXeatXN73ytjDfcUNsoyRxLLK3tqmmzEjhbLhoGCDKt

What version client are you using?

Latest one 2.0.6, but when it went out of sync it was on v2.0.5

0xrjman commented 6 months ago

https://telemetry.polkadot.io/#/0xe61a41c53f5dcd0beb09df93b34402aada44cb05117b71059cce40a2723a4e97

🙏 Thanks, everyone! The downtime earlier was due to a disk space issue, our bootnodes is up and running again. If you still encounter the same issue, feel free to raise it again.

SixoneKui commented 6 months ago

https://telemetry.polkadot.io/#/0xe61a41c53f5dcd0beb09df93b34402aada44cb05117b71059cce40a2723a4e97

🙏 Thanks, everyone! The downtime earlier was due to a disk space issue, our bootnodes is up and running again. If you still encounter the same issue, feel free to raise it again.

Hi,

I tried 2.0.6 without specifying the bootnodes. I still cannot sync. Please take a look.

Many thanks

2024-05-13 13:49:42 [Relaychain] ✨ Imported #20756044 (0x7452…52c5)
2024-05-13 13:49:43 [Relaychain] 💤 Idle (13 peers), best: #20756044 (0x7452…52c5), finalized #20756041 (0x7f68…76f9), ⬇ 688.4kiB/s ⬆ 290.6kiB/s
2024-05-13 13:49:43 [Parachain] 💤 Idle (0 peers), best: #0 (0xe61a…4e97), finalized #0 (0xe61a…4e97), ⬇ 1.8kiB/s ⬆ 1.6kiB/s
2024-05-13 13:49:48 [Relaychain] ✨ Imported #20756045 (0xffb5…e35d)
2024-05-13 13:49:48 [Relaychain] 💤 Idle (13 peers), best: #20756045 (0xffb5…e35d), finalized #20756042 (0x1d9a…f283), ⬇ 334.4kiB/s ⬆ 217.1kiB/s
2024-05-13 13:49:48 [Parachain] 💤 Idle (0 peers), best: #0 (0xe61a…4e97), finalized #0 (0xe61a…4e97), ⬇ 0.3kiB/s ⬆ 44 B/s
2024-05-13 13:49:53 [Relaychain] 💤 Idle (15 peers), best: #20756045 (0xffb5…e35d), finalized #20756043 (0x4f43…5a7f), ⬇ 153.0kiB/s ⬆ 256.3kiB/s
2024-05-13 13:49:53 [Parachain] 💤 Idle (0 peers), best: #0 (0xe61a…4e97), finalized #0 (0xe61a…4e97), ⬇ 2.8kiB/s ⬆ 1.0kiB/s
2024-05-13 13:49:54 [Relaychain] ✨ Imported #20756046 (0xc138…30c3)
2024-05-13 13:49:58 [Relaychain] 💤 Idle (17 peers), best: #20756046 (0xc138…30c3), finalized #20756044 (0x7452…52c5), ⬇ 149.3kiB/s ⬆ 329.3kiB/s
2024-05-13 13:49:58 [Parachain] 💤 Idle (0 peers), best: #0 (0xe61a…4e97), finalized #0 (0xe61a…4e97), ⬇ 0.3kiB/s ⬆ 0.3kiB/s
2024-05-13 13:50:00 [Relaychain] ✨ Imported #20756047 (0x539f…ca5d)
2024-05-13 13:50:03 [Relaychain] 💤 Idle (16 peers), best: #20756047 (0x539f…ca5d), finalized #20756044 (0x7452…52c5), ⬇ 159.3kiB/s ⬆ 1.9MiB/s
2024-05-13 13:50:03 [Parachain] 💤 Idle (0 peers), best: #0 (0xe61a…4e97), finalized #0 (0xe61a…4e97), ⬇ 4.0kiB/s ⬆ 2.1kiB/s
2024-05-13 13:50:06 [Relaychain] ✨ Imported #20756048 (0x223d…75eb)
2024-05-13 13:50:08 [Relaychain] 💤 Idle (15 peers), best: #20756048 (0x223d…75eb), finalized #20756045 (0xffb5…e35d), ⬇ 150.5kiB/s ⬆ 1.2MiB/s
2024-05-13 13:50:08 [Parachain] 💤 Idle (0 peers), best: #0 (0xe61a…4e97), finalized #0 (0xe61a…4e97), ⬇ 0.2kiB/s ⬆ 0.4kiB/s
2024-05-13 13:50:12 [Relaychain] ✨ Imported #20756049 (0x94b8…114a)
2024-05-13 13:50:13 [Relaychain] 💤 Idle (16 peers), best: #20756049 (0x94b8…114a), finalized #20756046 (0xc138…30c3), ⬇ 472.1kiB/s ⬆ 265.7kiB/s
2024-05-13 13:50:13 [Parachain] 💤 Idle (0 peers), best: #0 (0xe61a…4e97), finalized #0 (0xe61a…4e97), ⬇ 3.5kiB/s ⬆ 1.7kiB/s ****

I still se the error mentioned by @dcolley

2024-05-14 03:46:47 [Parachain] 🔍 Discovered new external address for our node: /ip4/159.203.92.164/tcp/30333/ws/p2p/12D3KooWAkMsQJu5AcWvxcYnQkeDNk9P5HygfG36rSzBLg21GUqP
2024-05-14 03:46:48 [Parachain] 💔 Error importing block 0x51c5ee0668b3c2aa257ccf19f4e48ba8d854a45e59f654761621550bcbb5ec12: consensus error: Import failed: Runtime Ethereum block not found, rejecting!
2024-05-14 03:46:48 [Parachain] 💔 Error importing block 0x51c5ee0668b3c2aa257ccf19f4e48ba8d854a45e59f654761621550bcbb5ec12: consensus error: Import failed: Runtime Ethereum block not found, rejecting!
2024-05-14 03:46:48 [Parachain] 💔 Error importing block 0x51c5ee0668b3c2aa257ccf19f4e48ba8d854a45e59f654761621550bcbb5ec12: consensus error: Import failed: Runtime Ethereum block not found, rejecting!
2024-05-14 03:46:48 [Parachain] 💔 Error importing block 0x51c5ee0668b3c2aa257ccf19f4e48ba8d854a45e59f654761621550bcbb5ec12: consensus error: Import failed: Runtime Ethereum block not found, rejecting!
2024-05-14 03:46:48 [Parachain] 💔 Error importing block 0x51c5ee0668b3c2aa257ccf19f4e48ba8d854a45e59f654761621550bcbb5ec12: consensus error: Import failed: Runtime Ethereum block not found, rejecting!
2024-05-14 03:46:49 [Parachain] 💔 Error importing block 0x51c5ee0668b3c2aa257ccf19f4e48ba8d854a45e59f654761621550bcbb5ec12: consensus error: Import failed: Runtime Ethereum block not found, rejecting!
2024-05-14 03:46:51 [Relaychain] ⚙️ Syncing, target=#20764357 (6 peers), best: #20763490 (0xf9b1…c1ce), finalized #20763136 (0x53ef…50dc), ⬇ 7.3MiB/s ⬆ 73.6kiB/s
2024-05-14 03:46:51 [Parachain] 💤 Idle (0 peers), best: #0 (0xe61a…4e97), finalized #0 (0xe61a…4e97), ⬇ 1.6MiB/s ⬆ 6.4kiB/s
2024-05-14 03:46:54 [Parachain] 💔 Error importing block 0x51c5ee0668b3c2aa257ccf19f4e48ba8d854a45e59f654761621550bcbb5ec12: consensus error: Import failed: Runtime Ethereum block not found, rejecting!
2024-05-14 03:46:54 [Parachain] 💔 Error importing block 0x51c5ee0668b3c2aa257ccf19f4e48ba8d854a45e59f654761621550bcbb5ec12: consensus error: Import failed: Runtime Ethereum block not found, rejecting!
2024-05-14 03:46:54 [Parachain] 💔 Error importing block 0x51c5ee0668b3c2aa257ccf19f4e48ba8d854a45e59f654761621550bcbb5ec12: consensus error: Import failed: Runtime Ethereum block not found, rejecting!