search

maticnetwork / bor

Official repository for the Polygon Blockchain
https://polygon.technology/
GNU Lesser General Public License v3.0
1k stars 490 forks source link

Bor on Mumbai testnet does not download new blocks #1070

Closed letsbangout closed 10 months ago

letsbangout commented 10 months ago

Several days ago on three different Mumbai testnet nodes, they all had bor stop downloading new blocks. Two of them froze at height 41874052, I don't have the height from the third. On at least one of those nodes I am seeing a "bad block" message. I know how to rewind the database, and I may do that. But more concerning is that I can no longer make a new node work from scratch.

On that third node, I wiped the bor database contents under /var/lib/bor/data and restarted with a new snapshot. Same result, does not download blocks. Finally I wiped the machine and started with a new Ubuntu 20.04 LTS installation and installed the latest heimdall/bor as per your guide https://wiki.polygon.technology/docs/operate/full-node-binaries/ The snapshots downloaded correctly, and I installed the node as I have probably 50 times in the past. The following is my startup for bor. As you can see, it does not seem to download blocks. It apparently has not connected to a peer, not even once, as it reports that it is not synchronizing:

# bor attach /var/lib/bor/bor.ipc --exec eth.syncing
false

Logs are below. Please help.

Nov 05 21:33:58 polymum-hk2 systemd[1]: Started bor. Nov 05 21:33:58 polymum-hk2 bor[2333]: WARN [11-05|21:33:58.152] Config File provided, this will overwrite the cli flags path=/var/lib/bor/config.toml Nov 05 21:33:58 polymum-hk2 bor[2333]: INFO [11-05|21:33:58.153] GRPC Server started addr=[::]:3131 Nov 05 21:33:58 polymum-hk2 bor[2333]: INFO [11-05|21:33:58.154] Set global gas cap cap=50,000,000 Nov 05 21:33:58 polymum-hk2 bor[2333]: INFO [11-05|21:33:58.154] Allocated trie memory caches clean=153.00MiB dirty=256.00MiB Nov 05 21:33:58 polymum-hk2 bor[2333]: INFO [11-05|21:33:58.234] Using leveldb as the backing database Nov 05 21:33:58 polymum-hk2 bor[2333]: INFO [11-05|21:33:58.234] Allocated cache and file handles database=/var/lib/bor/data/bor/chaindata cache=512.00MiB handles=262,144 compactionTableSize=2,097,152 compactionTableSizeMultiplier=1.000 compactionTotalSize=10,485,760 compactionTotalSizeMultiplier=10.000 Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.129] Using LevelDB as the backing database Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.129] Found legacy ancient chain path location=/var/lib/bor/data/bor/chaindata/ancient Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.133] Opened ancient database database=/var/lib/bor/data/bor/chaindata/ancient readonly=false Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.341] Initialising Ethereum protocol network=80001 dbversion=8 Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.346] Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.346] --------------------------------------------------------------------------------------------------------------------------------------------------------- Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] Chain ID: 80001 (mumbai) Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] Consensus: Bor (proof-of-stake), merged from Ethash (proof-of-work) Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] Pre-Merge hard forks (block based): Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] - Homestead: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md) Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] - Tangerine Whistle (EIP 150): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md) Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] - Spurious Dragon/1 (EIP 155): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] - Spurious Dragon/2 (EIP 158): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] - Byzantium: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md) Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] - Constantinople: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md) Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] - Petersburg: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md) Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] - Istanbul: #2722000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md) Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] - Muir Glacier: #2722000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md) Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] - Berlin: #13996000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md) Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] - London: #22640000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md) Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] The Merge is not yet available for this network! Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] Post-Merge hard forks (timestamp based): Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] --------------------------------------------------------------------------------------------------------------------------------------------------------- Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.347] Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.350] Loaded most recent local header number=41,951,701 hash=5c0d02..11b8e2 td=256,640,981 age=2d22h45m Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.350] Loaded most recent local block number=41,951,574 hash=738f78..a4df28 td=256,640,219 age=2d22h50m Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.351] Loaded most recent local snap block number=41,951,701 hash=5c0d02..11b8e2 td=256,640,981 age=2d22h45m Nov 05 21:33:59 polymum-hk2 bor[2333]: WARN [11-05|21:33:59.352] Enabling snapshot recovery chainhead=41,951,574 diskbase=41,951,574 Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.358] Gasprice oracle is ignoring threshold set threshold=2 Nov 05 21:33:59 polymum-hk2 bor[2333]: WARN [11-05|21:33:59.427] Old unclean shutdowns found count=122 Nov 05 21:33:59 polymum-hk2 bor[2333]: WARN [11-05|21:33:59.427] Unclean shutdown detected booted=2022-05-12T00:08:56-0400 age=1y6mo2d Nov 05 21:33:59 polymum-hk2 bor[2333]: WARN [11-05|21:33:59.427] Unclean shutdown detected booted=2022-05-12T00:14:38-0400 age=1y6mo2d Nov 05 21:33:59 polymum-hk2 bor[2333]: WARN [11-05|21:33:59.427] Unclean shutdown detected booted=2022-05-12T00:29:40-0400 age=1y6mo2d Nov 05 21:33:59 polymum-hk2 bor[2333]: WARN [11-05|21:33:59.427] Unclean shutdown detected booted=2022-05-12T00:36:03-0400 age=1y6mo2d Nov 05 21:33:59 polymum-hk2 bor[2333]: WARN [11-05|21:33:59.427] Unclean shutdown detected booted=2022-05-12T00:45:39-0400 age=1y6mo2d Nov 05 21:33:59 polymum-hk2 bor[2333]: WARN [11-05|21:33:59.427] Unclean shutdown detected booted=2022-05-12T01:00:35-0400 age=1y6mo2d Nov 05 21:33:59 polymum-hk2 bor[2333]: WARN [11-05|21:33:59.427] Unclean shutdown detected booted=2022-05-12T01:11:09-0400 age=1y6mo2d Nov 05 21:33:59 polymum-hk2 bor[2333]: WARN [11-05|21:33:59.427] Unclean shutdown detected booted=2023-10-31T15:46:01-0400 age=5d6h47m Nov 05 21:33:59 polymum-hk2 bor[2333]: WARN [11-05|21:33:59.427] Unclean shutdown detected booted=2023-10-31T15:50:08-0400 age=5d6h43m Nov 05 21:33:59 polymum-hk2 bor[2333]: WARN [11-05|21:33:59.427] Unclean shutdown detected booted=2023-10-31T16:13:56-0400 age=5d6h20m Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.428] Enabling metrics collection Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.428] Enabling metrics export to prometheus path=http://127.0.0.1:7071/debug/metrics/prometheus Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.428] Starting peer-to-peer node instance=bor/v1.0.6/linux-amd64/go1.20.8 Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.774] New local node record seq=1,699,236,371,127 id=0861958a2c4b0a3e ip=127.0.0.1 udp=30303 tcp=30303 Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.776] IPC endpoint opened url=/var/lib/bor/bor.ipc Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.776] Started P2P networking self=enode://b7648052672721d5daddb2e79c61f00ae77afc66aaa372dc4126f5838cfc7c5b72350b289ab647b6bcef8060df6f2d80cdc862767c7ab31788785cb923258733@127.0.0.1:30303 Nov 05 21:33:59 polymum-hk2 bor[2333]: WARN [11-05|21:33:59.777] Sanitizing invalid HTTP read header timeout provided=0s updated=30s Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.777] HTTP server started endpoint=[::]:8545 auth=false prefix= cors= vhosts= Nov 05 21:33:59 polymum-hk2 bor[2333]: INFO [11-05|21:33:59.778] TxFetcher txArrivalWait=500ms Nov 05 21:34:00 polymum-hk2 bor[2333]: INFO [11-05|21:34:00.084] Got new milestone from heimdall start=42,060,620 end=42,060,644 hash=0x5a6badbcce0237c20b125cc1eb099dc97194392271d0305a8268609c3083c531 Nov 05 21:34:00 polymum-hk2 bor[2333]: WARN [11-05|21:34:00.084] unable to start the whitelist milestone service - first run err="missing blocks" Nov 05 21:34:00 polymum-hk2 bor[2333]: INFO [11-05|21:34:00.086] Got new checkpoint from heimdall start=42,058,755 end=42,059,266 rootHash=0x8bdb30740804001789d88bdbdf17ca727d45d7782008038b52ecaad178c93546 Nov 05 21:34:00 polymum-hk2 bor[2333]: WARN [11-05|21:34:00.086] Failed to whitelist checkpoint err="missing blocks" Nov 05 21:34:00 polymum-hk2 bor[2333]: WARN [11-05|21:34:00.087] unable to start the whitelist checkpoint service - first run err="missing blocks" Nov 05 21:34:03 polymum-hk2 bor[2333]: INFO [11-05|21:34:03.260] New local node record seq=1,699,236,371,128 id=0861958a2c4b0a3e ip=209.58.191.9 udp=30303 tcp=30303 Nov 05 21:34:09 polymum-hk2 bor[2333]: INFO [11-05|21:34:09.778] Looking for peers peercount=1 tried=181 static=0 Nov 05 21:34:12 polymum-hk2 bor[2333]: INFO [11-05|21:34:12.088] Got new milestone from heimdall start=42,060,620 end=42,060,644 hash=0x5a6badbcce0237c20b125cc1eb099dc97194392271d0305a8268609c3083c531 Nov 05 21:34:12 polymum-hk2 bor[2333]: WARN [11-05|21:34:12.088] unable to handle whitelist milestone err="missing blocks" Nov 05 21:34:20 polymum-hk2 bor[2333]: INFO [11-05|21:34:20.090] Looking for peers peercount=0 tried=66 static=0 Nov 05 21:34:22 polymum-hk2 bor[2333]: WARN [11-05|21:34:22.530] Snapshot extension registration failed peer=04199d53 err="peer connected on snap without compatible eth support" Nov 05 21:34:24 polymum-hk2 bor[2333]: INFO [11-05|21:34:24.087] Got new milestone from heimdall start=42,060,620 end=42,060,644 hash=0x5a6badbcce0237c20b125cc1eb099dc97194392271d0305a8268609c3083c531 Nov 05 21:34:24 polymum-hk2 bor[2333]: WARN [11-05|21:34:24.088] unable to handle whitelist milestone err="missing blocks" Nov 05 21:34:30 polymum-hk2 bor[2333]: INFO [11-05|21:34:30.302] Looking for peers peercount=0 tried=105 static=0 Nov 05 21:34:36 polymum-hk2 bor[2333]: INFO [11-05|21:34:36.087] Got new milestone from heimdall start=42,060,620 end=42,060,644 hash=0x5a6badbcce0237c20b125cc1eb099dc97194392271d0305a8268609c3083c531 Nov 05 21:34:36 polymum-hk2 bor[2333]: WARN [11-05|21:34:36.087] unable to handle whitelist milestone err="missing blocks" Nov 05 21:34:40 polymum-hk2 bor[2333]: INFO [11-05|21:34:40.312] Looking for peers peercount=1 tried=81 static=0 Nov 05 21:34:48 polymum-hk2 bor[2333]: INFO [11-05|21:34:48.088] Got new milestone from heimdall start=42,060,645 end=42,060,666 hash=0x6b4cae1db1fe407a4f12c4f0b4c7e3872461e212b86a17c9a8ef3fd2fa0262de Nov 05 21:34:48 polymum-hk2 bor[2333]: WARN [11-05|21:34:48.088] unable to handle whitelist milestone err="missing blocks" Nov 05 21:34:50 polymum-hk2 bor[2333]: INFO [11-05|21:34:50.541] Looking for peers peercount=1 tried=120 static=0 Nov 05 21:35:00 polymum-hk2 bor[2333]: INFO [11-05|21:35:00.087] Got new milestone from heimdall start=42,060,645 end=42,060,666 hash=0x6b4cae1db1fe407a4f12c4f0b4c7e3872461e212b86a17c9a8ef3fd2fa0262de Nov 05 21:35:00 polymum-hk2 bor[2333]: WARN [11-05|21:35:00.087] unable to handle whitelist milestone err="missing blocks" Nov 05 21:35:00 polymum-hk2 bor[2333]: INFO [11-05|21:35:00.554] Looking for peers peercount=1 tried=174 static=0 Nov 05 21:35:10 polymum-hk2 bor[2333]: INFO [11-05|21:35:10.920] Looking for peers peercount=1 tried=91 static=0 Nov 05 21:35:12 polymum-hk2 bor[2333]: INFO [11-05|21:35:12.090] Got new milestone from heimdall start=42,060,645 end=42,060,666 hash=0x6b4cae1db1fe407a4f12c4f0b4c7e3872461e212b86a17c9a8ef3fd2fa0262de Nov 05 21:35:12 polymum-hk2 bor[2333]: WARN [11-05|21:35:12.090] unable to handle whitelist milestone err="missing blocks" Nov 05 21:35:20 polymum-hk2 bor[2333]: INFO [11-05|21:35:20.977] Looking for peers peercount=1 tried=118 static=0 Nov 05 21:35:24 polymum-hk2 bor[2333]: INFO [11-05|21:35:24.087] Got new milestone from heimdall start=42,060,667 end=42,060,686 hash=0xfa2ed8de1356e7eb005a539469879e2948e3fdd77d774335941fce4a97b5efec Nov 05 21:35:24 polymum-hk2 bor[2333]: WARN [11-05|21:35:24.087] unable to handle whitelist milestone err="missing blocks" Nov 05 21:35:31 polymum-hk2 bor[2333]: INFO [11-05|21:35:31.034] Looking for peers peercount=1 tried=21 static=0 Nov 05 21:35:36 polymum-hk2 bor[2333]: INFO [11-05|21:35:36.088] Got new milestone from heimdall start=42,060,667 end=42,060,686 hash=0xfa2ed8de1356e7eb005a539469879e2948e3fdd77d774335941fce4a97b5efec Nov 05 21:35:36 polymum-hk2 bor[2333]: WARN [11-05|21:35:36.088] unable to handle whitelist milestone err="missing blocks" Nov 05 21:35:40 polymum-hk2 bor[2333]: INFO [11-05|21:35:40.091] Got new checkpoint from heimdall start=42,058,755 end=42,059,266 rootHash=0x8bdb30740804001789d88bdbdf17ca727d45d7782008038b52ecaad178c93546 Nov 05 21:35:40 polymum-hk2 bor[2333]: WARN [11-05|21:35:40.091] Failed to whitelist checkpoint err="missing blocks" Nov 05 21:35:40 polymum-hk2 bor[2333]: WARN [11-05|21:35:40.091] unable to handle whitelist checkpoint err="missing blocks" Nov 05 21:35:41 polymum-hk2 bor[2333]: INFO [11-05|21:35:41.182] Looking for peers peercount=1 tried=154 static=0 Nov 05 21:35:48 polymum-hk2 bor[2333]: INFO [11-05|21:35:48.088] Got new milestone from heimdall start=42,060,667 end=42,060,686 hash=0xfa2ed8de1356e7eb005a539469879e2948e3fdd77d774335941fce4a97b5efec Nov 05 21:35:48 polymum-hk2 bor[2333]: WARN [11-05|21:35:48.088] unable to handle whitelist milestone err="missing blocks" Nov 05 21:35:51 polymum-hk2 bor[2333]: INFO [11-05|21:35:51.211] Looking for peers peercount=2 tried=107 static=0 Nov 05 21:36:00 polymum-hk2 bor[2333]: INFO [11-05|21:36:00.087] Got new milestone from heimdall start=42,060,667 end=42,060,686 hash=0xfa2ed8de1356e7eb005a539469879e2948e3fdd77d774335941fce4a97b5efec Nov 05 21:36:00 polymum-hk2 bor[2333]: WARN [11-05|21:36:00.088] unable to handle whitelist milestone err="missing blocks" Nov 05 21:36:00 polymum-hk2 bor[2333]: WARN [11-05|21:36:00.493] Snapshot extension registration failed peer=04199d53 err="peer connected on snap without compatible eth support" Nov 05 21:36:09 polymum-hk2 bor[2333]: WARN [11-05|21:36:09.998] Snapshot extension registration failed peer=1fadfc83 err="peer connected on snap without compatible eth support" Nov 05 21:36:11 polymum-hk2 bor[2333]: INFO [11-05|21:36:11.299] Looking for peers peercount=1 tried=119 static=0 Nov 05 21:36:12 polymum-hk2 bor[2333]: INFO [11-05|21:36:12.090] Got new milestone from heimdall start=42,060,667 end=42,060,686 hash=0xfa2ed8de1356e7eb005a539469879e2948e3fdd77d774335941fce4a97b5efec Nov 05 21:36:12 polymum-hk2 bor[2333]: WARN [11-05|21:36:12.090] unable to handle whitelist milestone err="missing blocks" Nov 05 21:36:21 polymum-hk2 bor[2333]: INFO [11-05|21:36:21.353] Looking for peers peercount=1 tried=87 static=0 Nov 05 21:36:24 polymum-hk2 bor[2333]: INFO [11-05|21:36:24.091] Got new milestone from heimdall start=42,060,667 end=42,060,686 hash=0xfa2ed8de1356e7eb005a539469879e2948e3fdd77d774335941fce4a97b5efec Nov 05 21:36:24 polymum-hk2 bor[2333]: WARN [11-05|21:36:24.091] unable to handle whitelist milestone err="missing blocks" Nov 05 21:36:24 polymum-hk2 bor[2333]: WARN [11-05|21:36:24.233] Snapshot extension registration failed peer=b0c6a97a err="peer connected on snap without compatible eth support" Nov 05 21:36:26 polymum-hk2 bor[2333]: WARN [11-05|21:36:26.041] Snapshot extension registration failed peer=61e34ab3 err="peer connected on snap without compatible eth support" Nov 05 21:36:31 polymum-hk2 bor[2333]: INFO [11-05|21:36:31.565] Looking for peers peercount=2 tried=87 static=0 Nov 05 21:36:36 polymum-hk2 bor[2333]: INFO [11-05|21:36:36.087] Got new milestone from heimdall start=42,060,667 end=42,060,686 hash=0xfa2ed8de1356e7eb005a539469879e2948e3fdd77d774335941fce4a97b5efec Nov 05 21:36:36 polymum-hk2 bor[2333]: WARN [11-05|21:36:36.087] unable to handle whitelist milestone err="missing blocks" Nov 05 21:36:41 polymum-hk2 bor[2333]: INFO [11-05|21:36:41.645] Looking for peers peercount=2 tried=150 static=0 Nov 05 21:36:46 polymum-hk2 bor[2333]: WARN [11-05|21:36:46.087] an error while trying fetching from Heimdall path=/milestone/lastNoAck attempt=1 error="Get \"http://localhost:1317/milestone/lastNoAck\": context deadline exceeded" Nov 05 21:36:46 polymum-hk2 bor[2333]: INFO [11-05|21:36:46.087] Retrying again in 5 seconds to fetch data from Heimdall path=/milestone/lastNoAck attempt=1 Nov 05 21:36:46 polymum-hk2 bor[2333]: ERROR[11-05|21:36:46.087] Failed to fetch latest no-ack milestone err="context deadline exceeded" Nov 05 21:36:46 polymum-hk2 bor[2333]: WARN [11-05|21:36:46.087] unable to handle no-ack-milestone service err="failed to fetch latest milestone" Nov 05 21:36:51 polymum-hk2 bor[2333]: WARN [11-05|21:36:51.191] Snapshot extension registration failed peer=8faf3e54 err="peer connected on snap without compatible eth support" Nov 05 21:36:51 polymum-hk2 bor[2333]: INFO [11-05|21:36:51.924] Looking for peers peercount=2 tried=95 static=0 Nov 05 21:36:52 polymum-hk2 bor[2333]: WARN [11-05|21:36:52.087] an error while trying fetching from Heimdall path=/milestone/lastNoAck attempt=1 error="Get \"http://localhost:1317/milestone/lastNoAck\": context deadline exceeded" Nov 05 21:36:52 polymum-hk2 bor[2333]: INFO [11-05|21:36:52.087] Retrying again in 5 seconds to fetch data from Heimdall path=/milestone/lastNoAck attempt=1 Nov 05 21:36:52 polymum-hk2 bor[2333]: ERROR[11-05|21:36:52.087] Failed to fetch latest no-ack milestone err="context deadline exceeded" Nov 05 21:36:52 polymum-hk2 bor[2333]: WARN [11-05|21:36:52.087] unable to handle no-ack-milestone service err="failed to fetch latest milestone" Nov 05 21:36:53 polymum-hk2 bor[2333]: WARN [11-05|21:36:53.086] an error while trying fetching from Heimdall path=/milestone/latest attempt=1 error="Get \"http://localhost:1317/milestone/latest\": context deadline exceeded" Nov 05 21:36:53 polymum-hk2 bor[2333]: INFO [11-05|21:36:53.086] Retrying again in 5 seconds to fetch data from Heimdall path=/milestone/latest attempt=1 Nov 05 21:36:58 polymum-hk2 bor[2333]: WARN [11-05|21:36:58.087] an error while trying fetching from Heimdall path=/milestone/lastNoAck attempt=1 error="Get \"http://localhost:1317/milestone/lastNoAck\": context deadline exceeded" Nov 05 21:36:58 polymum-hk2 bor[2333]: INFO [11-05|21:36:58.087] Retrying again in 5 seconds to fetch data from Heimdall path=/milestone/lastNoAck attempt=1 Nov 05 21:36:58 polymum-hk2 bor[2333]: ERROR[11-05|21:36:58.087] Failed to fetch latest no-ack milestone err="context deadline exceeded" Nov 05 21:36:58 polymum-hk2 bor[2333]: WARN [11-05|21:36:58.088] unable to handle no-ack-milestone service err="failed to fetch latest milestone" Nov 05 21:37:01 polymum-hk2 bor[2333]: INFO [11-05|21:37:01.930] Looking for peers peercount=3 tried=95 static=0 Nov 05 21:37:03 polymum-hk2 bor[2333]: INFO [11-05|21:37:03.088] Retrying again in 5 seconds to fetch data from Heimdall path=/milestone/latest attempt=2 Nov 05 21:37:04 polymum-hk2 bor[2333]: WARN [11-05|21:37:04.087] an error while trying fetching from Heimdall path=/milestone/lastNoAck attempt=1 error="Get \"http://localhost:1317/milestone/lastNoAck\": context deadline exceeded" Nov 05 21:37:04 polymum-hk2 bor[2333]: INFO [11-05|21:37:04.088] Retrying again in 5 seconds to fetch data from Heimdall path=/milestone/lastNoAck attempt=1 Nov 05 21:37:04 polymum-hk2 bor[2333]: ERROR[11-05|21:37:04.088] Failed to fetch latest no-ack milestone err="context deadline exceeded" Nov 05 21:37:04 polymum-hk2 bor[2333]: WARN [11-05|21:37:04.088] unable to handle no-ack-milestone service err="failed to fetch latest milestone" Nov 05 21:37:08 polymum-hk2 bor[2333]: INFO [11-05|21:37:08.089] Retrying again in 5 seconds to fetch data from Heimdall path=/milestone/latest attempt=3 Nov 05 21:37:10 polymum-hk2 bor[2333]: WARN [11-05|21:37:10.087] an error while trying fetching from Heimdall path=/milestone/lastNoAck attempt=1 error="Get \"http://localhost:1317/milestone/lastNoAck\": context deadline exceeded" Nov 05 21:37:10 polymum-hk2 bor[2333]: INFO [11-05|21:37:10.087] Retrying again in 5 seconds to fetch data from Heimdall path=/milestone/lastNoAck attempt=1 Nov 05 21:37:10 polymum-hk2 bor[2333]: ERROR[11-05|21:37:10.087] Failed to fetch latest no-ack milestone err="context deadline exceeded" Nov 05 21:37:10 polymum-hk2 bor[2333]: WARN [11-05|21:37:10.087] unable to handle no-ack-milestone service err="failed to fetch latest milestone" Nov 05 21:37:11 polymum-hk2 bor[2333]: INFO [11-05|21:37:11.446] Got new milestone from heimdall start=42,060,667 end=42,060,686 hash=0xfa2ed8de1356e7eb005a539469879e2948e3fdd77d774335941fce4a97b5efec Nov 05 21:37:11 polymum-hk2 bor[2333]: WARN [11-05|21:37:11.447] unable to handle whitelist milestone err="missing blocks" Nov 05 21:37:11 polymum-hk2 bor[2333]: INFO [11-05|21:37:11.489] Got new milestone from heimdall start=42,060,687 end=42,060,707 hash=0x2f1648214e18ba92b8ed0643397127d8c69e9b4b5cc1dab8dae68546f9f55eb9 Nov 05 21:37:11 polymum-hk2 bor[2333]: WARN [11-05|21:37:11.489] unable to handle whitelist milestone err="missing blocks" Nov 05 21:37:11 polymum-hk2 bor[2333]: INFO [11-05|21:37:11.939] Looking for peers peercount=2 tried=110 static=0 Nov 05 21:37:12 polymum-hk2 bor[2333]: INFO [11-05|21:37:12.087] Got new milestone from heimdall start=42,060,687 end=42,060,707 hash=0x2f1648214e18ba92b8ed0643397127d8c69e9b4b5cc1dab8dae68546f9f55eb9 Nov 05 21:37:12 polymum-hk2 bor[2333]: WARN [11-05|21:37:12.087] unable to handle whitelist milestone err="missing blocks" Nov 05 21:37:20 polymum-hk2 bor[2333]: INFO [11-05|21:37:20.091] Got new checkpoint from heimdall start=42,059,267 end=42,060,034 rootHash=0xdd4e9e7df53635fe7428a71cef88b2b41a5168e87be5ff2738f391cccf98f091 Nov 05 21:37:20 polymum-hk2 bor[2333]: WARN [11-05|21:37:20.091] Failed to whitelist checkpoint err="missing blocks" Nov 05 21:37:20 polymum-hk2 bor[2333]: WARN [11-05|21:37:20.091] unable to handle whitelist checkpoint err="missing blocks" Nov 05 21:37:22 polymum-hk2 bor[2333]: INFO [11-05|21:37:22.166] Looking for peers peercount=1 tried=150 static=0 Nov 05 21:37:24 polymum-hk2 bor[2333]: INFO [11-05|21:37:24.087] Got new milestone from heimdall start=42,060,687 end=42,060,707 hash=0x2f1648214e18ba92b8ed0643397127d8c69e9b4b5cc1dab8dae68546f9f55eb9 Nov 05 21:37:24 polymum-hk2 bor[2333]: WARN [11-05|21:37:24.088] unable to handle whitelist milestone err="missing blocks" Nov 05 21:37:30 polymum-hk2 bor[2333]: WARN [11-05|21:37:30.639] Snapshot extension registration failed peer=05180ebe err="peer connected on snap without compatible eth support" Nov 05 21:37:32 polymum-hk2 bor[2333]: INFO [11-05|21:37:32.166] Looking for peers peercount=1 tried=91 static=0 Nov 05 21:37:36 polymum-hk2 bor[2333]: INFO [11-05|21:37:36.087] Got new milestone from heimdall start=42,060,708 end=42,060,728 hash=0xec3e7cece7219d68543397d4abe033e737ad6c241f681cacb63a835719758003 Nov 05 21:37:36 polymum-hk2 bor[2333]: WARN [11-05|21:37:36.088] unable to handle whitelist milestone err="missing blocks" Nov 05 21:37:42 polymum-hk2 bor[2333]: INFO [11-05|21:37:42.304] Looking for peers peercount=1 tried=131 static=0 Nov 05 21:37:48 polymum-hk2 bor[2333]: INFO [11-05|21:37:48.087] Got new milestone from heimdall start=42,060,708 end=42,060,728 hash=0xec3e7cece7219d68543397d4abe033e737ad6c241f681cacb63a835719758003 Nov 05 21:37:48 polymum-hk2 bor[2333]: WARN [11-05|21:37:48.088] unable to handle whitelist milestone err="missing blocks" Nov 05 21:38:00 polymum-hk2 bor[2333]: INFO [11-05|21:38:00.088] Got new milestone from heimdall start=42,060,729 end=42,060,742 hash=0x7e3cf6d816f3da0a6241544002b102e23471e2347742853cbd5a108e894d65d1 Nov 05 21:38:00 polymum-hk2 bor[2333]: WARN [11-05|21:38:00.088] unable to handle whitelist milestone err="missing blocks" Nov 05 21:38:02 polymum-hk2 bor[2333]: INFO [11-05|21:38:02.608] Looking for peers peercount=1 tried=105 static=0 Nov 05 21:38:05 polymum-hk2 bor[2333]: WARN [11-05|21:38:05.091] Snapshot extension registration failed peer=c1e0a49a err="peer connected on snap without compatible eth support" Nov 05 21:38:05 polymum-hk2 bor[2333]: WARN [11-05|21:38:05.325] Snapshot extension registration failed peer=c1e77d06 err="peer connected on snap without compatible eth support" Nov 05 21:38:12 polymum-hk2 bor[2333]: INFO [11-05|21:38:12.090] Got new milestone from heimdall start=42,060,729 end=42,060,742 hash=0x7e3cf6d816f3da0a6241544002b102e23471e2347742853cbd5a108e894d65d1 Nov 05 21:38:12 polymum-hk2 bor[2333]: WARN [11-05|21:38:12.091] unable to handle whitelist milestone err="missing blocks" Nov 05 21:38:12 polymum-hk2 bor[2333]: INFO [11-05|21:38:12.910] Looking for peers peercount=1 tried=131 static=0 Nov 05 21:38:22 polymum-hk2 bor[2333]: INFO [11-05|21:38:22.925] Looking for peers peercount=1 tried=160 static=0 Nov 05 21:38:24 polymum-hk2 bor[2333]: INFO [11-05|21:38:24.088] Got new milestone from heimdall start=42,060,729 end=42,060,742 hash=0x7e3cf6d816f3da0a6241544002b102e23471e2347742853cbd5a108e894d65d1 Nov 05 21:38:24 polymum-hk2 bor[2333]: WARN [11-05|21:38:24.088] unable to handle whitelist milestone err="missing blocks"

0xsajal commented 10 months ago

Can you please tell me what set of peers you are using?

letsbangout commented 10 months ago

I don't understand the question. As I said:

It apparently has not connected to a peer, not even once, as it reports that it is not synchronizing

I have configured the node with the set of seeds from the guide mentioned above. If there is something specifically you would like me to check, please let me know what, and how.

0xsajal commented 10 months ago

As I can see your node is connected to a single peer, I need the list of peers you have entered in your config file. This line states so:

Nov 05 21:38:12 polymum-hk2 bor[2333]: INFO [11-05|21:38:12.910] Looking for peers peercount=1 tried=131 static=0

Also, it would be better if you can share your bor config file here once?

letsbangout commented 10 months ago

You did not specify how you wanted me to find a peer list. So to be productive I've generated a list of established sockets like so:

~# netstat -n|grep ESTAB|awk {'print $5'} 34.29.204.22:26656 15.235.43.227:30665 107.6.91.43:18212 127.0.0.1:40288 107.6.113.60:18212 143.110.185.126:26656 15.165.109.50:26656 202.61.229.79:26656 43.201.135.93:26656 127.0.0.1:26657 157.90.106.242:36656 52.203.92.134:26656 18.189.127.76:26656 54.211.234.44:26656 64.25.109.179:26656 195.191.219.107:30303 152.228.132.72:26656 176.9.148.15:26656 103.141.171.130:26656 54.170.47.95:26656 147.135.31.203:26656 3.39.35.120:26656 158.101.119.64:30303 15.165.98.61:26656 127.0.0.1:1317 35.197.198.100:26656 54.229.32.80:26656 15.235.14.210:31304 65.109.106.6:30304 34.198.90.97:26656 54.224.89.242:26656 129.80.28.81:30303 54.194.216.201:26656 52.214.83.78:26656 35.188.32.51:26656 34.240.114.47:26656 54.155.37.4:52490 64.120.89.172:40040 64.120.89.172:40050 51.81.244.9:60972 173.0.155.149:43653 65.109.102.141:45916 152.69.217.151:57044 46.4.49.28:41994 65.108.231.124:55316 54.208.58.200:33196 38.95.13.166:49708 5.9.8.151:38178 178.32.176.187:44864 64.120.89.172:46582 54.39.18.77:35420 121.165.96.3:53378 168.119.36.107:40420 144.24.151.219:40674 129.159.136.96:52410 127.0.0.1:36118 108.52.90.30:37160 222.252.98.127:45352 176.107.161.32:36032 65.75.210.202:49670 15.165.142.140:29337 34.123.193.237:39264 64.120.89.172:44384 54.254.161.27:52722

And the config:

~# cat /var/lib/bor/config.toml
chain = "mumbai"
# identity = "node_name"
# verbosity = 3
# vmdebug = false
datadir = "/var/lib/bor/data"
# ancient = ""
# db.engine = "leveldb"
# keystore = ""
# "rpc.batchlimit" = 100
# "rpc.returndatalimit" = 100000
syncmode = "full"
# gcmode = "full"
# snapshot = true
# ethstats = ""
# devfakeauthor = false

# ["eth.requiredblocks"]

# [log]
    # vmodule = ""
    # json = false
    # backtrace = ""
    # debug = true

[p2p]
    maxpeers = 50
    port = 30303
    # maxpendpeers = 50
    # bind = "0.0.0.0"
    # nodiscover = false
    # nat = "any"
    # netrestrict = ""
    # nodekey = ""
    # nodekeyhex = ""
    # txarrivalwait = "500ms"
  [p2p.discovery]
        # v5disc = false
    bootnodes = ["enode://bdcd4786a616a853b8a041f53496d853c68d99d54ff305615cd91c03cd56895e0a7f6e9f35dbf89131044e2114a9a782b792b5661e3aff07faf125a98606a071@43.200.206.40:30303", "enode://209aaf7ed549cf4a5700fd833da25413f80a1248bd3aa7fe2a87203e3f7b236dd729579e5c8df61c97bf508281bae4969d6de76a7393bcbd04a0af70270333b3@54.216.248.9:30303"]
        # bootnodesv4 = []
        # bootnodesv5 = []
        # static-nodes = []
        # trusted-nodes = []
        # dns = []

# [heimdall]
    # url = "http://localhost:1317"
    # "bor.without" = false
    # grpc-address = ""

[txpool]
    nolocals = true
    accountslots = 16
    globalslots = 131072
    accountqueue = 64
    globalqueue = 131072
    lifetime = "1h30m0s"
    # locals = []
    # journal = ""
    # rejournal = "1h0m0s"
    # pricelimit = 1
    # pricebump = 10

[miner]
    gaslimit = 30000000
    # gasprice = "1000000000"
    # mine = false
    # etherbase = ""
    # extradata = ""
    # recommit = "2m5s"
    # commitinterrupt = true

[jsonrpc]
    ipcpath = "/var/lib/bor/bor.ipc"
    # ipcdisable = false
    # gascap = 50000000
    # evmtimeout = "5s"
    # txfeecap = 5.0
    # allow-unprotected-txs = false
    # enabledeprecatedpersonal = false
    [jsonrpc.http]
        enabled = true
        port = 8545
        host = "0.0.0.0"
        api = ["eth", "net", "web3", "txpool", "bor"]
        vhosts = ["*"]
        corsdomain = ["*"]
        # prefix = ""
        # ep-size = 40
        # ep-requesttimeout = "0s"
    # [jsonrpc.ws]
        # enabled = false
        # port = 8546
        # prefix = ""
        # host = "localhost"
        # api = ["web3", "net"]
        # origins = ["*"]
        # ep-size = 40
        # ep-requesttimeout = "0s"
    # [jsonrpc.graphql]
        # enabled = false
        # port = 0
        # prefix = ""
        # host = ""
        # vhosts = ["*"]
        # corsdomain = ["*"]
    # [jsonrpc.auth]
        # jwtsecret = ""
        # addr = "localhost"
        # port = 8551
        # vhosts = ["localhost"]
    # [jsonrpc.timeouts]
        # read = "10s"
        # write = "30s"
        # idle = "2m0s"

# [gpo]
    # blocks = 20
    # percentile = 60
    # maxheaderhistory = 1024
    # maxblockhistory = 1024
    # maxprice = "5000000000000"
    # ignoreprice = "2"

[telemetry]
    metrics = true
    # expensive = false
    # prometheus-addr = ""
    # opencollector-endpoint = ""
    # [telemetry.influx]
        # influxdb = false
        # endpoint = ""
        # database = ""
        # username = ""
        # password = ""
        # influxdbv2 = false
        # token = ""
        # bucket = ""
        # organization = ""
    # [telemetry.influx.tags]

# [cache]
    # cache = 1024
    # gc = 25
    # snapshot = 10
    # database = 50
    # trie = 15
    # journal = "triecache"
    # rejournal = "1h0m0s"
    # noprefetch = false
    # preimages = false
    # txlookuplimit = 2350000
    # timeout = "1h0m0s"
    # fdlimit = 0

# [accounts]
    # unlock = []
    # password = ""
    # allow-insecure-unlock = false
    # lightkdf = false
    # disable-bor-wallet = false

# [grpc]
    # addr = ":3131"

# [developer]
    # dev = false
    # period = 0
    # gaslimit = 11500000

# [pprof]
#   pprof = false
#   port = 6060
#   addr = "127.0.0.1"
#   memprofilerate = 524288
#   blockprofilerate = 0
letsbangout commented 10 months ago

Could this be related to a hardfork on Mumbai?

https://github.com/maticnetwork/bor/releases/tag/v1.1.0-beta3

Where and how are such hardforks announced? Please don't say the Polygon Discord. Does this project have a forum where such things are always posted? An RSS feed? An email list?

0xsajal commented 10 months ago

So v1.1.0-beta2 was the hardfork patch. If not upgraded yet, I would request you to upgrade it and check. You can use v1.1.0-beta4 for bor. Also please use Heimdall v1.0.3.

Ofcourse we announce it on discord space, but there is forum where we announce about the latest upgrades. Here is the link: https://forum.polygon.technology/

letsbangout commented 10 months ago

Ok so before I address this, I have a critical commentary on your system of announcements, and I hope that you will take note and communicate this to those in the Polygon organization who can make a change:

Things seem very inconsistent on your forum. For example, recently there was a hardfork on the ZKEVM mainnet, which was announced in the "Zero knowledge" section of the forum. But it was not announced in the "Announcements" forum section. Most shockingly, there was absolutely NO mention of a hard-fork in the ChangeLog for the relevant zkevm-node releases on github (0.3.3, 0.3.4):

https://github.com/0xPolygonHermez/zkevm-node/releases

Many people "subscribe" to the feed of releases on GitHub to get their news about such hardforks. And in the example above, even subscribing would not have alerted the sysadmin to an impending hardfork for which a code upgrade would become mandatory.

Furthermore, your main Polygon Discord has a horrible and stupid "verification" system. It requires that a natural person link a crypto wallet and sign a transaction in order to join the Discord. I find this to be against the ethos of crypto and crypto projects in general, because 1) participants may not own crypto, or may not wish to reveal their wallet to you or to third parties, 2) it requires that contributors PAY MONEY to acquire crypto in the first place, and 3) the mere fact that someone submits a crypto wallet for "verification" does NOT mean they aren't a bot, or a previously banned user, or any number of other things. There seems to be no way around this verification, no way to provide "credentials" or "evidence" of contributions in the crypto space. You could be an original developer of bitcoind since 2010 and still not be able to join your Polygon Discord chat.

So I recommend that you guys take these criticisms to heart, make changes to your Discord verification system so I can join, and do a better job of documenting significant service-impacting changes like hardforks as I have outlined above.

As for this issue of Mumbai not pulling in new blocks, that does seem to be related to the hardfork (which did seem properly announced on the forums). I have pulled the required code and reset my databases, and I can now pull in new blocks past the point of the hardfork. On this basis you can mark this issue closed. Thank you for your help and your consideration of the above criticisms.

0xsajal commented 10 months ago

Noted, will pass on the feedback. Thanks!