search

node-real / bsc-erigon

Ethereum implementation on the efficiency frontier
GNU Lesser General Public License v3.0
118 stars 44 forks source link

After version upgrade node is not syncing to the tip #409

Open tofanovskiy opened 1 week ago

tofanovskiy commented 1 week ago

System information

Erigon version: 1.2.9-dev

OS & Version: Linux

Erigon Command (with flags/config): erigon --txpool.disable --metrics.addr=0.0.0.0 --log.console.verbosity=info --db.pagesize=16k --datadir /company/bsc/ --private.api.addr=localhost:9090 --chain=bsc --snapshots=true --torrent.download.rate=500mb --maxpeers 100 --downloader.verify --log.console.verbosity=info

Chain/Network: mainnet

Latest heigh which is stuck: 37,777,756

Latest logs after start: `[DBUG] [06-17|15:33:27.880] [db] open label=consensus sizeLimit=2TB pageSize=16384 [INFO] [06-17|15:33:27.881] Opening Database label=blob path=/company/bsc/blobs/blob [DBUG] [06-17|15:33:27.923] [db] open label=blob sizeLimit=2TB pageSize=4096 [INFO] [06-17|15:33:31.341] Starting private RPC server on=localhost:9090 [INFO] [06-17|15:33:31.341] new subscription to logs established [INFO] [06-17|15:33:31.357] rpc filters: subscribing to Erigon events [DBUG] [06-17|15:33:31.358] [diskutils] Implemented only for darwin [DBUG] [06-17|15:33:31.359] Establishing event subscription channel with the RPC daemon ... [INFO] [06-17|15:33:31.359] new subscription to newHeaders established [INFO] [06-17|15:33:31.360] JsonRpc endpoint opened ws=false ws.compression=true grpc=false http.url=127.0.0.1:8545 [INFO] [06-17|15:33:31.361] Reading JWT secret path=/company/bsc/jwt.hex [INFO] [06-17|15:33:31.362] HTTP endpoint opened for Engine API url=127.0.0.1:8551 ws=true ws.compression=true [DBUG] [06-17|15:33:31.364] [db] open label=sentry sizeLimit=8GB pageSize=4096 [WARN] [06-17|15:33:31.365] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered" [DBUG] [06-17|15:33:31.365] Couldn't add port mapping proto=tcp extport=30303 intport=30303 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered" [DBUG] [06-17|15:33:31.365] Couldn't add port mapping proto=udp extport=30303 intport=30303 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered" [DBUG] [06-17|15:33:31.366] QuerySeeds read nodes from the node DB count=0 [DBUG] [06-17|15:33:31.366] QuerySeeds read nodes from the node DB count=0 [INFO] [06-17|15:33:31.367] Started P2P networking version=68 self=enode://36d16e47e5cb3098b7cd5c1ef9c5eededa3b7886e03d142d98d6c4a613950b98d31f0539c14b722c65e473963c85962f2f987cfc208e6fcae9ce7cb94043bae2@127.0.0.1:30303 name=erigon/v1.2.9/linux-amd64/go1.21.6 [INFO] [06-17|15:33:31.401] [1/15 Snapshots] Requesting downloads [INFO] [06-17|15:33:31.402] [snapshots] Verify start [DBUG] [06-17|15:33:39.002] AnchorTree add anchor anchor.blockHeight=39694980 success=false [INFO] [06-17|15:33:51.403] [snapshots] Verify progress=0.44% files=50/237 sz_gb=4 [DBUG] [06-17|15:34:01.948] AnchorTree add anchor anchor.blockHeight=39694986 success=false [INFO] [06-17|15:34:11.440] [snapshots] Verify progress=0.59% files=56/237 sz_gb=5 [DBUG] [06-17|15:34:14.178] AnchorTree add anchor anchor.blockHeight=39694993 success=false [WARN] [06-17|15:34:30.704] [rpc] served conn=127.0.0.1:42796 method=/ reqid=0 t=13.551µs err="the method / does not exist/is not available" [DBUG] [06-17|15:34:31.373] [p2p] Server protocol=68 peers=6 trusted=0 inbound=0 too many peers=2 i/o timeout=10 [DBUG] [06-17|15:34:31.373] [p2p] Discovery table protocol=68 version=v4 len=70 live=0 unsol=117 ips=79 db=0 reval=4 RPC timeout=3 unsolicited reply=65 unknown node=7 [DBUG] [06-17|15:34:31.375] [p2p] Dial scheduler protocol=68 peers=6/33 tried=38 static=6 connect: connection refused=6 i/o timeout=2 [INFO] [06-17|15:34:31.487] [snapshots] Verify progress=0.75% files=68/237 sz_gb=6

`

tofanovskiy commented 1 week ago

[DBUG] [06-17|16:57:04.119] HeaderEvicted link.blockHeight=38443059 [DBUG] [06-17|16:57:04.121] justifiedNumber = 38508595, curJustifiedNumber = 38508594, header.number = 38508596, hd.highestInDb = 38508595 [DBUG] [06-17|16:57:04.121] HeaderEvicted link.blockHeight=38443060 [DBUG] [06-17|16:57:04.122] justifiedNumber = 38508596, curJustifiedNumber = 38508595, header.number = 38508597, hd.highestInDb = 38508596 [DBUG] [06-17|16:57:04.122] HeaderEvicted link.blockHeight=38443061 [DBUG] [06-17|16:57:04.124] justifiedNumber = 38508597, curJustifiedNumber = 38508596, header.number = 38508598, hd.highestInDb = 38508597 [DBUG] [06-17|16:57:04.124] HeaderEvicted link.blockHeight=38443062 [DBUG] [06-17|16:57:04.126] justifiedNumber = 38508598, curJustifiedNumber = 38508597, header.number = 38508599, hd.highestInDb = 38508598 [DBUG] [06-17|16:57:04.126] HeaderEvicted link.blockHeight=38443063 [DBUG] [06-17|16:57:04.127] justifiedNumber = 38508599, curJustifiedNumber = 38508598, header.number = 38508600, hd.highestInDb = 38508599 [DBUG] [06-17|16:57:04.127] HeaderEvicted link.blockHeight=38443064 [DBUG] [06-17|16:57:04.129] justifiedNumber = 38508600, curJustifiedNumber = 38508599, header.number = 38508601, hd.highestInDb = 38508600 [DBUG] [06-17|16:57:04.129] HeaderEvicted link.blockHeight=38443065 [DBUG] [06-17|16:57:04.131] justifiedNumber = 38508601, curJustifiedNumber = 38508600, header.number = 38508602, hd.highestInDb = 38508601 [DBUG] [06-17|16:57:04.131] HeaderEvicted link.blockHeight=38443066 [DBUG] [06-17|16:57:04.132] justifiedNumber = 38508602, curJustifiedNumber = 38508601, header.number = 38508603, hd.highestInDb = 38508602 [DBUG] [06-17|16:57:04.132] HeaderEvicted link.blockHeight=38443067 [DBUG] [06-17|16:57:04.134] justifiedNumber = 38508603, curJustifiedNumber = 38508602, header.number = 38508604, hd.highestInDb = 38508603 [DBUG] [06-17|16:57:04.134] HeaderEvicted link.blockHeight=38443068 [DBUG] [06-17|16:57:04.136] justifiedNumber = 38508604, curJustifiedNumber = 38508603, header.number = 38508605, hd.highestInDb = 38508604 [DBUG] [06-17|16:57:04.136] HeaderEvicted link.blockHeight=38443069 [DBUG] [06-17|16:57:04.137] justifiedNumber = 38508605, curJustifiedNumber = 38508604, header.number = 38508606, hd.highestInDb = 38508605 [DBUG] [06-17|16:57:04.137] HeaderEvicted link.blockHeight=38443070 [DBUG] [06-17|16:57:04.139] justifiedNumber = 38508606, curJustifiedNumber = 38508605, header.number = 38508607, hd.highestInDb = 38508606 [DBUG] [06-17|16:57:04.139] HeaderEvicted link.blockHeight=38443071 [DBUG] [06-17|16:57:04.141] justifiedNumber = 38508607, curJustifiedNumber = 38508606, header.number = 38508608, hd.highestInDb = 38508607

blxdyx commented 1 week ago

Seem normal, it syncing and need hours.

tofanovskiy commented 3 days ago

hey @blxdyx

latest logs: [DBUG] [06-25|13:20:05.998] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:06.999] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:08.000] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:09.001] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:10.002] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:11.002] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:12.003] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:13.004] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:14.005] [downloader] Request skeleton anchors=0 highestInDb=37777756 [INFO] [06-25|13:20:14.982] [2/15 Headers] No block headers to write in this log period number=37777756 blk/second=0.000 alloc=2.8GB sys=5.0GB invalidHeaders=0 rejectedBadHeaders=0 [INFO] [06-25|13:20:14.982] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [DBUG] [06-25|13:20:14.982] [downloader] Queue sizes anchors=0 links=0 persisted=65536 [DBUG] [06-25|13:20:14.982] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:15.983] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:16.984] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:17.985] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:18.985] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:19.986] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:20.987] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:21.988] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:22.989] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:23.989] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:24.991] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:25.991] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:26.992] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:27.993] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:28.994] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:29.995] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:30.995] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:31.996] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:32.997] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:33.997] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:34.998] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:35.999] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:37.000] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:38.001] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:39.002] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:40.002] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:41.003] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:42.004] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:42.868] [p2p] Discovery table protocol=68 version=v4 len=77 live=0 unsol=159 ips=97 db=0 reval=9 RPC timeout=8 unsolicited reply=93 expired=51 [DBUG] [06-25|13:20:42.868] [p2p] Server protocol=68 peers=0 trusted=0 inbound=0 i/o timeout=5 EOF=6 too many peers=8 closed by remote=10 [DBUG] [06-25|13:20:42.868] [p2p] Dial scheduler protocol=68 peers=0/33 tried=87 static=6 connect: connection refused=15 i/o timeout=7 [DBUG] [06-25|13:20:43.004] [downloader] Request skeleton anchors=0 highestInDb=37777756 [DBUG] [06-25|13:20:44.004] [downloader] Request skeleton anchors=0 highestInDb=37777756 [INFO] [06-25|13:20:44.982] [2/15 Headers] No block headers to write in this log period number=37777756 blk/second=0.000 alloc=2.7GB sys=5.0GB invalidHeaders=0 rejectedBadHeaders=0 [INFO] [06-25|13:20:44.982] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [DBUG] [06-25|13:20:44.982] [downloader] Queue sizes anchors=0 links=0 persisted=65536 [DBUG] [06-25|13:20:44.982] [downloader] Request skeleton anchors=0 highestInDb=37777756

tofanovskiy commented 3 days ago

I tried also command I found in other related issues, but it also didn't help. integration stage_headers --unwind=1000 --datadir=/company/bsc --chain=bsc

maybe the problem is 68 protocol that is used by default? @blxdyx

blxdyx commented 3 days ago

Yes, bsc only support 68 protocol now. try remove --maxpeers 100 flag

tofanovskiy commented 2 days ago

@blxdyx still the same [INFO] [06-26|15:05:18.302] [mem] memory stats Rss=27.1GB Size=0B Pss=26.9GB SharedClean=424.5MB SharedDirty=0B PrivateClean=23.5GB PrivateDirty=3.1GB Referenced=26.6GB Anonymous=3.1GB Swap=0B alloc=2.7GB sys=4.8GB [DBUG] [06-26|15:05:18.886] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:19.887] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:20.889] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:21.890] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:22.891] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:23.892] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:24.893] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:25.894] [downloader] Request skeleton anchors=0 highestInDb=37776656 [WARN] [06-26|15:05:26.862] [rpc] served conn=127.0.0.1:46304 method=/ reqid=0 t=55.842µs err="the method / does not exist/is not available" [DBUG] [06-26|15:05:26.895] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:27.896] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:28.896] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:29.896] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:30.897] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:31.898] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:32.898] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:33.899] [downloader] Request skeleton anchors=0 highestInDb=37776656 [INFO] [06-26|15:05:34.875] [2/15 Headers] No block headers to write in this log period number=37776656 blk/second=0.000 alloc=2.7GB sys=4.8GB invalidHeaders=0 rejectedBadHeaders=0 [INFO] [06-26|15:05:34.875] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [DBUG] [06-26|15:05:34.875] [downloader] Queue sizes anchors=0 links=0 persisted=65536 [DBUG] [06-26|15:05:34.875] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:35.876] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:36.877] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:37.878] [downloader] Request skeleton anchors=0 highestInDb=37776656 [DBUG] [06-26|15:05:38.879] [downloader] Request skeleton anchors=0 highestInDb=37776656