Closed morzhanov closed 9 months ago
Hello, thanks for the detailed description of the problem. It helps a lot :) Let me try to reproduce the issue in my own environment.
Btw, you're setting --cache
to 16000 which is ~15.64GB, while your system's RAM is only 16GB. Try setting cache to a lower amount like 5000 and sync again.
Btw, you're setting
--cache
to 16000 which is ~15.64GB, while your system's RAM is only 16GB. Try setting cache to a lower amount like 5000 and sync again.
Thanks for pointing this out! I'm using arguments from my previous node that had 16cpu/32gb. I reduced cache value to 8000 as I see that c5n.2xlarge ec2 instance has 20gb ram (I updated issue description).
BTW before the update with 16gb cache htop shows that node is only using 5gb ram:
Can you show the logs with verbosity
set to 3?
Can you show the logs with
verbosity
set to 3?
Sure, here are the logs:
And here's the eth_syncing result:
{
"jsonrpc": "2.0",
"id": 1,
"result": {
"currentBlock": "0x1fd35c4",
"healedBytecodeBytes": "0xf4e859d",
"healedBytecodes": "0x7465",
"healedTrienodeBytes": "0x35935ef3fc",
"healedTrienodes": "0x2db0187f",
"healingBytecode": "0x0",
"healingTrienodes": "0x454",
"highestBlock": "0x1fd3625",
"startingBlock": "0x1fd2d6d",
"syncedAccountBytes": "0xe51757115",
"syncedAccounts": "0x10fe8e32",
"syncedBytecodeBytes": "0x4d796afbe",
"syncedBytecodes": "0x251458",
"syncedStorage": "0xcf0fd89e",
"syncedStorageBytes": "0xad8db11043"
}
}
Peers count is 30
That's interesting, those logs typically only appear when you run a node with snap sync.
Did you happen to run a node previously with snap sync enabled? Or did you start fresh by downloading the latest snapshot and do a full sync?
That's interesting, those logs typically only appear when you run a node with snap sync.
Did you happen to run a node previously with snap sync enabled? Or did you start fresh by downloading the latest snapshot and do a full sync?
Yes, from the start I tried to use syncmode=snap and I changed it to full after 3 days.
BTW I used this snapshot https://github.com/48Club/bsc-snapshots#geth-full-node
Official snapshot from this page is not working for me: https://github.com/bnb-chain/bsc-snapshots
When I tried to use official snapshot I was getting peers error: node was only connected to 3-8 peers and eth_syncing response was always:
{
"jsonrpc": "2.0",
"id": 1,
"result": false
}
Yes, from the start I tried to use syncmode=snap and I changed it to full after 3 days.
Ah, then I think running snap sync initially corrupted your database. There is some issue with snap sync right now (refer to #1934).
What I'd advise is to scrap off the old database, download the latest snapshot again, and sync it with --syncmode=full
.
What I'd advice is to scrap off the old database, download the latest snapshot again, and sync it with
--syncmode=false
.
Thanks for the advice! I'll try to restart node with the latest snapshot from here: https://github.com/48Club/bsc-snapshots#geth-full-node. Also, I'll try to run my second node with the latest snapshot from here: https://github.com/bnb-chain/bsc-snapshots.
Both nodes will use --syncmode=false.
BTW what geth version you'd advice to use with the latest official snapshot? Should I use 1.2.12 or 1.3.0 ?
Also, is false
a valid value for syncmode parameter? I only see 3 options for this parameter:
--syncmode value (default: snap) ($GETH_SYNCMODE)
Blockchain sync mode ("snap", "full" or "light")
Sorry, I meant --syncmode=full
🥲. Edited the comment above.
BTW what geth version you'd advice to use with the latest official snapshot? Should I use 1.2.12 or 1.3.0 ?
If your use case isn't business-critical, you may try v1.3.1-beta. Otherwise, you can use the latest stable version which is v1.2.13.
What I'd advise is to scrap off the old database, download the latest snapshot again, and sync it with
--syncmode=full
.
I ran a new bsc node with the latest snapshot and now I'm getting connecting to peers error.
I used geth-20231104.tar.lz4 snapshot and only copied /server/data-seed/geth/chaindata
and /server/data-seed/geth/triecache
directories into my data directory.
I'm using 1.2.13 bsc node and downloaded config and genesis files from here: https://github.com/bnb-chain/bsc/releases/tag/v1.2.13.
Geth parameters:
- '--config'
- /node/bsc/config/config.toml
- '--datadir'
- /node/data
- '--rpc.allow-unprotected-txs'
- '--syncmode'
- full
- '--http'
- '--http.api'
- eth,txpool,net,web3,engine
- '--http.vhosts'
- '*'
- '--graphql.vhosts'
- '*'
- '--http.corsdomain'
- '*'
- '--graphql'
- '--ws'
- '--ws.api'
- web3,eth
- '--rpc.evmtimeout'
- 15s
- '--ws.origins'
- '*'
- '--verbosity'
- '3'
- '--cache'
- '8000'
- '--txlookuplimit'
- '0'
- '--tries-verify-mode'
- none
- '--ipcdisable'
Here's my config.toml file:
[Eth]
NetworkId = 56
LightPeers = 20
TrieTimeout = 150000000000
[Eth.Miner]
GasCeil = 140000000
GasPrice = 3000000000
Recommit = 10000000000
[Eth.TxPool]
Locals = []
NoLocals = true
Journal = "transactions.rlp"
Rejournal = 3600000000000
PriceLimit = 3000000000
PriceBump = 10
AccountSlots = 200
GlobalSlots = 8000
AccountQueue = 200
GlobalQueue = 4000
[Eth.GPO]
Blocks = 20
Percentile = 60
OracleThreshold = 1000
[Node]
IPCPath = "geth.ipc"
HTTPHost = "localhost"
InsecureUnlockAllowed = false
HTTPPort = 8545
HTTPVirtualHosts = ["localhost"]
HTTPModules = ["eth", "net", "web3", "txpool", "parlia"]
WSPort = 8546
WSModules = ["net", "web3", "eth"]
[Node.P2P]
MaxPeers = 40
NoDiscovery = false
StaticNodes = []
ListenAddr = ":30310"
EnableMsgEvents = false
net_peerCount
response:
{
"jsonrpc": "2.0",
"id": 1,
"result": "0x0"
}
eth_syncing
response:
{
"jsonrpc": "2.0",
"id": 1,
"result": {
"currentBlock": "0xb442",
"healedBytecodeBytes": "0x0",
"healedBytecodes": "0x0",
"healedTrienodeBytes": "0x0",
"healedTrienodes": "0x0",
"healingBytecode": "0x0",
"healingTrienodes": "0x0",
"highestBlock": "0xb4e8a1",
"startingBlock": "0xb442",
"syncedAccountBytes": "0x0",
"syncedAccounts": "0x0",
"syncedBytecodeBytes": "0x0",
"syncedBytecodes": "0x0",
"syncedStorage": "0x0",
"syncedStorageBytes": "0x0"
}
}
Logs with verbosity=3:
INFO [11-10|22:18:23.994] Initialised chain configuration config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Ramanujan: 0, Niels: 0, MirrorSync: 0, Bruno: 0, Berlin: 12244000, YOLO v3: , CatalystBlock: , London: 12965000, ArrowGlacier: 13773000, MergeFork:, Euler: 0, Gibbs: , Nano: , Moran: , Planck: ,Luban: , Plato: , Hertz: , Engine: ethash}"
This log states that you're running ChainID of 1, which is the ETH mainnet. Also, the other config fields seem off. You may double check if the config file has been placed in the correct directory.
This log states that you're running ChainID of 1, which is the ETH mainnet. Also, the other config fields seem off. You may double check if the config file has been placed in the correct directory.
I think config.toml location is correct and it is the same as in the geth argument. I tried to add WrongConfigValue
to the config and now my bsc node is failing to start:
INFO [11-11|12:34:55.193] Maximum peer count ETH=50 LES=0 total=50
INFO [11-11|12:34:55.193] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [11-11|12:34:55.193] Set global gas cap cap=50,000,000
INFO [11-11|12:34:55.193] Allocated cache and file handles database=/bsc/geth.ipc/geth/chaindata cache=16.00MiB handles=16
INFO [11-11|12:34:55.200] Writing custom genesis block
INFO [11-11|12:34:55.201] Persisted trie from memory database nodes=15 size=2.32KiB time="59.472µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [11-11|12:34:55.201] Successfully wrote genesis state database=chaindata hash=0d2184..d57b5b
INFO [11-11|12:34:55.201] Allocated cache and file handles database=/bsc/geth.ipc/geth/lightchaindata cache=16.00MiB handles=16
INFO [11-11|12:34:55.207] Writing custom genesis block
INFO [11-11|12:34:55.208] Persisted trie from memory database nodes=15 size=2.32KiB time="38.623µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [11-11|12:34:55.208] Successfully wrote genesis state database=lightchaindata hash=0d2184..d57b5b
INFO [11-11|12:34:55.296] Starting Geth on Ethereum mainnet...
Fatal: /node/bsc/config/config.toml, line 4: field 'WrongConfigValue' is not defined in ethconfig.Config, see https://godoc.org/github.com/ethereum/go-ethereum/eth/ethconfig#Config for available fields
Fatal: /node/bsc/config/config.toml, line 4: field 'WrongConfigValue' is not defined in ethconfig.Config, see https://godoc.org/github.com/ethereum/go-ethereum/eth/ethconfig#Config for available fields
Here's chunk of genesis.json file, here chainId also set to 56:
{
"config": {
"chainId": 56,
"homesteadBlock": 0,
"eip150Block": 0,
"eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"eip155Block": 0,
"eip158Block": 0,
"byzantiumBlock": 0,
"constantinopleBlock": 0, "petersburgBlock": 0, "istanbulBlock": 0, "muirGlacierBlock": 0, "ramanujanBlock": 0, "nielsBlock": 0, "parlia": { "period": 3, "ep$
"gasLimit": "0x2625a00",
"difficulty": "0x1",
"mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"coinbase": "0xffffFFFfFFffffffffffffffFfFFFfffFFFfFFfE",
So I guess the node read the config correctly, but I'm not sure why ChainID: 1
is in the logs.
Also, you can see that network=56 and I'm only setting this value in the config.toml
INFO [11-11|13:23:42.431] Initialising Ethereum protocol network=56 dbversion=8
I didn't manage to reproduce the issue and was able to get my node up and running from scratch. Let me just post the exact steps that I did here:
Extract the latest snapshot
wget -q -O - <snapshot URL> | tar -I lz4 -xvf -
Get the config and genesis files
wget $(curl -s https://api.github.com/repos/bnb-chain/bsc/releases/latest |grep browser_ |grep mainnet |cut -d\" -f4)
unzip mainnet.zip
Run the node
./server/bsc --config server/config.toml --cache 5000 --pprof --pprof.port 6071 --metrics --allow-insecure-unlock --txlookuplimit 0 --ws.port 8545 --rpc.allow-unprotected-txs --ws --datadir server/data-seed --http.corsdomain *
Do check again if any of the steps differ from yours (aside from the difference in command line flags, that shouldn't be the cause of the issue). If something seems odd, we can work together to solve it.
I didn't manage to sync node using this snapshot: https://pub-c0627345c16f47ab858c9469133073a8.r2.dev/geth-20231114.tar.lz4 because of the issue that I've described in my previous message.
But I did manage to sync it using a secondary snapshot from here: https://github.com/48Club/bsc-snapshots#geth-full-node.
I think you were right - the main issue why the previous node sync attempt with the unofficial snapshot stuck for 2 weeks is because of the snap
sync parameter. When I started it with full
sync from the beginning it managed to sync after 2 days.
Awesome! :)
System information
Geth version:
1.2.12
OS & Version: Amazon Linux release 2 (Karoo) / x86_64 arch OS Resources: 8cpu / 20gb / 25gbit Storage: 3.5T EBS volume (GP3 16000 iops) Environment: AWS EKS clusterExpected behaviour
Fullnode state is successfully synced after 1 week when blocks are synced.
eth_syncing endpoint returns:
Actual behaviour
The full node is not synced after 1 week of the sync process. The latest block is successfully updated but the node cannot exit from the sync phase. Main issue is that we cannot use websocket subscription endpoints as those require node to be fully synced.
eth_syncing endpoint returns:
Steps to reproduce the behaviour
Run full node using the latest snapshot from https://github.com/bnb-chain/bsc-snapshots
genesis.json file location: https://api.github.com/repos/bnb-chain/bsc/releases/latest
config.toml file:
geth parameters:
net_peerCount endpoint response:
Chunk of node logs
DEBUG[11-09|15:47:28.306] Persisted set of healing data type=trienodes bytes=33.58KiB TRACE[11-09|15:47:28.306] Fetching set of trie nodes peer=2e753bc8 reqid=6,591,669,147,356,816,575 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:28.402] Delivering set of healing trienodes peer=2e753bc8 reqid=6,591,669,147,356,816,575 trienodes=137 bytes=27.26KiB DEBUG[11-09|15:47:28.624] Persisted set of healing data type=trienodes bytes=29.71KiB TRACE[11-09|15:47:28.625] Fetching set of trie nodes peer=73f0313a reqid=1,100,057,035,030,134,389 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:28.633] Fetching full headers peer=e93f789d count=192 from=33,346,247 DEBUG[11-09|15:47:28.633] Fetching batch of headers id=e93f789d6a79c157 conn=dyndial count=192 fromnum=33,346,247 skip=0 reverse=false DEBUG[11-09|15:47:28.633] Recalculated msgrate QoS values proto=eth rtt=2s confidence=1.000 ttl=6s next=2023-11-09T15:47:30+0000 TRACE[11-09|15:47:28.633] Debug dump of mean capacities proto=eth caps="map[4:1341.580776158541 6:34.404474701038744 16:25.524947977666255]" TRACE[11-09|15:47:28.634] Scheduling new headers peer=e93f789d count=1 from=33,346,247 TRACE[11-09|15:47:28.634] Fetching full headers peer=e93f789d count=192 from=33,346,248 DEBUG[11-09|15:47:28.634] Fetching batch of headers id=e93f789d6a79c157 conn=dyndial count=192 fromnum=33,346,248 skip=0 reverse=false TRACE[11-09|15:47:28.635] All headers delayed, waiting peer=e93f789d INFO [11-09|15:47:28.637] Imported new block headers count=1 elapsed=3.180ms number=33,346,247 hash=66beaa..904e9a TRACE[11-09|15:47:28.637] Requesting new batch of receipts peer=2e753bc8 count=1 from=33,346,247 DEBUG[11-09|15:47:28.637] Fetching batch of receipts id=2e753bc8398aa451 conn=dyndial count=1 TRACE[11-09|15:47:28.638] Requesting new batch of bodies peer=e93f789d count=1 from=33,346,247 DEBUG[11-09|15:47:28.638] Fetching batch of block bodies id=e93f789d6a79c157 conn=dyndial count=1 TRACE[11-09|15:47:28.641] Delivered new batch of bodies peer=e93f789d count=1 accepted=1 TRACE[11-09|15:47:28.642] Delivered new batch of receipts peer=2e753bc8 count=1 accepted=1 TRACE[11-09|15:47:28.756] Delivering set of healing trienodes peer=73f0313a reqid=1,100,057,035,030,134,389 trienodes=141 bytes=36.73KiB DEBUG[11-09|15:47:28.886] Persisted set of healing data type=trienodes bytes=42.60KiB TRACE[11-09|15:47:28.886] Fetching set of trie nodes peer=82ed1450 reqid=8,614,644,942,323,186,820 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:28.929] Delivering set of healing trienodes peer=82ed1450 reqid=8,614,644,942,323,186,820 trienodes=138 bytes=31.04KiB DEBUG[11-09|15:47:29.150] Persisted set of healing data type=trienodes bytes=29.88KiB DEBUG[11-09|15:47:29.150] Updated trie node heal throttler rate=511.017 pending=827 throttle=1.000 TRACE[11-09|15:47:29.151] Fetching set of trie nodes peer=2b4afa01 reqid=360,651,606,719,197,818 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB DEBUG[11-09|15:47:29.151] Peer discarded announcement by distance peer=1db7fbf2a9eda081230bfa0aac466edc84a6547be759c50559de354787c85a6a number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.158] Peer discarded announcement by distance peer=e95fa0f62c8b1c3192c2335516729263fa7225c94f6a97074e52c6b2042bcecf number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.164] Peer discarded announcement by distance peer=f09744a5d630b727815525078c89246237d724745381aa9b40052bcdd63ff568 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.187] Peer discarded announcement by distance peer=919e705de10a9ac4ec7870eb6037fd79362c879f61b486d6a3732380c26ec7a8 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.190] Peer discarded announcement by distance peer=cf43792e96e46a793084f47c657ef287f22634c5e73cb6c04ebc6d1b305ab88a number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.237] Peer discarded announcement by distance peer=94f1f316c7aea29dd0bd3fefce6e3863e11bdfebcaff96bc0dd808e51a04ae17 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.239] Peer discarded announcement by distance peer=b398012f04b48b843660d982d66237cd49b17f11ed5e825a3d9ec9ed159dfeef number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.244] Peer discarded announcement by distance peer=52bd007b441691b1820c8646dd3d053f03cce0a3f266fae4c98ec145a342e985 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.254] Peer discarded announcement by distance peer=c4f3d0ff2f3b511470fa1995a0b0cf21edd8edaf17433834289f23bfc77fa51a number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 TRACE[11-09|15:47:29.254] Delivering set of healing trienodes peer=2b4afa01 reqid=360,651,606,719,197,818 trienodes=139 bytes=31.68KiB DEBUG[11-09|15:47:29.257] Peer discarded announcement by distance peer=c73a8be46fe54d49acdd1ccac0be37f188bfac6b3649751eeb634251502197fe number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.264] Peer discarded announcement by distance peer=82ed1450fee0eaafebd857fa397e9f2833c9ca7f9c2fecf85720225847dcadb5 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.292] Peer discarded announcement by distance peer=2025f9246b21510d94a6e877d02d2f7d17ec7aa7a3697b9db571dfdd8417c00b number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.309] Peer discarded announcement by distance peer=2b4afa01dc8034e502e17786db30b6ab783db9f9ccb1a5e654abf3f68f3a10e8 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.337] Peer discarded announcement by distance peer=10c1fae0b25ac54c851e46fb65d4c615d694edb20752fac0b01cec23426c865f number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.348] Peer discarded announcement by distance peer=1f3dafdc07212ff4d21de9c7cb8ad71922fc4b245d46565b4e1b169e40642a90 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.353] Peer discarded announcement by distance peer=13edaeca09ce0d2dede2c4d63fb8215de29b0919a3bf7e820f5ca6b5e1bba213 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.399] Peer discarded announcement by distance peer=b31b39d65990b97a2906fa5bf0da3571cd90cd41c9e0c506afa3e1bd5c7ecc32 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.418] Peer discarded announcement by distance peer=85f414213ca8e09fb8ba068c1308904d0615fd8756bac0e275a010fdddcaecfb number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.425] Persisted set of healing data type=trienodes bytes=37.05KiB TRACE[11-09|15:47:29.426] Fetching set of trie nodes peer=52bd007b reqid=3,696,757,289,557,932,233 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:29.452] Delivering set of healing trienodes peer=52bd007b reqid=3,696,757,289,557,932,233 trienodes=135 bytes=24.98KiB DEBUG[11-09|15:47:29.486] Peer discarded announcement by distance peer=73f0313ac1420b1a5ba614919ef5d2c83b92a75f54f108404184a66de12f8af2 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.501] Peer discarded announcement by distance peer=7058d2dcec9436a05db578a2589a201f963a6518e3bc337815b345a0084b6774 number=33,346,244 hash=d949e4..24386e distance=33,346,244 DEBUG[11-09|15:47:29.657] Persisted set of healing data type=trienodes bytes=35.77KiB DEBUG[11-09|15:47:29.657] Recalculated msgrate QoS values proto=snap rtt=2s confidence=1.000 ttl=6s next=2023-11-09T15:47:31+0000 TRACE[11-09|15:47:29.657] Debug dump of mean capacities proto=snap caps="map[5:51.13902036936107 7:497.800846178016]" TRACE[11-09|15:47:29.657] Fetching set of trie nodes peer=2025f924 reqid=4,970,885,386,494,792,360 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:29.740] Delivering set of healing trienodes peer=2025f924 reqid=4,970,885,386,494,792,360 trienodes=138 bytes=30.60KiB DEBUG[11-09|15:47:29.784] Peer discarded announcement by distance peer=fbca7bb1c8e97cb5d3b16a8169b3a8b668af5621718f19348f08b1d4564aada5 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.814] Peer discarded announcement by distance peer=52f89ab89ba938e1e78fc5a09b90ef53818adc5b7db647639f30f3ad57a5e278 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.814] Peer discarded announcement by distance peer=aae0ed11c47b653fa4b1d6bdf1790fd22aba8c30d9daa3e280eee75b37d13372 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:29.890] Persisted set of healing data type=trienodes bytes=40.62KiB TRACE[11-09|15:47:29.890] Fetching set of trie nodes peer=e93f789d reqid=8,434,732,720,110,603,629 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:30.013] Delivering set of healing trienodes peer=e93f789d reqid=8,434,732,720,110,603,629 trienodes=139 bytes=34.13KiB DEBUG[11-09|15:47:30.136] Persisted set of healing data type=trienodes bytes=39.03KiB INFO [11-09|15:47:30.136] State heal in progress accounts=464,573@26.88MiB slots=889,522@65.42MiB codes=28485@233.84MiB nodes=730,028,391@203.75GiB pending=33410 TRACE[11-09|15:47:30.136] Fetching set of trie nodes peer=2e753bc8 reqid=7,145,491,214,201,630,368 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:30.252] Delivering set of healing trienodes peer=2e753bc8 reqid=7,145,491,214,201,630,368 trienodes=142 bytes=36.82KiB DEBUG[11-09|15:47:30.253] Peer discarded announcement by distance peer=7cf4b4e98140319b444d953c157f2d959586ce06da7f4c4e89937bec35040cb5 number=33,346,250 hash=2475e6..fc0f59 distance=33,346,250 DEBUG[11-09|15:47:30.427] Persisted set of healing data type=trienodes bytes=34.43KiB DEBUG[11-09|15:47:30.427] Updated trie node heal throttler rate=523.125 pending=831 throttle=1.000 TRACE[11-09|15:47:30.434] Fetching set of trie nodes peer=73f0313a reqid=4,645,309,470,360,138,868 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:30.560] Delivering set of healing trienodes peer=73f0313a reqid=4,645,309,470,360,138,868 trienodes=139 bytes=32.03KiB DEBUG[11-09|15:47:30.996] Persisted set of healing data type=trienodes bytes=28.74KiB TRACE[11-09|15:47:30.996] Fetching set of trie nodes peer=82ed1450 reqid=1,081,337,485,761,318,948 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:31.061] Delivering set of healing trienodes peer=82ed1450 reqid=1,081,337,485,761,318,948 trienodes=140 bytes=34.77KiB DEBUG[11-09|15:47:31.238] Persisted set of healing data type=trienodes bytes=35.29KiB TRACE[11-09|15:47:31.238] Fetching set of trie nodes peer=2b4afa01 reqid=757,213,434,095,439,570 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:31.330] Delivering set of healing trienodes peer=2b4afa01 reqid=757,213,434,095,439,570 trienodes=142 bytes=36.69KiB DEBUG[11-09|15:47:31.436] Persisted set of healing data type=trienodes bytes=44.15KiB DEBUG[11-09|15:47:31.436] Updated trie node heal throttler rate=579.430 pending=840 throttle=1.000 TRACE[11-09|15:47:31.436] Fetching set of trie nodes peer=52bd007b reqid=2,326,365,468,671,992,664 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:31.465] Delivering set of healing trienodes peer=52bd007b reqid=2,326,365,468,671,992,664 trienodes=133 bytes=21.68KiB TRACE[11-09|15:47:31.636] Fetching full headers peer=e93f789d count=192 from=33,346,248 DEBUG[11-09|15:47:31.636] Fetching batch of headers id=e93f789d6a79c157 conn=dyndial count=192 fromnum=33,346,248 skip=0 reverse=false DEBUG[11-09|15:47:31.636] Recalculated msgrate QoS values proto=eth rtt=2s confidence=1.000 ttl=6s next=2023-11-09T15:47:33+0000 TRACE[11-09|15:47:31.636] Debug dump of mean capacities proto=eth caps="map[4:1341.580776158541 6:34.4413276436704 16:25.589237588360827]" TRACE[11-09|15:47:31.637] Scheduling new headers peer=e93f789d count=1 from=33,346,248 TRACE[11-09|15:47:31.637] Fetching full headers peer=e93f789d count=192 from=33,346,249 DEBUG[11-09|15:47:31.638] Fetching batch of headers id=e93f789d6a79c157 conn=dyndial count=192 fromnum=33,346,249 skip=0 reverse=false TRACE[11-09|15:47:31.639] All headers delayed, waiting peer=e93f789d INFO [11-09|15:47:31.639] Imported new block headers count=1 elapsed=1.753ms number=33,346,248 hash=9e9841..a941e0 TRACE[11-09|15:47:31.639] Requesting new batch of receipts peer=2e753bc8 count=1 from=33,346,248 DEBUG[11-09|15:47:31.639] Fetching batch of receipts id=2e753bc8398aa451 conn=dyndial count=1 TRACE[11-09|15:47:31.639] Requesting new batch of bodies peer=e93f789d count=1 from=33,346,248 DEBUG[11-09|15:47:31.639] Fetching batch of block bodies id=e93f789d6a79c157 conn=dyndial count=1 TRACE[11-09|15:47:31.643] Delivered new batch of bodies peer=e93f789d count=1 accepted=1 TRACE[11-09|15:47:31.645] Delivered new batch of receipts peer=2e753bc8 count=1 accepted=1 DEBUG[11-09|15:47:31.677] Persisted set of healing data type=trienodes bytes=36.36KiB DEBUG[11-09|15:47:31.677] Recalculated msgrate QoS values proto=snap rtt=2s confidence=1.000 ttl=6s next=2023-11-09T15:47:33+0000 TRACE[11-09|15:47:31.677] Debug dump of mean capacities proto=snap caps="map[5:51.13902036936106 7:492.49352640690256]" TRACE[11-09|15:47:31.677] Fetching set of trie nodes peer=2025f924 reqid=6,524,510,854,336,740,441 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:31.801] Delivering set of healing trienodes peer=2025f924 reqid=6,524,510,854,336,740,441 trienodes=137 bytes=29.81KiB DEBUG[11-09|15:47:31.941] Persisted set of healing data type=trienodes bytes=43.09KiB TRACE[11-09|15:47:31.941] Fetching set of trie nodes peer=e93f789d reqid=5,248,302,664,143,663,489 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:32.087] Delivering set of healing trienodes peer=e93f789d reqid=5,248,302,664,143,663,489 trienodes=144 bytes=40.48KiB DEBUG[11-09|15:47:32.133] Peer discarded announcement by distance peer=f09744a5d630b727815525078c89246237d724745381aa9b40052bcdd63ff568 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.143] Peer discarded announcement by distance peer=1db7fbf2a9eda081230bfa0aac466edc84a6547be759c50559de354787c85a6a number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.144] Peer discarded announcement by distance peer=919e705de10a9ac4ec7870eb6037fd79362c879f61b486d6a3732380c26ec7a8 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.149] Peer discarded announcement by distance peer=c73a8be46fe54d49acdd1ccac0be37f188bfac6b3649751eeb634251502197fe number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.151] Peer discarded announcement by distance peer=e95fa0f62c8b1c3192c2335516729263fa7225c94f6a97074e52c6b2042bcecf number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.190] Peer discarded announcement by distance peer=c4f3d0ff2f3b511470fa1995a0b0cf21edd8edaf17433834289f23bfc77fa51a number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.203] Persisted set of healing data type=trienodes bytes=35.98KiB TRACE[11-09|15:47:32.203] Fetching set of trie nodes peer=2e753bc8 reqid=3,044,235,293,014,042,147 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB DEBUG[11-09|15:47:32.204] Peer discarded announcement by distance peer=52f89ab89ba938e1e78fc5a09b90ef53818adc5b7db647639f30f3ad57a5e278 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.205] Peer discarded announcement by distance peer=aae0ed11c47b653fa4b1d6bdf1790fd22aba8c30d9daa3e280eee75b37d13372 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.206] Peer discarded announcement by distance peer=b398012f04b48b843660d982d66237cd49b17f11ed5e825a3d9ec9ed159dfeef number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.221] Peer discarded announcement by distance peer=82ed1450fee0eaafebd857fa397e9f2833c9ca7f9c2fecf85720225847dcadb5 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.229] Peer discarded announcement by distance peer=2b4afa01dc8034e502e17786db30b6ab783db9f9ccb1a5e654abf3f68f3a10e8 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.230] Peer discarded announcement by distance peer=94f1f316c7aea29dd0bd3fefce6e3863e11bdfebcaff96bc0dd808e51a04ae17 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.235] Peer discarded announcement by distance peer=52bd007b441691b1820c8646dd3d053f03cce0a3f266fae4c98ec145a342e985 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.248] Peer discarded announcement by distance peer=73f0313ac1420b1a5ba614919ef5d2c83b92a75f54f108404184a66de12f8af2 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.265] Peer discarded announcement by distance peer=e93f789d6a79c157cb926d2f478f81078fe25de7c5010b2a6f8e22986429c8f2 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.267] Peer discarded announcement by distance peer=2e753bc8398aa45144eefc47d0dcc23e48cca403cb43c519f04258c34e9db89a number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.289] Peer discarded announcement by distance peer=cf43792e96e46a793084f47c657ef287f22634c5e73cb6c04ebc6d1b305ab88a number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.297] Peer discarded announcement by distance peer=2025f9246b21510d94a6e877d02d2f7d17ec7aa7a3697b9db571dfdd8417c00b number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.325] Peer discarded announcement by distance peer=85f414213ca8e09fb8ba068c1308904d0615fd8756bac0e275a010fdddcaecfb number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 TRACE[11-09|15:47:32.353] Delivering set of healing trienodes peer=2e753bc8 reqid=3,044,235,293,014,042,147 trienodes=139 bytes=33.27KiB DEBUG[11-09|15:47:32.357] Peer discarded announcement by distance peer=13edaeca09ce0d2dede2c4d63fb8215de29b0919a3bf7e820f5ca6b5e1bba213 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.399] Peer discarded announcement by distance peer=10c1fae0b25ac54c851e46fb65d4c615d694edb20752fac0b01cec23426c865f number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.414] Peer discarded announcement by distance peer=b31b39d65990b97a2906fa5bf0da3571cd90cd41c9e0c506afa3e1bd5c7ecc32 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 DEBUG[11-09|15:47:32.464] Persisted set of healing data type=trienodes bytes=31.11KiB DEBUG[11-09|15:47:32.464] Updated trie node heal throttler rate=540.063 pending=835 throttle=1.000 TRACE[11-09|15:47:32.465] Fetching set of trie nodes peer=73f0313a reqid=5,172,624,451,882,216,167 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB DEBUG[11-09|15:47:32.534] Peer discarded announcement by distance peer=1f3dafdc07212ff4d21de9c7cb8ad71922fc4b245d46565b4e1b169e40642a90 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 TRACE[11-09|15:47:32.630] Delivering set of healing trienodes peer=73f0313a reqid=5,172,624,451,882,216,167 trienodes=142 bytes=37.01KiB DEBUG[11-09|15:47:32.760] Persisted set of healing data type=trienodes bytes=37.98KiB TRACE[11-09|15:47:32.760] Fetching set of trie nodes peer=82ed1450 reqid=6,476,641,583,680,851,893 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:32.815] Delivering set of healing trienodes peer=82ed1450 reqid=6,476,641,583,680,851,893 trienodes=139 bytes=33.53KiB TRACE[11-09|15:47:32.952] >> PING/v4 id=e5c17559715cadf1 addr=100.26.14.92:30311 err=nil TRACE[11-09|15:47:32.953] << PONG/v4 id=e5c17559715cadf1 addr=100.26.14.92:30311 err=nil TRACE[11-09|15:47:32.953] >> ENRREQUEST/v4 id=e5c17559715cadf1 addr=100.26.14.92:30311 err=nil TRACE[11-09|15:47:32.955] << ENRRESPONSE/v4 id=e5c17559715cadf1 addr=100.26.14.92:30311 err=nil DEBUG[11-09|15:47:32.955] Revalidated node b=14 id=e5c17559715cadf1 checks=1 TRACE[11-09|15:47:32.957] << PING/v4 id=e5c17559715cadf1 addr=100.26.14.92:30311 err=nil TRACE[11-09|15:47:32.957] >> PONG/v4 id=e5c17559715cadf1 addr=100.26.14.92:30311 err=nil TRACE[11-09|15:47:32.957] >> ENRREQUEST/v4 id=e5c17559715cadf1 addr=100.26.14.92:30311 err=nil TRACE[11-09|15:47:32.959] << ENRRESPONSE/v4 id=e5c17559715cadf1 addr=100.26.14.92:30311 err=nil DEBUG[11-09|15:47:33.075] Persisted set of healing data type=trienodes bytes=30.62KiB TRACE[11-09|15:47:33.075] Fetching set of trie nodes peer=2b4afa01 reqid=6,335,063,153,583,463,714 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB DEBUG[11-09|15:47:33.103] Peer discarded announcement by distance peer=fbca7bb1c8e97cb5d3b16a8169b3a8b668af5621718f19348f08b1d4564aada5 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 TRACE[11-09|15:47:33.183] Delivering set of healing trienodes peer=2b4afa01 reqid=6,335,063,153,583,463,714 trienodes=139 bytes=32.37KiB DEBUG[11-09|15:47:33.261] Persisted set of healing data type=trienodes bytes=42.47KiB TRACE[11-09|15:47:33.261] Fetching set of trie nodes peer=52bd007b reqid=6,544,860,613,471,661,503 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:33.292] Delivering set of healing trienodes peer=52bd007b reqid=6,544,860,613,471,661,503 trienodes=135 bytes=26.15KiB TRACE[11-09|15:47:33.458] << ENRREQUEST/v4 id=e5c17559715cadf1 addr=100.26.14.92:30311 err=nil TRACE[11-09|15:47:33.458] >> ENRRESPONSE/v4 id=e5c17559715cadf1 addr=100.26.14.92:30311 err=nil DEBUG[11-09|15:47:33.504] Persisted set of healing data type=trienodes bytes=32.38KiB DEBUG[11-09|15:47:33.504] Updated trie node heal throttler rate=579.187 pending=838 throttle=1.000 TRACE[11-09|15:47:33.505] Fetching set of trie nodes peer=2025f924 reqid=7,302,565,316,647,046,830 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:33.603] Delivering set of healing trienodes peer=2025f924 reqid=7,302,565,316,647,046,830 trienodes=139 bytes=33.51KiB DEBUG[11-09|15:47:33.818] Persisted set of healing data type=trienodes bytes=27.87KiB DEBUG[11-09|15:47:33.818] Recalculated msgrate QoS values proto=snap rtt=2s confidence=1.000 ttl=6s next=2023-11-09T15:47:35+0000 TRACE[11-09|15:47:33.818] Debug dump of mean capacities proto=snap caps="map[5:51.13902036936108 7:482.0361671659068]" TRACE[11-09|15:47:33.818] Fetching set of trie nodes peer=e93f789d reqid=6,628,998,626,171,587,144 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB DEBUG[11-09|15:47:33.831] Peer discarded announcement by distance peer=7cf4b4e98140319b444d953c157f2d959586ce06da7f4c4e89937bec35040cb5 number=33,346,251 hash=aa870c..5a80ed distance=33,346,251 TRACE[11-09|15:47:34.083] Delivering set of healing trienodes peer=e93f789d reqid=6,628,998,626,171,587,144 trienodes=140 bytes=34.14KiB DEBUG[11-09|15:47:34.085] Persisted set of healing data type=trienodes bytes=44.80KiB TRACE[11-09|15:47:34.085] Fetching set of trie nodes peer=2e753bc8 reqid=5,945,243,426,260,316,416 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:34.287] Delivering set of healing trienodes peer=2e753bc8 reqid=5,945,243,426,260,316,416 trienodes=138 bytes=32.18KiB DEBUG[11-09|15:47:34.379] Persisted set of healing data type=trienodes bytes=23.54KiB TRACE[11-09|15:47:34.380] Fetching set of trie nodes peer=73f0313a reqid=1,461,345,841,690,776,926 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:34.528] Delivering set of healing trienodes peer=73f0313a reqid=1,461,345,841,690,776,926 trienodes=143 bytes=39.26KiB DEBUG[11-09|15:47:34.636] Persisted set of healing data type=trienodes bytes=39.06KiB DEBUG[11-09|15:47:34.636] Updated trie node heal throttler rate=522.353 pending=834 throttle=1.000 TRACE[11-09|15:47:34.636] Fetching set of trie nodes peer=82ed1450 reqid=5,971,607,732,878,028,983 root=0ec148..6aed4b pathsets=1024 bytes=512.00KiB TRACE[11-09|15:47:34.639] Fetching full headers peer=e93f789d count=192 from=33,346,249 DEBUG[11-09|15:47:34.639] Fetching batch of headers id=e93f789d6a79c157 conn=dyndial count=192 fromnum=33,346,249 skip=0 reverse=false DEBUG[11-09|15:47:34.639] Recalculated msgrate QoS values proto=eth rtt=2s confidence=1.000 ttl=6s next=2023-11-09T15:47:36+0000 TRACE[11-09|15:47:34.639] Debug dump of mean capacities proto=eth caps="map[4:1341.580776158542 6:34.43174381582538 16:25.5623930566584]"