What is the upper bound of "imported new state entries"? When will it end?

[idotial]

System information

Geth version: 1.7.2 OS & Version: OSX

Expected behaviour

geth --fast should finish soon.

Actual behaviour

it run for 3days and print “Imported new state entries count=384 elapsed=26.970ms processed=50023987 pending=33074 retry=0 duplicate=19087 unexpected=47765” constantly

Steps to reproduce the behaviour

run geth --fast in console

Backtrace

INFO [12-06|07:08:00] Imported new state entries count=1259 elapsed=12.971ms processed=50014526 pending=34891 retry=0 duplicate=19087 unexpected=47765 INFO [12-06|07:08:23] Imported new state entries count=774 elapsed=8.950ms processed=50015300 pending=34311 retry=0 duplicate=19087 unexpected=47765 INFO [12-06|07:08:31] Imported new state entries count=1125 elapsed=9.513ms processed=50016425 pending=33428 retry=0 duplicate=19087 unexpected=47765 INFO [12-06|07:08:39] Imported new state entries count=1061 elapsed=11.198ms processed=50017486 pending=32566 retry=0 duplicate=19087 unexpected=47765 INFO [12-06|07:08:49] Imported new state entries count=1314 elapsed=12.041ms processed=50018800 pending=31248 retry=0 duplicate=19087 unexpected=47765 INFO [12-06|07:09:10] Imported new state entries count=1028 elapsed=10.446ms processed=50019828 pending=30496 retry=0 duplicate=19087 unexpected=47765 INFO [12-06|07:09:25] Imported new state entries count=1241 elapsed=10.423ms processed=50021069 pending=30088 retry=0 duplicate=19087 unexpected=47765 INFO [12-06|07:09:37] Imported new state entries count=777 elapsed=6.224ms processed=50021846 pending=29851 retry=26 duplicate=19087 unexpected=47765

[Duncan-Brain]

What I am looking for from this issue is an approximate formula for states, and some link to a blog or something that has accumulated a bunch of failures of when your system is not ever going to catch up to the header block.

The current method I am using uses the estimate of 3-6k state changes per block (Reference) . Of those 3-6k changes, 2-4k are additions to the state tree and 1-2k are deletions. So delta ~ 3-6k, net ~1-2k per block.

If we use these estimates on @Neurone local state above to make a lower bound ( 650,634,632 in 64 hours) "Total states" = 650,634,632 - (3000x4x60x64) = 604554632 at the start of sync, and "Total states" = 604554632+(1000x4x60x64) = 619914632 at the end of sync.

I think this is a good formula for estimating in lieu of a better way -- so far for me it has correctly predicted my inability to catch up to the header block, I just can't quite figure out my system's issue (my data here). The 3-6k seems outdated so there is also that.

Any formulas or indicators for "never catching up"?

[Capsicle1]

Running a 4gb Raspberry Pi and approaching 900 million state entries and 325gb used.

processed=898488115

[estebaneu]

FYI, I finished syncing last night:

State entry: 664 401 205

Hardware NUC i5-8259U -16 GB RAM - 1TB NMVE Sabrent Q

[nemwiper]

Just an update for February:

11797042 blocks I did not be able to see the last state entry but I think it was around 700 000 000

Node synced in 12 hours on a i7, 32Gb RAM and 500GBSSD in RAID1, dedicated server.

[LEDRGB]

Not sync yet with 753780733 states :(

[LEDRGB]

Hey, today I finally managed to synchronize the node, I leave you the latest data in case it is useful for someone, it has taken a lot, practically a month :)

[DiveInto]

Re-Sync, very close to sync complete

[kraymond37]

Currently

Imported new state entries count=384  elapsed=1.277ms     processed=721310525 pending=19751 trieretry=0   coderetry=0 duplicate=0 unexpected=0

Command

geth --cache=1024 --maxpeers=25 --datadir=/data/eth

SSD used, 8G RAM, currently about 200k state entries per hour, it will take over a month to arrive 900m!

Is there anything that I can do?

[NotSureBTC]

I am currently at 1059769034 states and still syncing on a Pi 4. Starting to think it may never catch up.

[NotSureBTC]

Upped the clock rate, still not caught up.

[damiandennis]

wow 1059769034, I am only at 810953009, still got a long ways to go :/ about two months ago it took me about three days to sync, now its taken a week already, and its super slow, I think it was related to using the lastest client before the berlin fork as soon as it went live the state sync went up significantly but it has slowed down again :/

[damiandennis]

also about three months ago the state count was ~700000000 seems like it goes up fast.

[NotSureBTC]

OK I discovered that state entries are misleading, I copied the chain over from my synchronized machine and it started at a smaller number. I believe they count a certain number from where it started to where it is now. The longer it takes the higher that number can go, so I believe tracking them here is pointless.

[syedabbast]

It's been a month and knowns states are increasing. should i stop my server and forget about sync? because known states numbers keep changing from 40K to 50K to 35K and next hour it goes back to 40K.

Your help will be appreciated.

This is what I see

eth.syncing { currentBlock: 12559714, highestBlock: 12559782, knownStates: 54981791, pulledStates: 54725893, startingBlock: 12555163 }

Best Regards, Syed

[LEDRGB]

Don't worry, I also thought about turning off many times, but you just have to let it work, it is normal that it takes a long time, do not despair :)

[jp-engineer]

I just finished a fast-sync in the past couple days, unfortunately I had left it running whilst being away so I can't confirm the final number but I saw up to 1,000,000,000 and estimate that at time of writing it's closer to 1,100,000,000.

For those in the thread commenting about Raspberry Pi's I completed the sync with the following setup:

Pi 4 8gb

• Overclocked to 1.8Ghz

• Running Raspian Lite 64 Bit (In-Dev)

• go version: go1.16.5 linux/arm64

• geth version: CoreGeth 1.12.1-unstable

• Storage: 1Tb Nvme drive connected via Powered USB 3.0 hub

  • Including: 64GB swap file (swappiness set to 20, VFS cache pressure set to 50)

• Internet speed: 350 Mbps+

• Port Forwarding

Unfortunately I tried several configfurations and didn't keep perfect records on the speed increases however: based on the final speeds achieved with the above configuration, I estimate a Pi 4 8gb could sync in about 4-6 days as of July 2021.

EDIT: -cache size 2048, -max_peers 40. Running as a systemd service.

[holiman]

Fast-sync is no longer available in geth, so I'm closing this