geth --fast stalls before crossing finish line

[Dirksterson]

System information

Geth version: geth version 1.5.9-stable, Go1.7.4 OS & Version: OSX 10.12.6 MacMini 4GB RAM (latest MacMini doesn't support field RAM upgrade anymore) VDSL connection with an average of 20-40Mbit throughput. Ethereum Wallet 0.9.0 Commit hash : (if develop)

Expected behaviour

fast sync to current latest block followed by auto disabling

Actual behaviour

stalling from a few thousand blocks up to a few hundred to current latest block. Tries to catch up to latest block, but number of new blocks is greater than the speed of adding fast blocks. Never auto disables fast sync mode.

Steps to reproduce the behaviour

Removedb and geth --fast --cache=1024. 5 times on that machine over the last weeks.

Fast sync is already my workaround, starting a fresh fast sync from scratch. Before I was unsuccessful on that machine trying to sync with existing blockchain data instead. This was also a lost race of catching up to the latest block on that machine. This workaround was good until now.

Today even the workaround in fast sync mode (cache -1024) will not completely load the blockchain anymore. It catches up some hundred blocks to the latest block and stalls for hours. By the time it catches up a few hundred blocks, the latest block moved ahead again. The closer geth is getting to import to the latest block (at time of writing 4173161), the slower it gets. It does not catch up anymore. Tried 5 times now over the last weeks and giving up at around 4-5 days each.

Does the machine not meet todays minimum hardware requirement anymore or is this a major bug?

Backtrace

latest block 13 hours ago (!)

I0818 00:15:26.444933 core/blockchain.go:805] imported 148 receipts in 2.775s. #4169952 [e3f556fc… / 36f4d3c9…]

...

latest header chain 50 minutes ago

I0818 12:47:45.107445 core/headerchain.go:342] imported 1 headers in 4.954ms. #4173009 [350d1426… / 350d1426…]

...

currently only importing nothing but state entries

I0818 13:36:41.103101 eth/downloader/downloader.go:966] imported 172 state entries in 10.009s: processed 10010213, pending at least 129361 I0818 13:36:41.103131 eth/downloader/downloader.go:966] imported 384 state entries in 783.519ms: processed 10010597, pending at least 129361 I0818 13:36:41.103154 eth/downloader/downloader.go:966] imported 381 state entries in 6.963s: processed 10010978, pending at least 129361 I0818 13:36:41.103167 eth/downloader/downloader.go:966] imported 25 state entries in 87.654ms: processed 10011003, pending at least 129360 I0818 13:36:46.014244 eth/downloader/downloader.go:966] imported 384 state entries in 2.482s: processed 10011387, pending at least 127584 I0818 13:36:49.074483 eth/downloader/downloader.go:966] imported 381 state entries in 7.082s: processed 10011768, pending at least 127105 I0818 13:36:49.074553 eth/downloader/downloader.go:966] imported 384 state entries in 7.971s: processed 10012152, pending at least 127105 I0818 13:36:49.074574 eth/downloader/downloader.go:966] imported 384 state entries in 3.772s: processed 10012536, pending at least 127105 I0818 13:36:49.074603 eth/downloader/downloader.go:966] imported 162 state entries in 5.822s: processed 10012698, pending at least 127105 I0818 13:36:49.074622 eth/downloader/downloader.go:966] imported 25 state entries in 4.050s: processed 10012723, pending at least 127105 I0818 13:36:49.074639 eth/downloader/downloader.go:966] imported 381 state entries in 3.060s: processed 10013104, pending at least 127105 I0818 13:36:49.074742 eth/downloader/downloader.go:966] imported 85 state entries in 7.117s: processed 10013189, pending at least 127105 I0818 13:36:49.074765 eth/downloader/downloader.go:966] imported 375 state entries in 2.219s: processed 10013564, pending at least 127105 I0818 13:36:49.074782 eth/downloader/downloader.go:966] imported 87 state entries in 3.915s: processed 10013651, pending at least 127105 I0818 13:36:49.074795 eth/downloader/downloader.go:966] imported 23 state entries in 271.734ms: processed 10013674, pending at least 127104

[MartinDace]

@tomtom87 thank you. I have found the ethereum wallet files in Libraries - Application support. Where would the json file be?

[holiman]

General advisory: do not share the wallet file with anyone you don't trust fully. The json wallet file contains your encrypted private key. Anyone in control of the private key has full access to all funds on the account.

[tomtom87]

@MartinDace yes don't send anything like @holiman said. Ok just get the file in the keystore directory and you can use that to unlock on myetherwallet i do it all the time when it wont sync.

[MartinDace]

@tomtom87 thanks for your advice but I don't know where to start. There are three files on my computer with the suffix .json and none of them have the private key. Where is the keystore directory? I cannot find any file or folder with that name. Likewise I don't know how to go about creating a 'Mist exported json file.' I do not see any export facility in the Mist menu. This may all seem obvious to you but I have no idea where to begin. Your further advice would be much appreciated.

[holiman]

@MartinDace this ticket is not the right forum. Here's some info about where the keystore is located https://ethereum.stackexchange.com/a/109 .

[tomtom87]

Hi mate just go to /Library/Ethereum/keystore

its the file name start UTC does not have json file extension. Open in myetherwallet as json format enter your password to unlock.

If still confused in Mist app just click the backup menu option it will launch finder in the keystore directory. Load the file with filename starting UTC an a bunch of numbers. This is your wallet json. Now in myetherwallet transfer to your exodus and spit on mist never use it again its not production ready.

Peace

On 6 Oct 2017, at 02:21, MartinDace notifications@github.com wrote:

@tomtom87 thanks for your advice but I don't know where to start. There are three files on my computer with the suffix .json and none of them have the private key. Where is the keystore directory? I cannot find any file or folder with that name. Likewise I don't know how to go about creating a 'Mist exported json file.' I do not see any export facility in the Mist menu. This may all seem obvious to you but I have no idea where to begin. Your further advice would be much appreciated.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[MartinDace]

@tomtom87 Hello Tom, thanks, I finally found it. It's not in Application support/ ethereum where I was looking for it before, but in User/ [name] /Library... as you say. Not easy as the Library was hidden by the Mac OS. I think I have finally rescued my ether, which while all this was going on appears to have increased in value by 50%. There should be a warning on the web site that the wallet is not for regular folk. Thanks also @holiman Martin for a helpful attitude. Over and out!

[abl1234]

I'm thrilled for you

On 10 Oct 2017 8:58 pm, "MartinDace" notifications@github.com wrote:

@tomtom87 https://github.com/tomtom87 Hello Tom, thanks, I finally found it. It's not in Application support/ ethereum where I was looking for it before, but in User/ [name] /Library... as you say. Not easy as the Library was hidden by the Mac OS. I think I have finally rescued my ether, which while all this was going on appears to have increased in value by 50%. There should be a warning on the web site that the wallet is not for regular folk. Thanks also @holiman https://github.com/holiman Martin for a helpful attitude. Over and out!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-335589580, or mute the thread https://github.com/notifications/unsubscribe-auth/Ae3NWZ7LabS1Mj3DcyK4eU23BBa9MZokks5sq8yAgaJpZM4O7ctW .

[ilia2s]

same problem

[MartinDace]

@ilia2s follow the advice above. If you have a mac I can tell you how to find the files.

[ilia2s]

thank you, the update helps

[MobiD2017]

I have the same problem but with the Windows Version. The last Blocks will not be loaded.

instance: Geth/v1.7.2-stable-1db4ecdc/windows-amd64/go1.9 modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txp ool:1.0 web3:1.0

eth.syncing { currentBlock: 4564714, highestBlock: 4564788, knownStates: 1223685, pulledStates: 1195180, startingBlock: 4564561 }

can you help me in this case?

[karalabe]

It is downloading the state trie. There are over 30M state entries to pull before it can finalize the last blocks.

On Nov 16, 2017 20:10, "MobiD2017" notifications@github.com wrote:

I have the same problem but with the Windows Version. The last Blocks will not be loaded.

instance: Geth/v1.7.2-stable-1db4ecdc/windows-amd64/go1.9 modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txp ool:1.0 web3:1.0

eth.syncing { currentBlock: 4564714, highestBlock: 4564788, knownStates: 1223685, pulledStates: 1195180, startingBlock: 4564561 }

can you help me in this case?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-345009205, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH6GUHDTzj5HPohPZlmuN3Cn6cV0bHfks5s3HqEgaJpZM4O7ctW .

[tomtom87]

use ssd or it never sync

On 17 Nov 2017, at 01:14, Péter Szilágyi notifications@github.com wrote:

It is downloading the state trie. There are over 30M state entries to pull before it can finalize the last blocks.

On Nov 16, 2017 20:10, "MobiD2017" notifications@github.com wrote:

I have the same problem but with the Windows Version. The last Blocks will not be loaded.

instance: Geth/v1.7.2-stable-1db4ecdc/windows-amd64/go1.9 modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txp ool:1.0 web3:1.0

eth.syncing { currentBlock: 4564714, highestBlock: 4564788, knownStates: 1223685, pulledStates: 1195180, startingBlock: 4564561 }

can you help me in this case?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-345009205, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH6GUHDTzj5HPohPZlmuN3Cn6cV0bHfks5s3HqEgaJpZM4O7ctW .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[MobiD2017]

@tomtom87

The system is installed and running on SSD. This night the System was automatically rebooted. If I restart the process with "geth –rpc" new chain are imported: INFO [11-17|06:40:03] Imported new chain segment blocks=1 txs=128 mgas=6.107 elapsed=240.612ms mgasps=25.379 number=4567898 hash=1f699c.19ef7b INFO [11-17|06:40:09] Imported new chain segment blocks=1 txs=27 mgas=0.842 elapsed=54.201ms mgasps=15.539 number=4567899 hash=bceae4.2b190d INFO [11-17|06:40:09] Imported new chain segment blocks=1 txs=86 mgas=6.685 elapsed=157.207ms mgasps=42.524 number=4567899 hash=11a9a7.561e6e INFO [11-17|06:40:10] Imported new chain segment blocks=1 txs=80 mgas=3.048 elapsed=151.008ms mgasps=20.183 number=4567900 hash=35129d.95609f

but the "geth attach eth.syncing" is showing:

eth.syncing { currentBlock: 4567699, highestBlock: 4567709, knownStates: 0, pulledStates: 0, startingBlock: 4567694 } eth.syncing false eth.syncing false

[tomtom87]

Ok, I have had this problem before and solved by using docker image (and uncountable restarts)

On 17/11/2017 12:47, MobiD2017 wrote:

@tomtom87 https://github.com/tomtom87

The system is installed and running on SSD. This night the System was automatically rebooted. If I restart the process with "geth –rpc" new chain are imported: INFO [11-17|06:40:03] Imported new chain segment blocks=1 txs=128 mgas=6.107 elapsed=240.612ms mgasps=25.379 number=4567898 hash=1f699c.19ef7b INFO [11-17|06:40:09] Imported new chain segment blocks=1 txs=27 mgas=0.842 elapsed=54.201ms mgasps=15.539 number=4567899 hash=bceae4.2b190d INFO [11-17|06:40:09] Imported new chain segment blocks=1 txs=86 mgas=6.685 elapsed=157.207ms mgasps=42.524 number=4567899 hash=11a9a7.561e6e INFO [11-17|06:40:10] Imported new chain segment blocks=1 txs=80 mgas=3.048 elapsed=151.008ms mgasps=20.183 number=4567900 hash=35129d.95609f

but the "geth attach eth.syncing" is showing:

eth.syncing
{
currentBlock: 4567699,
highestBlock: 4567709,
knownStates: 0,
pulledStates: 0,
startingBlock: 4567694
}
eth.syncing
false
eth.syncing
false

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-345151503, or mute the thread https://github.com/notifications/unsubscribe-auth/AB7p9_o_mOyz0ejdD3hz3UZ0s7pRi4qVks5s3R3wgaJpZM4O7ctW.

[karalabe]

Seems you are indeed in sync. Odd that the eth.syncing endpoint still tells you otherwise. I'll need to take a look if that's indeed the case. Could you double check please?

[MobiD2017]

I have restarted the sync after a removedb, the result is the same.

the data for:
knownStates: 0, pulledStates: 0,

will not be loaded.

I have restarted the sync process with geth --fast --cache=1024, the result is: INFO [11-19|21:09:40] Imported new chain segment blocks=4 txs=522 mgas=23.742 elapsed=2.536s mgasps=9.361 number=4584075 hash=8679be.584ea3 INFO [11-19|21:09:40] Imported new chain segment blocks=1 txs=0 mgas=0.000 elapsed=7.000ms mgasps=0.000 number=4584076 hash=03e044.93ba60

eth.syncing shows me: currentBlock: 4584043, highestBlock: 4584048, knownStates: 0, pulledStates: 0, startingBlock: 4584043

can you help me in this case? For me its looks like im sync without the last blocks but it doesn't work.

[tomtom87]

i synced a fresh node last week it took approx 2 days just under 48 hours. first check you have ssd and can connect to the ethereum network, if its still not working and you are on mac osx try the docker image. Works for me after several soft restarts (not deleting chain data just starting docker again)

On 20 Nov 2017, at 03:13, MobiD2017 notifications@github.com wrote:

I have restarted the sync after a removedb, the result is the same.

the data for: knownStates: 0, pulledStates: 0,

will not be loaded.

I have restarted the sync process with geth --fast --cache=1024, the result is: INFO [11-19|21:09:40] Imported new chain segment blocks=4 txs=522 mgas=23.742 elapsed=2.536s mgasps=9.361 number=4584075 hash=8679be.584ea3 INFO [11-19|21:09:40] Imported new chain segment blocks=1 txs=0 mgas=0.000 elapsed=7.000ms mgasps=0.000 number=4584076 hash=03e044.93ba60

eth.syncing shows me: currentBlock: 4584043, highestBlock: 4584048, knownStates: 0, pulledStates: 0, startingBlock: 4584043

can you help me in this case? For me its looks like im sync without the last blocks but it doesn't work.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[MobiD2017]

yesterday evening after my posting I have removed the db and start a fresh sync. All Data will be stored on SSD and I have a windows system.

Started with the following command: geth --fast --cache=2048

eth.syncing show me:

{ currentBlock: 1105238, highestBlock: 4584613, knownStates: 1908357, pulledStates: 1895114, startingBlock: 0 }

This morning the sync block is higher then 4584613.INFO [11-20|05:59:05] Imported new chain segment blocks=1 txs=34 mgas=6.660 elapsed=145.607ms mgasps=45.738 number=4586397 hash=2428ac.0ca952 INFO [11-20|05:59:16] Imported new chain segment blocks=1 txs=36 mgas=6.420 elapsed=119.606ms mgasps=53.680 number=4586398 hash=e49bcf.21b785 INFO [11-20|05:59:25] Imported new chain segment blocks=1 txs=43 mgas=1.941 elapsed=80.004ms mgasps=24.263 number=4586399 hash=c4c262.45254b

eth.syncing show me on "false". If I restart the sync process nothing will change. Only I get the following Warning message: WARN [11-20|05:53:40] Blockchain not empty, fast sync disabled

[tomtom87]

Oh its because you using windows bruh

Just playing, but really is probably somethingto do with it. Try using docker for the node. On linux or mac its take 48 hours currently from fresh install i done two on thursday last week.

Basically its syncing just at a gnats pace.

On 20 Nov 2017, at 12:03, MobiD2017 notifications@github.com wrote:

yesterday evening after my posting I have removed the db and start a fresh sync. All Data will be stored on SSD and I have a windows system.

Started with the following command: geth --fast --cache=2048

eth.syncing show me:

{ currentBlock: 1105238, highestBlock: 4584613, knownStates: 1908357, pulledStates: 1895114, startingBlock: 0 }

This morning the sync block is higher then 4584613.INFO [11-20|05:59:05] Imported new chain segment blocks=1 txs=34 mgas=6.660 elapsed=145.607ms mgasps=45.738 number=4586397 hash=2428ac.0ca952 INFO [11-20|05:59:16] Imported new chain segment blocks=1 txs=36 mgas=6.420 elapsed=119.606ms mgasps=53.680 number=4586398 hash=e49bcf.21b785 INFO [11-20|05:59:25] Imported new chain segment blocks=1 txs=43 mgas=1.941 elapsed=80.004ms mgasps=24.263 number=4586399 hash=c4c262.45254b

eth.syncing show me on "false". If I restart the sync process nothing will change. Only I get the following Warning message: WARN [11-20|05:53:40] Blockchain not empty, fast sync disabled

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[TillDev]

ok, the final point is: this project is sucks, developers are disabled people. don't posting, don't fixing. maybe this project made for you not to mining as long time as it possible. trying to find better solution. thank you for killed my time

[holiman]

@MobiD2017 You are (were) in sync at block 4586399. What makes you think there's a problem?

[bibz0r]

Been stuck on "INFO [11-30|20:21:22] Imported new state entries" for a few days already.

This project should be abandoned imho; This is bad for the Ethereum project, since they call it "Official wallet"...

[mshvarts]

same problem.. what a waste of time.. Doesnt sync last 150-200 blocks even on an SSD and very fast network.. keeps importing new state entries or dropping peers/stalling.. I Tried restarting, switching between fast and normal mode, and looking for solutions online. Nothing works, this sucks. also knownStates and pulledStates keep increasing endlessly..

[holiman]

So,

• Please provide some snippets of logs. It's difficult to say anything based on a report saying just "me to this sucks". It does not help us solve anything. We need to figure out if people are having problems finding peers, or problems downloading data from peers, or if the problem is slow block processing, or if the problem is disk IO. Logs help us do that.

At least one person in this thread was apparently already in sync when reporting the bug.

[mshvarts]

@holiman i restarted it a few times and now its not stalling anymore, now its importing new chain segments and i think its done. Maybe this is how its supposed to go, but It doesn't make sense that I sync'd like 4,000,000 blocks (99% done) in like 2 hours but to sync the last <200 blocks its taking more than 5 hours.. Why is that a thing, i'm pretty sure it would take days or weeks on an normal HDD (because of the slow writing speed, what i was trying to do before) which is absurd. edit: but now it finally works just gonna be patient i guess. Also i'm almost out of space because the folder is 39 GB, was expecting it to be around 20gb

[tomtom87]

It never finishes on a mechanical hdd and barely on a ssd now

Only gonna get worse as ethereum gets more popular

On 6 Dec 2017, at 15:47, ugotjelly notifications@github.com wrote:

@holiman i restarted it a few times and now its not stalling anymore, now its importing new chain segments and i think almost done. Maybe this is how its supposed to go, but It doesn't make sense that I sync'd like 4,000,000 blocks (99% done) in like 2 hours but to sync the last <200 blocks its taking more than 5 hours.. Why is that a thing, i'm pretty sure it would take days or weeks on an normal HDD (what i was trying to do before) which is absurd

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[tomtom87]

There are logs??...

On 06/12/2017 15:15, Martin Holst Swende wrote:

So,

• Please provide some snippets of logs. It's difficult to say anything based on a report saying just "me to this sucks". It does not help us solve anything. We need to figure out if people are having problems finding peers, or problems downloading data from peers, or if the problem is slow block processing, or if the problem is disk IO. Logs help us do that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-349566475, or mute the thread https://github.com/notifications/unsubscribe-auth/AB7p99wvSVS3VE0QhGB7W-uwMdXoX37oks5s9k0mgaJpZM4O7ctW.

[Dirksterson]

@holiman : After reading this and other threads it seems to be clear there are solid hardware and connectivity requirements to sync the ETH blockchain. E.g. physical hard drives vs. SSD. I gave up on my attempts to try syncing with a new off the shelve MacMini equipped with just a physical drive shortly after starting this thread. It would be great to have a recommendation from the Ethereum team on minimum hardware requirements today plus a recommendation for required hardware for the near and mid term future. Logs of the slower machine all the way up in this thread.

I upgraded to a machine with a SSD and a 3.5Ghz i7 on a 50Mbit DSL line. That worked just fine a few months back overnight. However, last nights sync also didn't complete with a different error message now. "Bad Block". Here is a snippet:

WARN [12-06|13:47:57] Synchronisation failed, dropping peer    peer=76fac33c6494dc8f err="retrieved hash chain is invalid"
ERROR[12-06|13:48:36] 
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis: 9223372036854775807 Engine: ethash}

Number: 4370000
Hash: 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e

Error: invalid difficulty: have 2994347070619309, want 2998009606615050
##############################

WARN [12-06|13:48:36] Synchronisation failed, dropping peer    peer=76fac33c6494dc8f err="retrieved hash chain is invalid"
ERROR[12-06|13:48:45] 
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis: 9223372036854775807 Engine: ethash}

Number: 4370000
Hash: 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e

Error: invalid difficulty: have 2994347070619309, want 2998009606615050
##############################

WARN [12-06|13:48:45] Synchronisation failed, dropping peer    peer=1c92bbdd230e8683 err="retrieved hash chain is invalid"
ERROR[12-06|13:50:06] 
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis: 9223372036854775807 Engine: ethash}

Number: 4370000
Hash: 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e

Error: invalid difficulty: have 2994347070619309, want 2998009606615050
##############################

WARN [12-06|13:50:06] Synchronisation failed, dropping peer    peer=6eb61fa67b147e73 err="retrieved hash chain is invalid"

[holiman]

@Dirksterson the block 4370000 is the fork-block for Byzantium, and the block 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e is indeed the canonical block.

The problem is that you're using some old version of geth, which is not configured for Byzantium: Metropolis: 9223372036854775807

A common source of confusion is when someone installs geth, via e..g apt install geth or go install geth, and then later on builds from source, but keeps using the old binary which resides somewhere in the path.

[holiman]

@tomtom87 yes, there are logs -- in case you're starting geth via mist, there are some node.log or something similar which contains info about what's happened.

@ugotjelly it shouldn't go that slow, the first 4M+ blocks are fast-sync to the 'pivot point', after that is reached, every block is processed individually. Still, without logs, it's hard to say if it's due to block-starvation or simply that your machine can't process the blocks quickly enough.

[tomtom87]

I get the dropping peer hash chain is invalid error for months now, you just have to start over. Really not sure what is causing it. Start the docker again and try syncing. You do not need to delete chain data.

On 06/12/2017 19:55, Dirksterson wrote:

@holiman https://github.com/holiman : After reading this and other threads it seems to be clear there are solid hardware and connectivity requirements to sync the ETH blockchain. E.g. physical hard drives vs. SSD. I gave up on my attempts to try syncing with a new off the shelve MacMini equipped with just a physical drive shortly after starting this thread. It would be great to have a recommendation from the Ethereum team on minimum hardware requirements today plus a recommendation for required hardware for the near and mid term future. Logs of the slower machine all the way up in this thread.

I upgraded to a machine with a SSD and a 3.5Ghz i7 on a 50Mbit DSL line. That worked just fine a few months back overnight. However, last nights sync also didn't complete with a different error message now. "Bad Block". Here is a snippet:

WARN [12-06|13:47:57] Synchronisation failed, dropping peer peer=76fac33c6494dc8f err="retrieved hash chain is invalid" ERROR[12-06|13:48:36] ########## BAD BLOCK ######### Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis: 9223372036854775807 Engine: ethash}

Number: 4370000 Hash: 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e

Error: invalid difficulty: have 2994347070619309, want 2998009606615050 ##############################

WARN [12-06|13:48:36] Synchronisation failed, dropping peer peer=76fac33c6494dc8f err="retrieved hash chain is invalid" ERROR[12-06|13:48:45] ########## BAD BLOCK ######### Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis: 9223372036854775807 Engine: ethash}

Number: 4370000 Hash: 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e

Error: invalid difficulty: have 2994347070619309, want 2998009606615050 ##############################

WARN [12-06|13:48:45] Synchronisation failed, dropping peer peer=1c92bbdd230e8683 err="retrieved hash chain is invalid" ERROR[12-06|13:50:06] ########## BAD BLOCK ######### Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis: 9223372036854775807 Engine: ethash}

Number: 4370000 Hash: 0xb1fcff633029ee18ab6482b58ff8b6e95dd7c82a954c852157152a7a6d32785e

Error: invalid difficulty: have 2994347070619309, want 2998009606615050 ##############################

WARN [12-06|13:50:06] Synchronisation failed, dropping peer peer=6eb61fa67b147e73 err="retrieved hash chain is invalid"

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-349631420, or mute the thread https://github.com/notifications/unsubscribe-auth/AB7p97qzzPsWWBkqMJzxls0Z5adwFssYks5s9o7WgaJpZM4O7ctW.

[Dirksterson]

@holiman Thank you bunches for pointing that out!!! Greatly appreciated!!! FYI: Your latest Ethereum wallet 0.9.3 (Mac OSX) keeps using geth 1.6.6 even after trashing geth folder and reinstalling. I'm trying now with a clean install of Mist 0.9.3, using geth 1.7.2. This should support Byzantium and address the canonical block issue.

Any comment from anyone in the Ethereum team on minimum hardware requirements for all the Ethereum enthusiasts out there that will stumble over this issue? Apparently I wasn't alone, I assume this is a huge trap to fall into for many. Also important for future hardware purchasing considerations.

Will keep this thread posted on progress! Thanks again, Martin!!!

Cheers! Dirk

[holiman]

You're welcome :)

Your latest Ethereum wallet 0.9.3 (Mac OSX) keeps using geth 1.6.6 even after trashing geth folder and reinstalling.

@evertonfraga any ideas what would cause that ^ ?

[raahil190]

Same issue - keeps dropping peers and sync is behing by a 100-200 blocks even though eth.blockNumber seems to think its in sync (etherscan.io confirms its not)

[Dirksterson]

I got it to work with a hardware upgrade. It syncs today nicely on my MacPro with an I7 3.5Ghz and a SSD via a VDSL connection. To me it seems there are minimum hardware requirements. I have asked several times on a statement from the Ethereum guys. They are usually very helpful (e.g. Byzantium hint on this thread or many, many other threads). However, they seem to be very shy when it comes to a guideline here.

I suggest to everyone with this issue here to upgrade to a SSD hard drive and use a machine that has a processor that is not much older than from a few years ago. And of course some decent internet connection. That will do the trick for most for today. I'm a bit worried how long this will be sufficient.

[Dirksterson]

Still no comment on hardware and uplink requirements from Ethereum team. Similar issue on other threads if you google. Oh well....

I keep on updating this thread here with my personal experience, hope this helps you people out there not running into the same traps as I did. Yesterday I tried to sync again with my fast machine that currently syncs easy (see above) but now on a satellite link. Observation: Fail! I measure about 6Mbit/s up- and downstream. But with a 700ms delay (since packets have to travel to space). After successful syncing to block 4723322 (thank you cryptokitties) it stalled. I gave up after 10 hours. Drove an hour today back to civilisation to connect to a 5-7 Mbit DSL line. Progressing sync again. :)

Summary: Satellite link with 6 Mbit connection with 700ms delay NOT sufficient for syncing.

Snippet:

INFO [12-18|22:37:10] Imported new chain segment blocks=5 txs=1027 mgas=35.948 elapsed=3.329s mgasps=10.798 number=4723322 hash=912c52…af50cc WARN [12-18|22:37:10] Synchronisation failed, dropping peer peer=a979fb575495b8d6 err="retrieved hash chain is invalid" WARN [12-18|22:37:10] Skipping deep transaction reorg depth=81 INFO [12-18|23:31:59] Regenerated local transaction journal transactions=0 accounts=0 INFO [12-19|00:31:59] Regenerated local transaction journal transactions=0 accounts=0 INFO [12-19|01:31:59] Regenerated local transaction journal transactions=0 accounts=0

[tomtom87]

try to use light client and fast sync... Best use datacenter machine and login remotely via mist command line

On 20 Dec 2017, at 07:46, Dirksterson notifications@github.com wrote:

Still no comment on hardware and uplink requirements from Ethereum team. Similar issue on other threads if you google. Oh well....

I keep on updating this thread here with my personal experience, hope this helps you people out there not running into the same traps as I did. Yesterday I tried to sync again with my fast machine that currently syncs easy (see above) but now on a satellite link. Observation: Fail! I measure about 6Mbit/s up- and downstream. But with a 700ms delay (since packets have to travel to space). After successful syncing to block 4723322 (thank you cryptokitties) it stalled. I gave up after 10 hours. Drove an hour today back to civilisation to connect to a 5-7 Mbit DSL line. Progressing sync again. :)

Summary: Satellite link with 6 Mbit connection with 700ms delay NOT sufficient for syncing.

Snippet:

INFO [12-18|22:37:10] Imported new chain segment blocks=5 txs=1027 mgas=35.948 elapsed=3.329s mgasps=10.798 number=4723322 hash=912c52…af50cc WARN [12-18|22:37:10] Synchronisation failed, dropping peer peer=a979fb575495b8d6 err="retrieved hash chain is invalid" WARN [12-18|22:37:10] Skipping deep transaction reorg depth=81 INFO [12-18|23:31:59] Regenerated local transaction journal transactions=0 accounts=0 INFO [12-19|00:31:59] Regenerated local transaction journal transactions=0 accounts=0 INFO [12-19|01:31:59] Regenerated local transaction journal transactions=0 accounts=0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[evertonfraga]

@holiman missed that notification badly, sorry for the delay.

We're aware of this issue, aiming to fix it for 0.9.4.

Thanks

[garyng2000]

@evertonfraga

What do you mean by 0.9.4 ? This AFAIK is a geth issue and not Ethereum Wallet. I have 3 node running at three different location(countries) and all encountered syncing issues. Can stall for a few hours then start the catch up game which can barely keep up with the delay(as blocks continued to be created when the syncing stalled). It is kind of like when it functions normally, the node is in sync so can be used for submission of transaction etc. Then it stalled and with luck it can resume again but catch up the blocks and during this period, not usable. So work for a few hours, wait for a few hours and the cycle continue. In a few cases, completely stalled which requires restart. Need to give parity a try to see if that is better

[evertonfraga]

@garyng2000 I was referring to the Ethereum Wallet, in this comment:

Your latest Ethereum wallet 0.9.3 (Mac OSX) keeps using geth 1.6.6 even after trashing geth folder and reinstalling. (https://github.com/ethereum/go-ethereum/issues/15001#issuecomment-349664186)

[garyng2000]

ah thanks. there is an embedded version ? Anyway, I myself always started geth in a console then the app which IMO is a better way to isolate the problem. It is usually the backend(geth/parity) that is the problem.

[surferboy250]

I had the same issue with Ubuntu 16.04 LTS and Geth 1.7.3, but could resolve it. Maybe it will help beginners too. In short: be patient and expect a lot of data to be downloaded.

1) Download "https://gethstore.blob.core.windows.net/builds/geth-linux-amd64-1.7.3-4bb3c89d.tar.gz" binary for Linux (or 32 bit version). Unpack, open a terminal and "cd" into the folder that contains the "geth" binary. Then run: ./geth --fast --cache=1048 console

2) Open another terminal. cd .ethereum watch -n 2 du -h .

3) Now wait while "geth" is running and watch the "chaindata" folder grow. I own a new fast laptop with 8 Cores and NVMe. After two or three hours it nearly reached 100 percent, but not completely. I let it run overnight, next morning it was showing latest block (you will find latest block on "etherscan.io"). In the meantime "chaindata" folder was grown to 109 Gigabyte, so make sure you have enough place on your disc. Today it will be even bigger, with older computers and slow internet connection it might take several days.

4) Download https://github.com/ethereum/mist/releases/download/v0.9.3/Ethereum-Wallet-linux64-0-9-3.zip from Github, unzip ZIP archive, change into "linux-unpacked" folder and do ./ethereumwallet. It will connect with the running instance of Geth and the interface will pop up. If you already have an existing wallet, put it in .ethereum/keystore.

[suspended]

Still having problems - always stalls within 200 blocks of latest block. After that, it just downloads state entries forever

[cecilia-to]

downloading states is normal, just need to wait. I think now is at least 60M. just wait, we have recently do it twice and at the end of the day, it did finished.

[tomtom87]

I had problems earlier with a node loosing sync. Restarted and in 3 hours it had managed to catch up. SSD is struggling now really bad ... message to the bridge we just canny hold her captain we dont have the powaaahh

On 10 Jan 2018, at 00:06, ceciliato310 notifications@github.com wrote:

downloading states is normal, just need to wait. I think now is at least 60M. just wait, we have recently do it twice and at the end of the day, it did finished.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[MrHash]

I. Must. Have. Warp. Sync, Scotty...

<wormhole!>

[legkodymov]

Any resolution? I can provide my setup for analysis. Can't solve it for a week.

[suspended]

Scrap Geth and use Parity