search

ethereum / go-ethereum

Go implementation of the Ethereum protocol
https://geth.ethereum.org
GNU Lesser General Public License v3.0
47.19k stars 19.98k forks source link

geth --fast stalls before crossing finish line #15001

Closed Dirksterson closed 5 years ago

Dirksterson commented 7 years ago

System information

Geth version: geth version 1.5.9-stable, Go1.7.4 OS & Version: OSX 10.12.6 MacMini 4GB RAM (latest MacMini doesn't support field RAM upgrade anymore) VDSL connection with an average of 20-40Mbit throughput. Ethereum Wallet 0.9.0 Commit hash : (if develop)

Expected behaviour

fast sync to current latest block followed by auto disabling

Actual behaviour

stalling from a few thousand blocks up to a few hundred to current latest block. Tries to catch up to latest block, but number of new blocks is greater than the speed of adding fast blocks. Never auto disables fast sync mode.

Steps to reproduce the behaviour

Removedb and geth --fast --cache=1024. 5 times on that machine over the last weeks.

Fast sync is already my workaround, starting a fresh fast sync from scratch. Before I was unsuccessful on that machine trying to sync with existing blockchain data instead. This was also a lost race of catching up to the latest block on that machine. This workaround was good until now.

Today even the workaround in fast sync mode (cache -1024) will not completely load the blockchain anymore. It catches up some hundred blocks to the latest block and stalls for hours. By the time it catches up a few hundred blocks, the latest block moved ahead again. The closer geth is getting to import to the latest block (at time of writing 4173161), the slower it gets. It does not catch up anymore. Tried 5 times now over the last weeks and giving up at around 4-5 days each.

Does the machine not meet todays minimum hardware requirement anymore or is this a major bug?

Backtrace

latest block 13 hours ago (!)

I0818 00:15:26.444933 core/blockchain.go:805] imported 148 receipts in 2.775s. #4169952 [e3f556fc… / 36f4d3c9…]

...

latest header chain 50 minutes ago

I0818 12:47:45.107445 core/headerchain.go:342] imported 1 headers in 4.954ms. #4173009 [350d1426… / 350d1426…]

...

currently only importing nothing but state entries

I0818 13:36:41.103101 eth/downloader/downloader.go:966] imported 172 state entries in 10.009s: processed 10010213, pending at least 129361 I0818 13:36:41.103131 eth/downloader/downloader.go:966] imported 384 state entries in 783.519ms: processed 10010597, pending at least 129361 I0818 13:36:41.103154 eth/downloader/downloader.go:966] imported 381 state entries in 6.963s: processed 10010978, pending at least 129361 I0818 13:36:41.103167 eth/downloader/downloader.go:966] imported 25 state entries in 87.654ms: processed 10011003, pending at least 129360 I0818 13:36:46.014244 eth/downloader/downloader.go:966] imported 384 state entries in 2.482s: processed 10011387, pending at least 127584 I0818 13:36:49.074483 eth/downloader/downloader.go:966] imported 381 state entries in 7.082s: processed 10011768, pending at least 127105 I0818 13:36:49.074553 eth/downloader/downloader.go:966] imported 384 state entries in 7.971s: processed 10012152, pending at least 127105 I0818 13:36:49.074574 eth/downloader/downloader.go:966] imported 384 state entries in 3.772s: processed 10012536, pending at least 127105 I0818 13:36:49.074603 eth/downloader/downloader.go:966] imported 162 state entries in 5.822s: processed 10012698, pending at least 127105 I0818 13:36:49.074622 eth/downloader/downloader.go:966] imported 25 state entries in 4.050s: processed 10012723, pending at least 127105 I0818 13:36:49.074639 eth/downloader/downloader.go:966] imported 381 state entries in 3.060s: processed 10013104, pending at least 127105 I0818 13:36:49.074742 eth/downloader/downloader.go:966] imported 85 state entries in 7.117s: processed 10013189, pending at least 127105 I0818 13:36:49.074765 eth/downloader/downloader.go:966] imported 375 state entries in 2.219s: processed 10013564, pending at least 127105 I0818 13:36:49.074782 eth/downloader/downloader.go:966] imported 87 state entries in 3.915s: processed 10013651, pending at least 127105 I0818 13:36:49.074795 eth/downloader/downloader.go:966] imported 23 state entries in 271.734ms: processed 10013674, pending at least 127104

kevingentile commented 7 years ago

I have been having a similar issue recently. Ubuntu 16.04. Stalling on the last ~100-200 blocks. Restarting the geth client has allowed for some of those missing blocks to be processed but it does not keep up with the highest block. The only fluctuation I see in eth.syncing is the number of knownStates and pulledStates.

Shem-Tov commented 7 years ago

I am having the exact same issue as Laughing Cabbage has described, also on Ubuntu 16.04, and also stuck on the last few hundred blocks. I am running geth1.6.7, at the moment. I have also tried versions 1.7.0, 1.6.6 and 1.6.5, with the same issue. I have tried applying --fast, and have tried without it. When I restart geth, it usually gets a few more blocks in, and starts "downloading" the chain structure from 0. Downloading is in parenthesis, because when checking the folder into which it should be downloading, the folder access date and time does not change, nor can I find any other folder to which it saves the chain structure to. Leaving it overnight, will get a chain structure in the millions, but the blocks will still not sync. Searching the web, I have seen this problem exists for many people, for a very long time, across every platform, and with every version of geth, and no one has come up with any kind of solution. And since I am at best an amateur programmer, I have given up with geth. I will try parity.io now, hopefully they have allowed people with little and no programming skills to connect to ethereum, and if not, then my solution is to give up on ethereum all together. That will solve the headache this issue is starting to create :-) I'll check back with geth when in reaches version 2.

kevingentile commented 7 years ago

If any current devs think they might have a lead as to where a good starting point might be for tracking this issue I'm happy to do some bug hunting, please let me know.

tomtom87 commented 7 years ago

@Dirksterson @LaughingCabbage I have exactly the same issues for past week and so do many of my colleagues.

After latest advice to run --fast --cache=1024 i now get the following:

WARN [08-21|11:48:26] Stalling state sync, dropping peer       peer=655c0278c317a012
WARN [08-21|11:48:26] Stalling state sync, dropping peer       peer=f26dce0aea871dc8
WARN [08-21|11:48:26] Stalling state sync, dropping peer       peer=0fb49536fda319d3
WARN [08-21|11:48:27] Stalling state sync, dropping peer       peer=ae8de9feee4df4e6
WARN [08-21|11:48:27] Stalling state sync, dropping peer       peer=e7a69c447cb83857
WARN [08-21|11:48:30] Stalling state sync, dropping peer       peer=8e8edc9627fedc6b
WARN [08-21|11:48:32] Stalling state sync, dropping peer       peer=606587b48a16fd10
WARN [08-21|11:48:32] Node data write error                    err="state node 638deb…cf0f09 failed with all peers (4 tries, 4 peers)"
csillag commented 7 years ago

Same here. I am also on v1.6.7.

Current status, after running it for more than a week:

Downloading block 4,179,697 of 4,179,911, Downloading chain structure 8,242,414 of 8,246,476

csillag commented 7 years ago

Isn't this a duplicate of #14988 and also #14995?

darksh1ne commented 7 years ago

The similar issue here. On Aug, 16th I had almost fully synced blockchain, just 10-20 hours behind the current block. I then started geth as:

$ geth --syncmode=fast --cache=$(( 1024 + 512 ))

All the time geth is behind the current block. Currently (Aug, 21st) its state is:

> eth.syncing
{
  currentBlock: 4181084,
  highestBlock: 4182536,
  knownStates: 0,
  pulledStates: 0,
  startingBlock: 4179967
}

whereas etherscan.io shows 4185672 as the last block.

There are no errors in geth's output, it is in its normal state of slowly importing new segments and using HDD at speed 5-10 MB/s (both reading and writting). No high CPU usage.

INFO [08-21|14:27:00] Imported new chain segment               blocks=1 txs=60  mgas=6.645  elapsed=26.766s   mgasps=0.248  number=4181082 hash=036737…8ef0ce
INFO [08-21|14:27:16] Imported new chain segment               blocks=1 txs=77  mgas=1.748  elapsed=16.123s   mgasps=0.108  number=4181083 hash=d498b7…8c64a9
INFO [08-21|14:28:44] Imported new chain segment               blocks=1 txs=137 mgas=6.699  elapsed=1m28.060s mgasps=0.076  number=4181084 hash=b8153c…a3bcbf
INFO [08-21|14:30:44] Imported new chain segment               blocks=1 txs=62  mgas=6.691  elapsed=1m59.831s mgasps=0.056  number=4181085 hash=4e7b58…7f71d5

My geth is:

$ geth attach
Welcome to the Geth JavaScript console!

instance: Geth/v1.6.7-stable/linux-amd64/go1.8
coinbase: <hidden>
at block: 4166508 (Wed, 16 Aug 2017 23:59:48 EEST)
 datadir: <hidden>
 modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0
$ geth version
Geth
Version: 1.6.7-stable
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.8
Operating System: linux
GOPATH=
GOROOT=/usr/lib/go
tomtom87 commented 7 years ago

same issue here, started around the same time. Looks like this is throughout everyone and affecting parity users also now

gdassori commented 7 years ago

Hello, Ubuntu 16.04 here and same issue: got stuck on the last ~2000 blocks.

tomtom87 commented 7 years ago

If you dont got ssd u aint ever going to get them.

If u got ssd, just constantly restart the docker container and your client and pray. eventually after several days. You must be persistant.. it will crash randomly and then when you reopen it will be syncing.

Really it took me 2 weeks to do this.

On 28 Aug 2017, at 21:11, dax notifications@github.com wrote:

Hello, Ubuntu 16.04 here and same issue: got stuck on the last ~2000 blocks.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

MrHash commented 7 years ago

Same problem. Can't sync last ~100 blocks on 1.6.7. Restarting gets close but lots of Stalling state sync, dropping peer messages. SSD and fibre connection.

tomtom87 commented 7 years ago

Try using a ssd drive and docker image. This is working for me, expect atleast 4 hours to sync up. Sundays when volume is low is a good time to try

On 1 Sep 2017, at 20:01, Hasham Ahmad notifications@github.com wrote:

Same problem. Can't sync last ~100 blocks on 1.6.7. Restarting gets close but lots of Stalling state sync, dropping peer messages

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

wtfiwtz commented 7 years ago

Is this related? https://github.com/ethereum/go-ethereum/commit/0042f13d47700987e93e413be549b312e81854ac and https://github.com/ethereum/go-ethereum/pull/14460

alfkors commented 7 years ago

@wtfiwtz I don't really know enough about the whole process, but I would say yeah probably... for what it's worth...

wtfiwtz commented 7 years ago

I was able to get it to successfully sync up, after switching from fast sync to normal sync and giving it a day or two to catch up on the last 35,000 or so blocks - using a 2012-era MacBook Pro with an SSD drive. It was necessary to be on the the latest block to be able to successfully submit a transaction with the Ethereum "Mist" wallet (or you get an error about insufficient gas).

Not sure if the light mode would make any difference, but I think you need to do it will a brand new wallet, not an existing blockchain download.

wtfiwtz commented 6 years ago

Ok I had to restart the sync from the beginning and have hit this problem again...

This is what I have found... Blocks are getting discarded from peers because the chain height is incorrectly set to 0.

https://github.com/wtfiwtz/go-ethereum/commit/f34b7758e52e84564693d42ee28b673ff86701f6

INFO [09-23|23:16:30] Loaded most recent local full block      number=0       hash=d4e567…cb8fa3 td=17179869184
INFO [09-23|23:16:30] Loaded most recent local fast block      number=4304570 hash=657bf3…912f25 td=1006522706491316931004
INFO [09-23|23:21:06] Peer discarded announcement              peer=d9c3012a7a0dfb3f number=4304681 hash=7e9da0…8db154 distance=4304681
INFO [09-23|23:21:06] ** Block number                          num=4304681
INFO [09-23|23:21:06] ** Chain height                          num=0
WARN [09-23|23:21:06] Discarded propagated block, too far away peer=d9c3012a7a0dfb3f number=4304681 hash=7e9da0…8db154 distance=4304681
INFO [09-23|23:21:06] Peer discarded announcement              peer=a8aafc6f4437be4f number=4304681 hash=7e9da0…8db154 distance=4304681
INFO [09-23|23:21:06] Peer discarded announcement              peer=ea4587bcfb02c92d number=4304681 hash=7e9da0…8db154 distance=4304681
INFO [09-23|23:21:06] ** Block number                          num=4304681
INFO [09-23|23:21:06] ** Chain height                          num=0
WARN [09-23|23:21:06] Discarded propagated block, too far away peer=ea4587bcfb02c92d number=4304681 hash=7e9da0…8db154 distance=4304681

The height is retrieved from a callback function such as this:

    heighter := func() uint64 {
        return blockchain.CurrentBlock().NumberU64()
    }

So this is probably an issue with switching between fast and normal sync modes, where the chain height is assumed to be 0 when it should be equal to the fast chain height on initialization.

Is this an area you are familiar with @karalabe since you did the original fast sync implementation?

wtfiwtz commented 6 years ago

If the peer's total diffficulty is much lower, does that mean they are only on the full sync mode and won't work with a fast sync peer?

INFO [09-24|07:31:28] ** Total difficulty                      ours="{neg:false abs:[13700445755005557100 54]}" theirs=17179869184
INFO [09-24|07:31:28] ** fast sync?                            peer=3000f1cf9e63ce38 enabled=1

Pretty much can't find any peers that are not with a significantly lower total difficulty!

This worries me because of the following comment:

// synchronise will select the peer and use it for synchronising. If an empty string is given
// it will use the best peer possible and synchronize if it's TD is higher than our own. If any of the
// checks fail an error will be returned. This method is synchronous
func (d *Downloader) synchronise(id string, hash common.Hash, td *big.Int, mode SyncMode) error {
wtfiwtz commented 6 years ago

Ok I left it running this morning, and at some point, it flipped from fast to full mode when it received just 1 more chain segment or block receipts (it hadn't received any for at least 1.5 hours since I last restarted):

INFO [09-24|09:07:27] Peer discarded announcement              peer=b057fec043b525ed number=4305853 hash=37f333…4a74d9 distance=4305853
INFO [09-24|09:07:27] ** Total difficulty                      ours="{neg:false abs:[14195334218426315772 54]}" theirs=17179869184
INFO [09-24|09:07:27] ** fast sync?                            peer=b057fec043b525ed enabled=1
INFO [09-24|09:07:27] ** Block number                          num=4305854
INFO [09-24|09:07:27] ** Chain height                          num=0
WARN [09-24|09:07:27] Discarded propagated block, too far away peer=b057fec043b525ed number=4305854 hash=2e8a61…ae021f distance=4305854
INFO [09-24|09:07:27] Imported new state entries               count=448  elapsed=1.479ms   processed=2608239 pending=2047  retry=2   duplicate=2846 unexpected=8434
INFO [09-24|09:07:29] Imported new state entries               count=779  elapsed=3.995ms   processed=2609018 pending=2225  retry=22  duplicate=2846 unexpected=8434
INFO [09-24|09:07:29] ** Total difficulty                      ours="{neg:false abs:[14195334218426315772 54]}" theirs=17179869184
INFO [09-24|09:07:29] ** fast sync?                            peer=479032d8362da82d enabled=1
INFO [09-24|09:07:31] Imported new state entries               count=1089 elapsed=10.173ms  processed=2610107 pending=1483  retry=1   duplicate=2846 unexpected=8434
INFO [09-24|09:07:35] Imported new state entries               count=1081 elapsed=14.713ms  processed=2611188 pending=48    retry=0   duplicate=2846 unexpected=8434
INFO [09-24|09:07:35] Imported new state entries               count=35   elapsed=853.5µs   processed=2611223 pending=0     retry=0   duplicate=2846 unexpected=8434
INFO [09-24|09:07:35] Imported new block receipts              count=0    elapsed=3.752ms   bytes=0 number=4305451 hash=ac92d6…397f6c ignored=1
INFO [09-24|09:07:35] Committed new head block                 number=4305451 hash=ac92d6…397f6c
INFO [09-24|09:07:35] Imported new chain segment               blocks=1 txs=17 mgas=0.442 elapsed=28.174ms  mgasps=15.701 number=4305452 hash=4a61da…5f72e4
ERROR[09-24|09:07:35]
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 9223372036854775807 Engine: ethash}

Number: 4305453
Hash: 0x6c4471bed33ac85f132153650f4f69230e9ef972ff33cba1e79795fb72130c66

Error: unknown ancestor
##############################

WARN [09-24|09:07:35] Synchronisation failed, dropping peer    peer=cb8ebbf8130355a7 err="retrieved hash chain is invalid"
ERROR[09-24|09:07:35] Fast sync complete, auto disabling
INFO [09-24|09:07:35] Removing p2p peer                        id=cb8ebbf8130355a7 conn=inbound duration=1h32m36.442s peers=24 req=false err="useless peer"
INFO [09-24|09:07:36] Ethereum peer connected                  id=8453dbef52518caf conn=dyndial name=Geth/v1.6.7-stable-ab5646c5/linux-amd64/go1.8.1
INFO [09-24|09:07:36] ** Total difficulty                      ours="{neg:false abs:[14195334218426315772 54]}" theirs=1009137134152556054860
INFO [09-24|09:07:36] ** fast sync?                            peer=479032d8362da82d enabled=0
WARN [09-24|09:07:36] Ethereum handshake failed                id=8453dbef52518caf conn=dyndial err="Genesis block mismatch - 6577484f58748da6 (!= d4e56740f876aef8)"
INFO [09-24|09:07:36] Removing p2p peer                        id=8453dbef52518caf conn=dyndial duration=279.836ms    peers=24 req=false err="Genesis block mismatch - 6577484f58748da6 (!= d4e56740f876aef8)"
INFO [09-24|09:07:37] Peer discarded announcement              peer=ca40c7662d6ac5ed number=4305853 hash=37f333…4a74d9 distance=402
INFO [09-24|09:07:37] Peer discarded announcement              peer=ca40c7662d6ac5ed number=4305854 hash=2e8a61…ae021f distance=403
INFO [09-24|09:07:38] Ethereum peer connected                  id=6949cab8fc6d09bd conn=inbound name=Geth/v1.6.2-unstable-2a41e76b/linux-amd64/go1.8.3

The key log messages here are Committed new head block, Imported new block receipts and Imported new chain segment, which allows the full head blockchain count to update.

So I'm guessing that the network is starved of fast blocks, and they haven't yet reached their intended pivot point... before they flip to full mode.

Also note that you can't force it to use full mode on the command line, it doesn't work.

Is there some way to force this flipping from fast to full mode prematurely? Perhaps if we haven't received a new chain segment for over an hour? Or find a peer that has what we are looking for with a more broader peer search?

tomtom87 commented 6 years ago

I got this peer syncin problem constantly because i was not on usa time server but using asian one.

After changing my ntp time server settings would get 20 peers connecting - previous was 1 to 3. The peers connect but still same errors you show in log.

Currently only a few of our machines wallets will finish sync, latest macs newer then 2015 find it easiest. my 2011 mac is slowest. All have ssd. all are using fibre 100mb connections.

thanks for support

On 24 Sep 2017, at 09:20, Nigel Sheridan-Smith notifications@github.com wrote:

Ok I left it running this morning, and at some point, it flipped from fast to full mode when it received just 1 more chain segment:

INFO [09-24|09:07:27] Peer discarded announcement peer=b057fec043b525ed number=4305853 hash=37f333…4a74d9 distance=4305853 INFO [09-24|09:07:27] Total difficulty ours="{neg:false abs:[14195334218426315772 54]}" theirs=17179869184 INFO [09-24|09:07:27] fast sync? peer=b057fec043b525ed enabled=1 INFO [09-24|09:07:27] Block number num=4305854 INFO [09-24|09:07:27] Chain height num=0 WARN [09-24|09:07:27] Discarded propagated block, too far away peer=b057fec043b525ed number=4305854 hash=2e8a61…ae021f distance=4305854 INFO [09-24|09:07:27] Imported new state entries count=448 elapsed=1.479ms processed=2608239 pending=2047 retry=2 duplicate=2846 unexpected=8434 INFO [09-24|09:07:29] Imported new state entries count=779 elapsed=3.995ms processed=2609018 pending=2225 retry=22 duplicate=2846 unexpected=8434 INFO [09-24|09:07:29] Total difficulty ours="{neg:false abs:[14195334218426315772 54]}" theirs=17179869184 INFO [09-24|09:07:29] fast sync? peer=479032d8362da82d enabled=1 INFO [09-24|09:07:31] Imported new state entries count=1089 elapsed=10.173ms processed=2610107 pending=1483 retry=1 duplicate=2846 unexpected=8434 INFO [09-24|09:07:35] Imported new state entries count=1081 elapsed=14.713ms processed=2611188 pending=48 retry=0 duplicate=2846 unexpected=8434 INFO [09-24|09:07:35] Imported new state entries count=35 elapsed=853.5µs processed=2611223 pending=0 retry=0 duplicate=2846 unexpected=8434 INFO [09-24|09:07:35] Imported new block receipts count=0 elapsed=3.752ms bytes=0 number=4305451 hash=ac92d6…397f6c ignored=1 INFO [09-24|09:07:35] Committed new head block number=4305451 hash=ac92d6…397f6c INFO [09-24|09:07:35] Imported new chain segment blocks=1 txs=17 mgas=0.442 elapsed=28.174ms mgasps=15.701 number=4305452 hash=4a61da…5f72e4 ERROR[09-24|09:07:35] ########## BAD BLOCK ######### Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 9223372036854775807 Engine: ethash}

Number: 4305453 Hash: 0x6c4471bed33ac85f132153650f4f69230e9ef972ff33cba1e79795fb72130c66

Error: unknown ancestor ##############################

WARN [09-24|09:07:35] Synchronisation failed, dropping peer peer=cb8ebbf8130355a7 err="retrieved hash chain is invalid" ERROR[09-24|09:07:35] Fast sync complete, auto disabling INFO [09-24|09:07:35] Removing p2p peer id=cb8ebbf8130355a7 conn=inbound duration=1h32m36.442s peers=24 req=false err="useless peer" INFO [09-24|09:07:36] Ethereum peer connected id=8453dbef52518caf conn=dyndial name=Geth/v1.6.7-stable-ab5646c5/linux-amd64/go1.8.1 INFO [09-24|09:07:36] Total difficulty ours="{neg:false abs:[14195334218426315772 54]}" theirs=1009137134152556054860 INFO [09-24|09:07:36] fast sync? peer=479032d8362da82d enabled=0 WARN [09-24|09:07:36] Ethereum handshake failed id=8453dbef52518caf conn=dyndial err="Genesis block mismatch - 6577484f58748da6 (!= d4e56740f876aef8)" INFO [09-24|09:07:36] Removing p2p peer id=8453dbef52518caf conn=dyndial duration=279.836ms peers=24 req=false err="Genesis block mismatch - 6577484f58748da6 (!= d4e56740f876aef8)" INFO [09-24|09:07:37] Peer discarded announcement peer=ca40c7662d6ac5ed number=4305853 hash=37f333…4a74d9 distance=402 INFO [09-24|09:07:37] Peer discarded announcement peer=ca40c7662d6ac5ed number=4305854 hash=2e8a61…ae021f distance=403 INFO [09-24|09:07:38] Ethereum peer connected id=6949cab8fc6d09bd conn=inbound name=Geth/v1.6.2-unstable-2a41e76b/linux-amd64/go1.8.3 The key log messages here are Committed new head block and Imported new chain segment, which allows the full head blockchain count to update.

So I'm guessing that the network is starved of fast blocks, and they haven't yet reached their intended pivot point... before they flip to full mode.

Also note that you can't force it to use full mode on the command line, it doesn't work.

Is there some way to force this flipping from fast to full mode prematurely? Perhaps if we haven't received a new chain segment for over an hour? Or find a peer that has what we are looking for with a more broader peer search?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

wtfiwtz commented 6 years ago

You'll probably find it much easier to be on Parity (https://parity.io) - the wallet can do a light-mode sync in around 20-30 minutes... this is a good short-to-medium term solution. However, on Mac you need to be on OS X Sierra (or use the brew install instead)

I think someone needs to re-architect the fast sync in geth as the client needs to reach out to more diverse peers when it gets "stuck" for long periods of time like this. I have a few ideas, but very limited time, and it really needs to be done (or reviewed) by someone who knows what they are doing :P

tomtom87 commented 6 years ago

Thanks yeh we run parity but it wont sync just the same problem so what we do is run the docker image and everytime it fucks up 'turn it off an on again' Say a little prayer an one out of ten it will work. This is only way we have found to sync to run our business we have one employee her job just come in an run sync every day at 8am before we start then can teamviewer that machine. Ethereum is a labour of love right now.. dunno how that impression affects the new comers out there. Probably aint helping adoption being so un-user friendly.

On 25 Sep 2017, at 07:34, Nigel Sheridan-Smith notifications@github.com wrote:

You'll probably find it much easier to be on Parity (https://parity.io) - the wallet can do a light-mode sync in around 20-30 minutes... this is a good short-to-medium term solution. However, on Mac you need to be on OS X Sierra (or use the brew install instead)

I think someone needs to re-architect the fast sync in geth as the client needs to reach out to more diverse peers when it gets "stuck" for long periods of time like this. I have a few ideas, but very limited time, and it really needs to be done (or reviewed) by someone who knows what they are doing :P

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

wtfiwtz commented 6 years ago

Try restarting it a few times... you will re-connect with more diverse peers.

You might also be able to modify the config to increase MaxPeers, though I haven't figured out exactly how yet:

var DefaultConfig = Config{
    DataDir:     DefaultDataDir(),
    HTTPPort:    DefaultHTTPPort,
    HTTPModules: []string{"net", "web3"},
    WSPort:      DefaultWSPort,
    WSModules:   []string{"net", "web3"},
    P2P: p2p.Config{
        ListenAddr:      ":30303",
        DiscoveryV5Addr: ":30304",
        MaxPeers:        25,
        NAT:             nat.Any(),
    },
}
wtfiwtz commented 6 years ago

Looks like the configuration file is still coming, maybe later this week: https://github.com/ethereum/go-ethereum/issues/2067

wtfiwtz commented 6 years ago

@tomtom87 if you leave it running 24/7 it should keep sync ok... it is only if you disconnect for long periods of time (or the peers you have are not up-to-date either) that it will fall behind. Generally the client will get announcements of the latest blocks but it will automatically discard them if you are more than 32 blocks behind. Again, this is a limitation of the current implementation - it could theoretically cache them in advance of processing.

tomtom87 commented 6 years ago

Oh ok is a little impracticle for all officers to run it every day constantly, as when we go home maybe cleaner tries enter computer or is turn off from sleepmode.

A cache would be great. Well if any debugging or testing needed im here.

thank you

On 25 Sep 2017, at 15:17, Nigel Sheridan-Smith notifications@github.com wrote:

@tomtom87 if you leave it running 24/7 it should keep sync ok... it is only if you disconnect for long periods of time (or the peers you have are not up-to-date either) that it will fall behind. Generally the client will get announcements of the latest blocks but it will automatically discard them if you are more than 32 blocks behind. Again, this is a limitation of the current implementation - it could theoretically cache them in advance of processing.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

MartinDace commented 6 years ago

So, is our Ether lost forever, or is there a way to get it back using another application?

schroedingerscode commented 6 years ago

@MartinDace The state of your ETH accounts are kept on the blockchain and kept current irrespective of which client you use. The only thing that you need is the private key associated with your account. You can view your balance at etherscan.io if you'd like to ease your fears.

MartinDace commented 6 years ago

But is it possible move ether from there?

abl1234 commented 6 years ago

Hi

I am having the same issue with the blockchain syncing . but now I cannot get a eth.syncing but I do get a eth.blockNumber and peerCount reply now -- is this normal and am I synched yet since I do not want to stop Geth now since I have a blockNumber reply for first time so far in this week long process:) / $ geth attach Welcome to the Geth JavaScript console!

instance: Geth/v1.7.0-stable-6c6c7b2a/linux-amd64/go1.9 coinbase: 0x9ca8c0cf47576208c5b017d9a560ed33d84f2267 at block: 4319007 (Thu, 28 Sep 2017 13:21:59 IST) datadir: /home/qb78846/.ethereum modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

eth.syncing false net.peerCount 9 eth.blockNumber 4319073

[09-28|13:55:43] Imported new chain segment blocks=1 txs=125 mgas=6.531 elapsed=155.054ms mgasps=42.121 number=4319064 hash=d1e5f5…469d33 INFO [09-28|13:56:56] Imported new chain segment blocks=1 txs=124 mgas=6.559 elapsed=225.838ms mgasps=29.042 number=4319065 hash=396dcb…767c1e INFO [09-28|13:57:37] Imported new chain segment blocks=1 txs=172 mgas=6.683 elapsed=280.647ms mgasps=23.812 number=4319066 hash=036ac3…18debf INFO [09-28|13:57:44] Imported new chain segment blocks=1 txs=101 mgas=6.616 elapsed=220.905ms mgasps=29.949 number=4319067 hash=ea8c76…884246 INFO [09-28|13:58:01] Imported new chain segment blocks=1 txs=95 mgas=6.687 elapsed=236.463ms mgasps=28.278 number=4319068 hash=5315a5…42028a INFO [09-28|13:58:02] Imported new chain segment blocks=1 txs=136 mgas=6.698 elapsed=251.738ms mgasps=26.607 number=4319067 hash=e650a5…0e36cd INFO [09-28|13:58:23] Imported new chain segment blocks=1 txs=204 mgas=6.544 elapsed=323.078ms mgasps=20.256 number=4319069 hash=56271a…cbf0b2 INFO [09-28|13:58:24] Imported new chain segment blocks=1 txs=76 mgas=6.600 elapsed=350.838ms mgasps=18.811 number=4319070 hash=526d33…30fb23 INFO [09-28|13:58:46] Imported new chain segment blocks=1 txs=33 mgas=6.569 elapsed=713.876ms mgasps=9.202 number=4319071 hash=a51bf0…172919 INFO [09-28|13:59:04] Imported new chain segment blocks=1 txs=83 mgas=6.661 elapsed=310.312ms mgasps=21.464 number=4319072 hash=09ff09…30f2bd INFO [09-28|13:59:16] Imported new chain segment blocks=1 txs=109 mgas=2.409 elapsed=177.983ms mgasps=13.538 number=4319073 hash=6b7e6a…01553f INFO [09-28|14:00:13] Imported new chain segment blocks=1 txs=104 mgas=6.675 elapsed=180.008ms mgasps=37.082 number=4319074 hash=391698…bbda07 INFO [09-28|14:01:22] Imported new chain segment blocks=1 txs=187 mgas=6.693 elapsed=1.596s mgasps=4.192 number=4319075 hash=3b2aa4…bc3e8b

abl1234 commented 6 years ago

I think I am good since I read in another post that the importing chain segment happens approx every 14 seconds and if i check the last block on etherscan.io it is similiar - can anybody confirm this is expceted behaviour ? thnaks in advance

INFO [09-28|14:14:35] Imported new chain segment blocks=1 txs=132 mgas=6.722 elapsed=529.342ms mgasps=12.699 number=4319106 hash=15e59b…55f217 INFO [09-28|14:14:55] Imported new chain segment blocks=1 txs=161 mgas=6.699 elapsed=346.993ms mgasps=19.306 number=4319107 hash=716ce9…9e0a26

wtfiwtz commented 6 years ago

@MartinDace you can add your wallet details to any wallet application you like (by importing it) - as long as you have retained the wallet JSON file and password, or the private key. You can then add new transactions to the Ethereum blockchain from that wallet. The blockchain is a public register of all transactions so if you re-download it (by "syncing") then your wallet will update the the current transaction value that has been agreed by all of the Ethereum peers.

You just need to be patient with geth and Ethereum Mist as the syncing is very slow, even in fast mode. You might be able to try light mode instead.

wtfiwtz commented 6 years ago

@abl1234 yes you are close, the chain segments you will need are 4,320,572 at the present moment (I think), so about 1400 to go!

wtfiwtz commented 6 years ago

Also https://github.com/ethereum/mist/issues/3097 🎉

holiman commented 6 years ago

I've encountered this problem twice on 1.7:ish.

First occasion, it flipped to from fast to full, but somehow failed to commit the flip, and then kept on fast-synching past pivot point, IIUC.

Second occasion, stuck on importing state entries, no headers/blocks imported any longer. Restarting geth seems to make the block/header import resume.

tomtom87 commented 6 years ago

turn it off and on again

Works every time

On 30 Sep 2017, at 15:20, Martin Holst Swende notifications@github.com wrote:

I've encountered this problem twice on 1.7:ish. First occasion, it flipped to from fast to full, but somehow failed to commit the flip, and then kept on fast-synching past pivot point, IIUC. Second occasion, stuck on importing state entries, no headers/blocks imported any longer.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

MartinDace commented 6 years ago

@tomtom87 I've tried that a few times without success. Since I bought ether direct from the official ethereum web site and am using the default wallet they offer, does this not imply that the team behind ethereum are not on top of their game? And if so, why should we buy ether at all?

wtfiwtz commented 6 years ago

@MartinDace the issue is the peer-to-peer network is not 100% reliable and that is not the fault of the Ethereum team. If someone has the time and inclination, it would be possible to make the syncing process more efficient by enhancing the geth peer-to-peer client (as described above).

Alternatively, try the geth light mode in 0.7.1 or grab the http://parity.io wallet which is a separate implementation in the language Rust and you might have more luck with that (often syncs within a few hours or less).

abl1234 commented 6 years ago

Hi Nigel Thanks for the help - I still seem to be still stuck at the importing new chain segments - this copy was taken when the latest block on etherscan.io is the same as the last imported peers

NFO [10-02|11:35:50] Imported new chain segment blocks=6 txs=616 mgas=36.242 elapsed=9.698s mgasps=3.737 number=4328631 hash=56d1b3…86cc70 INFO [10-02|11:35:59] Imported new chain segment blocks=5 txs=512 mgas=29.294 elapsed=8.793s mgasps=3.331 number=4328636 hash=0f1e2b…af67bc INFO [10-02|11:36:08] Imported new chain segment blocks=6 txs=791 mgas=32.215 elapsed=8.798s mgasps=3.661 number=4328642 hash=33c3e4…006d36 INFO [10-02|11:36:17] Imported new chain segment blocks=5 txs=553 mgas=32.785 elapsed=8.691s mgasps=3.772 number=4328647 hash=abc579…7379fe INFO [10-02|11:36:25] Imported new chain segment blocks=6 txs=507 mgas=34.194 elapsed=8.807s mgasps=3.882 number=4328653 hash=b1d45a…16485b INFO [10-02|11:36:35] Imported new chain segment blocks=6 txs=703 mgas=27.595 elapsed=9.226s mgasps=2.991 number=4328659 hash=188adc…88f4bb

Last Block 4330217 (29.91s Avg) Transactions 63451085 Hash Rate Network Difficulty 98,899.45 GH/s 2,863.92 TH 14 day Ethereum Transaction History9/189/199/209/219/229/239/249/259/269/279/289/299/3010/1250k300k350k400k

abl1234 commented 6 years ago

I passed that number on the imported new chain segment

Last Block 4330217 (29.91s Avg) Transactions 63451085 Hash Rate Network Difficulty 98,899.45 GH/s 2,863.92 TH 14 day Ethereum Transaction History9/189/199/209/219/229/239/249/259/269/279/289/299/3010/1250k300k350k400k

abl1234 commented 6 years ago

INFO [10-02|11:45:41] Imported new chain segment blocks=6 txs=677 mgas=37.087 elapsed=8.639s mgasps=4.293 number=4329017 hash=b3c0eb…f352f1 INFO [10-02|11:45:49] Imported new chain segment blocks=10 txs=703 mgas=44.328 elapsed=8.426s mgasps=5.260 number=4329027 hash=77464a…04834e INFO [10-02|11:45:58] Imported new chain segment blocks=11 txs=904 mgas=64.091 elapsed=8.141s mgasps=7.872 number=4329038 hash=fb3b56…4f4768 INFO [10-02|11:46:08] Imported new chain segment blocks=4 txs=444 mgas=22.466 elapsed=10.129s mgasps=2.218 number=4329042 hash=95ed8e…7494ee INFO [10-02|11:46:17] Imported new chain segment blocks=3 txs=323 mgas=14.226 elapsed=9.256s mgasps=1.537 number=4329045 hash=082483…ac5e9a INFO [10-02|11:46:27] Imported new chain segment blocks=4 txs=433 mgas=20.678 elapsed=9.810s mgasps=2.108 number=4329049 hash=ec335e…7db925 INFO [10-02|11:46:36] Imported new chain segment blocks=3 txs=158 mgas=13.802 elapsed=9.064s mgasps=1.523 number=4329052 hash=2189dc…c52d5d INFO [10-02|11:46:44] Imported new chain segment blocks=5 txs=486 mgas=33.253 elapsed=8.532s mgasps=3.897 number=4329057 hash=fa1a66…37dc10 INFO [10-02|11:46:53] Imported new chain segment blocks=9 txs=613 mgas=40.955 elapsed=8.782s mgasps=4.663 number=4329066 hash=df4112…e2e078 INFO [10-02|11:47:02] Imported new chain segment blocks=6 txs=471 mgas=32.958 elapsed=8.375s mgasps=3.935 number=4329072 hash=3b9bb3…2e4d90 INFO [10-02|11:47:12] Imported new chain segment blocks=4 txs=281 mgas=21.135 elapsed=10.903s mgasps=1.938 number=4329076 hash=b765a8…d3eae7 INFO [10-02|11:47:24] Imported new chain segment blocks=10 txs=836 mgas=45.666 elapsed=11.480s mgasps=3.978 number=4329086 hash=4e4727…ce0d43 INFO [10-02|11:47:34] Imported new chain segment blocks=18 txs=1420 mgas=92.512 elapsed=9.831s mgasps=9.410 number=4329104 hash=63cb04…e15978 INFO [10-02|11:47:42] Imported new chain segment blocks=5 txs=586 mgas=28.044 elapsed=8.113s mgasps=3.456 number=4329109 hash=5ba422…d2e834 INFO [10-02|11:47:51] Imported new chain segment blocks=4 txs=578 mgas=20.966 elapsed=8.636s mgasps=2.428 number=4329113 hash=03f4c8…29fedd INFO [10-02|11:47:59] Imported new chain segment blocks=3 txs=537 mgas=14.794 elapsed=8.123s mgasps=1.821 number=4329116 hash=a41030…1c6dc9 INFO [10-02|11:48:07] Imported new chain segment blocks=4 txs=450 mgas=26.650 elapsed=8.236s mgasps=3.236 number=4329120 hash=4aecfb…cf6698 INFO [10-02|11:48:15] Imported new chain segment blocks=5 txs=552 mgas=24.531 elapsed=8.535s mgasps=2.874 number=4329125 hash=5c29a2…88954b INFO [10-02|11:48:24] Imported new chain segment blocks=8 txs=588 mgas=18.906 elapsed=8.419s mgasps=2.246 number=4329133 hash=4938a0…1a9f6f INFO [10-02|11:48:32] Imported new chain segment blocks=5 txs=396 mgas=22.023 elapsed=8.492s mgasps=2.593 number=4329138 hash=e38b10…2d1e93 INFO [10-02|11:48:41] Imported new chain segment blocks=8 txs=856 mgas=47.102 elapsed=8.771s mgasps=5.370 number=4329146 hash=9a7bef…a8f552 INFO [10-02|11:48:51] Imported new chain segment blocks=7 txs=670 mgas=32.824 elapsed=9.705s mgasps=3.382 number=4329153 hash=4c40b0…e92b24

abl1234 commented 6 years ago

so I am upto date with the blocks but not getting the sycnhronization completed

MartinDace commented 6 years ago

@wtfiwtz Hello Nigel, I downloaded the wallet at http://parity.io however I've idea how to proceed from there. I have a wallet address for the Mist wallet but I have been sent no private key or list of words so I do not see how I can access the ether than I have purchased. My balance is still registered at zero in the wallet app although I have verified the existence of my balance by using the wallet address at etherscan.io

MartinDace commented 6 years ago

In other words it looks to me like ethereum is a scam, since they invite you to buy ether and provide a default wallet from which it is impossible to extract the ether. "...it would be possible to make the syncing process more efficient by..." - If so, why don't they do it? Seems pretty basic to me.

holiman commented 6 years ago

Ethereum.org does not "invite you to buy" ether. This ticket is about synching the ethereum blockchain. It's fully possible to 'extract ether' without synching the blockchain, all that is needed is to construct a transaction. That can be done in multiple ways, e.g. via the geth console. 

var tx = { to : <receiver> , from:"0x...account...", value: web3.toWei( number_of_ether, "ether")}
var rlpdata = eth.signTransaction(tx)

And the actual transaction can be broadcast using some other node, e.g myetherwallet. There's no need to hijack this particular ticket @MartinDace . There is a lot of information out there if you're willing to dig a bit, otherwise please use the Ethereum gitter rooms to ask for help.

As for the ticket, yes, there's some glitch in switching over from fast-sync to full sync.

MartinDace commented 6 years ago

@holiman It may not be an intentional scam, and perhaps scam is the wrong word, but the web site allows you to download the ethereum wallet and allows you to purchase ether without any warning that what you purchase will never be available to use (as far as I can tell). Given the length of this thread (that I was redirected to from a thread that began with someone else making exactly that complaint) I can hardly be accused of hijacking it. It is fine to explain after the event that I should have done it some other way (incidentally requiring coding skill which I do not have) but no-one so far has provided an answer to the original question, which is, how do we get our ether, once having started this process? If there is a way, please explain.

holiman commented 6 years ago

allows you to purchase ether

What? There is no way of buying 'ether direct from the official ethereum web site ', unless you mean the pre-sale a couple of years ago.

Anyway, the simplest way to create a transaction without any coding skills or command line experience, is probably to use myetherwallet and import the wallet-file there. A note of warning, though: there are phishing-sites which masquerade as myetherwallet, and exposing a wallet to a website is dangerous without proper precautions.

The recommended way is to download myetherwallet and use in an offline computer, create a transaction, take the 'rlp'-data over to an online computer and broadcast the transaction (e.g. using myetherwallet again).

Also : "I have been sent no private key" - and never will you be sent a private key. if you have generated a mist wallet, and sent funds to it, you need to ensure the security of the wallet, which resides in a keystore folder (the location of which is platform-dependant).

Ultimately, remember that you are responsible for your ether - nobody else. All information about interacting with wallet(s) is available in various places - the ethereum wiki, or parity documentation if you're using parity, or our gitter channels.

MartinDace commented 6 years ago

@holiman Thank you for your explanation. I understand the ether is not from the ethereum web site itself. However when buying Bitcoin using (for example) bread all the information is there - the public key, the private key, the rescue words. I have used it without problems including purchasing my so far missing ether. Mist, however, is entirely obscure in its functioning, as well as appearing to be in an endless loop since 11 September. Regrettably my attempt to use myetherwallet stalled at the end and I see no way to use that either.

abl1234 commented 6 years ago

Hi

Getting back to the original thread about not syncing - I READ earlier that to get it to sync once needs to RUN Geth FULL sync from the beginning without any stops and it will work - The issue being that from a FULL to FAST sync is causing the problem - I do not want to run this process again since has taken a week to get his far so is there a new version of GETH coming out soon or a patch or should i try a development build 1.71? Thanks in advance Using this build instance: Geth/v1.7.0-stable-6c6c7b2a/linux-amd64/go1.9 coinbase: 0x9ca8c0cf47576208c5b017d9a560ed33d84f2267 at block: 4333102 (Tue, 03 Oct 2017 12:15:38 IST) datadir: /home/qb78846/.ethereum modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

abl1234 commented 6 years ago

???

tomtom87 commented 6 years ago

Ethereum is a free research software its not finished, its a work in progress. It's evolving and has bugs

I can rescue your ether, log in to myetherwallet with your mist exported json file and it will be there, then send it to exodus wallet and you are good to go.

Mist wallet is only for developers of smart contracts and tokens for retail users try exodus that is equivilent of bread. This needs to be clearly stated to customers really because mist wallet is not for newbs.

If you still got problems with gettin you ETH contact me i will help you recover it.

On 3 Oct 2017, at 04:34, MartinDace notifications@github.com wrote:

@holiman Thank you for your explanation. I understand the ether is not from the ethereum web site itself. However when buying Bitcoin using (for example) bread all the information is there - the public key, the private key, the rescue words. I have used it without problems including purchasing my so far missing ether. Mist, however, is entirely obscure in its functioning, as well as appearing to be in an endless loop since 11 September. Regrettably my attempt to use myetherwallet stalled at the end and I see no way to use that either.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.