Ethereum Wallet 65 blocks left (for 2 days)

[ZeeFF]

• [ ] I've asked for help in the Mist Gitter before filing this issue.

Version: `0.9.3`
OS & Version: windows
Node version: `geth 1.8.2` 
Number of blocks synchronized: 5,279,400 of 5,279,535
Always 65 blocks left or more and few peers

[ZeeFF]

this wallet is never gonna get sync ... is it? Downloading block 5,281,127 of 5,281,190 edit: it has been on for 3 or more days now....this is stupid... I understand 10 hours .. 1 day to sync.. but wtf???

[philsmd]

see https://github.com/ethereum/mist/issues/3738 and https://github.com/ethereum/mist/issues/3738#issuecomment-374160082 and https://github.com/ethereum/go-ethereum/issues/16251#issuecomment-371449572

[Hack06]

Couldn't sync the blockchain all the week-end. Today there were remaining 65 blocks, and now it fails to connect with an error dialog (screenshots attached).

[philsmd]

I think this usbwallet problem is completely unrelated to the sync problem (it's probably about a hardware wallet?). Are you sure you are using a fast SSD (solid state disk) and that your internet connection / CPU is fast enough? It could indeed take several days even for a fast CPU/SSD and fast internet. I think the number of state tries (chain structure) should be almost 70M or more (at the time of this writing), but only newer versions of geth increment the number shown to the user between restarts of geth/mist (see https://github.com/ethereum/mist/pull/3729#issuecomment-370583223). Older versions will still continue at the point they stopped (if mist is restarted), therefore no progress is lost, but the number shown (knownStates) will start from zero, while the newer version will continue counting up.

[Hack06]

I don't have an SSD on my laptop, and the Internet is 4G/LTE. Other parameters are in the attached screenshot.

[Hack06]

And yes, forgot to mention that I don't use any USB-wallet (yet :D).

[philsmd]

it's not possible to run a full node with a HDD (you need to have at least a fast SSD). If you only have a HDD (very slow compared to a SSD) you only can use light sync mode. This again is not a problem of mist, but a requirement of the full node (of the protocol). Again, see: https://github.com/ethereum/go-ethereum/issues/16251#issuecomment-371449572 :

Q: Wait, so I can't run a full node on an HDD?

A: Unfortunately not. Doing a fast sync on an HDD will take more time than you're willing to wait with the current data schema. Even if you do wait it out, an HDD will not be able to keep up with the read/write requirements of transaction processing on mainnet.

You however should be able to run a light client on an HDD with minimal impact on system resources. If you wish to run a full node however, an SSD is your only option.

[Hack06]

Are you serious? I've synced once and created an account, then put some change on it for testing.

[philsmd]

How do you know that it was fully synced? It's actually quite difficult to determine that. I think the eth.syncing + eth.blockNumber method is the only way to get an idea if your node has really downloaded and verified all state tries with the --fast sync mode: see https://ethereum.stackexchange.com/questions/6525/in-geth-eth-syncing-gives-false-even-when-blocks-are-imported/6527#6527

The problem is that many users think they have finished syncing just because all blocks were downloaded in the --fast sync mode, that's absolutely not true!

I don't think that a light node is a bad option (especially if you use a HDD it is your only acceptable option for the mainnet/testnet).

[Hack06]

I've avoided using beta/testing stuff so thought it was synced since the block counter on top of the wallet UI was increasing every 15 seconds with its little loading-like spinner. But after reading your references now I doubt about having a full-sync. I'll try a bit later and comment here if necessary.

[mrlukaszfuszara]

log.txt

The same problem. I attach the log.

[Allen-Liang]

I have the same problem. I am in China. Ethereum Wallet 65 blocks left (for N days)

[Xealott]

@mrlukaszfuszara @ZeeFF @Allen-Liang If you're like me, you were trying to run a full node on an HDD... which apparently can't be done (see: ethereum/go-ethereum#16251 (comment) ) . I've had the same frustration. Right now I'm just trying to figure out how to get this info to new adopters, to save everyone the frustration of watching their machine work for days with (apparently) only 65 blocks out of 5+ million left to go!

[evertonfraga]

Replying @Hack06 https://github.com/ethereum/mist/issues/3760#issuecomment-374234922

From your image I can see the following: the block count is at latest - 65; Chain structure (state entries) is at 11M;

Today's Go-ethereum default sync (called fast sync) works by decoupling some information and downloading them in some tracks: block headers and state entries. Although it seems that you've downloaded most of the blocks, you're far from the current total state entries (over 70M at the moment).

We've got a solution for you all, and recommend using the Light Client option. It's still experimental, as (most things in the crypto space), but it does the job by downloading information from other servers on-demand, making the syncing problem a thing of the past.

Please update to 0.10.0 and give it a try (Develop > Sync with Light client).

I am closing this issue, please reopen if necessary.

[Hack06]

Ok, that version 0.10.0 seems to work better, although had to restart and re-unzip couple of times to get in. Just another question, in my screenshot above with error logs, I don't get what's /home/travis/... folder pointing to, I don't have any user or folder named 'travis' on my linux, can you shed some light on this please?

[evertonfraga]

Hi @Hack06, that's quite intriguing indeed. That error message comes from go-ethereum (geth) runtime error. It is compiled in a travis runner, I'm afraid it has to do with some static linking in GO compile process.

Amirite @karalabe?

[Seetth]

Atomic clock sync, slow and I mean slow as hell sync and I was complaining about BTC... Yes I do have ssd I dont use standard hard drives ages ago, yes my computer is faster as hell "threadriper 64 GB ram blah blah blah etc" yes my internet is fast enough 200mb/s optic fiber, but still this thing is annoyingly slow....

Why this client is so annoying?

Im not a big ETH fan but as far I understand mist is the official wallet, as somebody said already I dont like beta stuff specially when is related to money... But in this situation im forced to use light client...

Oh yes geth changes nothing still is annoyingly slow...

[michaelgracie]

This issue has been replicated with both an external HDD and internal SSD storage. In either circumstance the block count sits 65 behind, with chain structure downloading in perpetuity. Watched for roughly an hour, and the blocks do update, simply reporting 65 behind no matter the storage device. ~20 peers either way.

Mist (Ethereum Wallet) version 0.10.0 macOS 10.13.4, MBP 2.5 GHz Intel Core i7, 16GB RAM External storage: 1TB 7200rpm HDD, USB 3 Internal storage: 0.5TB stock SSD Consistent 250 Mbps connection

Additionally tested simple wallet contract. Goes into creation mode, then after some time - presumably after another 65 blocks - comes up for confirmation. Cancelled thereafter.

[SpinnerRow]

Running Samsung SSD with an i7 CPU and 150mbit up/down internet. It isn't my conneciton or the hardware. I'm stuck on block 65 left and it won't go further.

How can we get this issue escalated!?!?!?!

[karalabe]

Syncing Ethereum is a pain point for many people, so I'll try to detail what's happening behind the scenes so there might be a bit less confusion.

The current default mode of sync for Geth is called fast sync. Instead of starting from the genesis block and reprocessing all the transactions that ever occurred (which could take weeks), fast sync downloads the blocks, and only verifies the associated proof-of-works. Downloading all the blocks is a straightforward and fast procedure and will relatively quickly reassemble the entire chain.

Many people falsely assume that because they have the blocks, they are in sync. Unfortunately this is not the case, since no transaction was executed, so we do not have any account state available (ie. balances, nonces, smart contract code and data). These need to be downloaded separately and cross checked with the latest blocks. This phase is called the state trie download and it actually runs concurrently with the block downloads; alas it take a lot longer nowadays than downloading the blocks.

So, what's the state trie? In the Ethereum mainnet, there are a ton of accounts already, which track the balance, nonce, etc of each user/contract. The accounts themselves are however insufficient to run a node, they need to be cryptographically linked to each block so that nodes can actually verify that the account's are not tampered with. This cryptographic linking is done by creating a tree data structure above the accounts, each level aggregating the layer below it into an ever smaller layer, until you reach the single root. This gigantic data structure containing all the accounts and the intermediate cryptographic proofs is called the state trie.

Ok, so why does this pose a problem? This trie data structure is an intricate interlink of hundreds of millions of tiny cryptographic proofs (trie nodes). To truly have a synchronized node, you need to download all the account data, as well as all the tiny cryptographic proofs to verify that noone in the network is trying to cheat you. This itself is already a crazy number of data items. The part where it gets even messier is that this data is constantly morphing: at every block (15s), about 1000 nodes are deleted from this trie and about 2000 new ones are added. This means your node needs to synchronize a dataset that is changing 200 times per second. The worst part is that while you are synchronizing, the network is moving forward, and state that you begun to download might disappear while you're downloading, so your node needs to constantly follow the network while trying to gather all the recent data. But until you actually do gather all the data, your local node is not usable since it cannot cryptographically prove anything about any accounts.

If you see that you are 64 blocks behind mainnet, you aren't yet synchronized, not even close. You are just done with the block download phase and still running the state downloads. You can see this yourself via the seemingly endless Imported state entries [...] stream of logs. You'll need to wait that out too before your node comes truly online.

---

Q: The node just hangs on importing state enties?!

A: The node doesn't hang, it just doesn't know how large the state trie is in advance so it keeps on going and going and going until it discovers and downloads the entire thing.

The reason is that a block in Ethereum only contains the state root, a single hash of the root node. When the node begins synchronizing, it knows about exactly 1 node and tries to download it. That node, can refer up to 16 new nodes, so in the next step, we'll know about 16 new nodes and try to download those. As we go along the download, most of the nodes will reference new ones that we didn't know about until then. This is why you might be tempted to think it's stuck on the same numbers. It is not, rather it's discovering and downloading the trie as it goes along.

Q: I'm stuck at 64 blocks behind mainnet?!

A: As explained above, you are not stuck, just finished with the block download phase, waiting for the state download phase to complete too. This latter phase nowadays take a lot longer than just getting the blocks.

Q: Why does downloading the state take so long, I have good bandwidth?

A: State sync is mostly limited by disk IO, not bandwidth.

The state trie in Ethereum contains hundreds of millions of nodes, most of which take the form of a single hash referencing up to 16 other hashes. This is a horrible way to store data on a disk, because there's almost no structure in it, just random numbers referencing even more random numbers. This makes any underlying database weep, as it cannot optimize storing and looking up the data in any meaningful way.

Not only is storing the data very suboptimal, but due to the 200 modification / second and pruning of past data, we cannot even download it is a properly pre-processed way to make it import faster without the underlying database shuffling it around too much. The end result is that even a fast sync nowadays incurs a huge disk IO cost, which is too much for a mechanical hard drive.

Q: Wait, so I can't run a full node on an HDD?

A: Unfortunately not. Doing a fast sync on an HDD will take more time than you're willing to wait with the current data schema. Even if you do wait it out, an HDD will not be able to keep up with the read/write requirements of transaction processing on mainnet.

You however should be able to run a light client on an HDD with minimal impact on system resources. If you wish to run a full node however, an SSD is your only option.