Issues related with Syncing

[evertonfraga]

Updated in 2018-01-08

Notes about sync

"Stuck" on 100-200 blocks away from current highest block

When using the default Geth setting fast mode, it downloads blocks, headers and chain structure (or state entries). As of today, the latter sums up more than 70 million entries. Know what to expect: it can take several hours to download on a consumer machine. For quicker access to funds or balances, refer to the Light Client section.

"Mist has eaten my hard drive space"

Regular geth sync can take dozens of GB from your hard drive. In order to reclaim space, you can:

1. execute geth removedb
2. enable the light client (see below)

For information directly related to blockchain sync, refer to https://github.com/ethereum/go-ethereum

Light Client

Soon-to-be default, yet still experimental. It downloads block headers and pulls blocks on demand. The sync times are reduced by an order of magnitude while staying safe. And the required storage is a lot smaller.

How to activate?

Mist Wiki: https://github.com/ethereum/mist/wiki Gitter channel, where the community is, and can certainly help as well: https://gitter.im/ethereum/mist

[quantumproducer]

Perpetually ~60 blocks away from syncing. Reading elsewhere that I cannot query the geth node until it is synced, making this unusable.

--syncmode=light seems to just not sync at all. Same results as @adjiandov

[adjiandov]

@quantumproducer @slugmandrew What I did at the end to be able to sync is I manually started geth (via terminal). I downloaded the latest a few days ago which was 1.8.7-stable. Started it using the following command: geth --testnet --light optionally I set the directory where to start and this worked, for some reason networkId 3 and testnet behave differently, but they sync to the same network. After this start syncing then I started also from terminal my wallet pointing to the url=~/username/Library/Ethereum/mydirectory/geth.ipc and this works for now :) Hope this helps

[quantumproducer]

@adjiandov were you using testnet then, with network id 3?

[quantumproducer]

From @karalabe , in https://github.com/ethereum/go-ethereum/issues/16743#issuecomment-389445550

begin quote: Syncing Ethereum is a pain point for many people, so I'll try to detail what's happening behind the scenes so there might be a bit less confusion.

The current default mode of sync for Geth is called fast sync. Instead of starting from the genesis block and reprocessing all the transactions that ever occurred (which could take weeks), fast sync downloads the blocks, and only verifies the associated proof-of-works. Downloading all the blocks is a straightforward and fast procedure and will relatively quickly reassemble the entire chain.

Many people falsely assume that because they have the blocks, they are in sync. Unfortunately this is not the case, since no transaction was executed, so we do not have any account state available (ie. balances, nonces, smart contract code and data). These need to be downloaded separately and cross checked with the latest blocks. This phase is called the state trie download and it actually runs concurrently with the block downloads; alas it take a lot longer nowadays than downloading the blocks.

So, what's the state trie? In the Ethereum mainnet, there are a ton of accounts already, which track the balance, nonce, etc of each user/contract. The accounts themselves are however insufficient to run a node, they need to be cryptographically linked to each block so that nodes can actually verify that the account's are not tampered with. This cryptographic linking is done by creating a tree data structure above the accounts, each level aggregating the layer below it into an ever smaller layer, until you reach the single root. This gigantic data structure containing all the accounts and the intermediate cryptographic proofs is called the state trie.

Ok, so why does this pose a problem? This trie data structure is an intricate interlink of hundreds of millions of tiny cryptographic proofs (trie nodes). To truly have a synchronized node, you need to download all the account data, as well as all the tiny cryptographic proofs to verify that noone in the network is trying to cheat you. This itself is already a crazy number of data items. The part where it gets even messier is that this data is constantly morphing: at every block (15s), about 1000 nodes are deleted from this trie and about 2000 new ones are added. This means your node needs to synchronize a dataset that is changing 200 times per second. The worst part is that while you are synchronizing, the network is moving forward, and state that you begun to download might disappear while you're downloading, so your node needs to constantly follow the network while trying to gather all the recent data. But until you actually do gather all the data, your local node is not usable since it cannot cryptographically prove anything about any accounts.

If you see that you are 64 blocks behind mainnet, you aren't yet synchronized, not even close. You are just done with the block download phase and still running the state downloads. You can see this yourself via the seemingly endless Imported state entries [...] stream of logs. You'll need to wait that out too before your node comes truly online.

Q: The node just hangs on importing state enties?!

A: The node doesn't hang, it just doesn't know how large the state trie is in advance so it keeps on going and going and going until it discovers and downloads the entire thing.

The reason is that a block in Ethereum only contains the state root, a single hash of the root node. When the node begins synchronizing, it knows about exactly 1 node and tries to download it. That node, can refer up to 16 new nodes, so in the next step, we'll know about 16 new nodes and try to download those. As we go along the download, most of the nodes will reference new ones that we didn't know about until then. This is why you might be tempted to think it's stuck on the same numbers. It is not, rather it's discovering and downloading the trie as it goes along.

Q: I'm stuck at 64 blocks behind mainnet?!

A: As explained above, you are not stuck, just finished with the block download phase, waiting for the state download phase to complete too. This latter phase nowadays take a lot longer than just getting the blocks.

Q: Why does downloading the state take so long, I have good bandwidth?

A: State sync is mostly limited by disk IO, not bandwidth.

The state trie in Ethereum contains hundreds of millions of nodes, most of which take the form of a single hash referencing up to 16 other hashes. This is a horrible way to store data on a disk, because there's almost no structure in it, just random numbers referencing even more random numbers. This makes any underlying database weep, as it cannot optimize storing and looking up the data in any meaningful way.

Not only is storing the data very suboptimal, but due to the 200 modification / second and pruning of past data, we cannot even download it is a properly pre-processed way to make it import faster without the underlying database shuffling it around too much. The end result is that even a fast sync nowadays incurs a huge disk IO cost, which is too much for a mechanical hard drive.

Q: Wait, so I can't run a full node on an HDD?

A: Unfortunately not. Doing a fast sync on an HDD will take more time than you're willing to wait with the current data schema. Even if you do wait it out, an HDD will not be able to keep up with the read/write requirements of transaction processing on mainnet.

You however should be able to run a light client on an HDD with minimal impact on system resources. If you wish to run a full node however, an SSD is your only option.

[adjiandov]

@quantumproducer I ended up using a command argument called --testnet, without specifying networkId, which at the end sync to the Ropsten network. So just try the following: geth --light --testnet and then point your wallet up to the opened geth.ipc from this node.

[quantumproducer]

Ah, I am trying to sync mainnet.

[devedse]

@quantumproducer , any idea what the upper limit is for the chain structure? (You say 100's of millions, but do you have any specific number?)

[whizzard80]

Thanks to everyone on the thread for all the help. I had quite a bit of trouble getting the "vanilla" eth client to sync fully... but after a lot of struggle I got it working well. I think I have a strategy to try if all else seems to fail. This is my first GitHub post, and take everything I say with a grain of salt. But this program gave me a lot of issues and I would love to keep others from a bad experience if I can. Using this method I can now fully sync in about 4 or 5 hours total instead of not getting it to sync at all for days. As with any advice try at your own risk and

Make sure you ALWAYS BACKUP your keystore accounts if you hold ETH in any of them or need the old accounts for any reason. If you don't know how to do this yet, it is in the menu of the GUI. Back them all up now.

I am going to make this little guide thorough so hopefully... anyone who finds this might finally get a sync. Even if they don't have much crypto experience. Here we go...

Syncing using vanilla GETH command line console (win)

OS: Windows 10 64bit Hard Disk: 800GB SSD Network Connection: 200mbps Installation Package Used: Ethereum-Wallet-installer-0-10-0.exe Date: 6/19/2018

**The method that finally worked (TLDR):

• Clean uninstall
• port forward 30303
• use the console for fast sync
• WAIT
• light sync
• full sync
• WAIT
• fast sync in default directories while monitoring with the GUI.**

Step by step guide to this strategy:

First:

Download the "vanilla" client from ethereum.org

Log into your router and open port 30303 for port forwarding. If you do not know how to do this please do a search and find out how to set up your router for port forwarding. It is different for every model and way too much to go into in this already long-winded post. Try a google search or this link: https://portforward.com/

I think the sync may work a bit better using the defaults wherever possible. Unless you have a high-speed drive other than your main system drive... keep your blockchain data on your main system. If you don't have enough space on your OS disk you need to get another SSD for the blockchain. If you have no SSD, NVME, or other high-speed data storage, then use light sync mode.

You must be running at least an SSD to sync a full node!!

Having a fast disk and network connection is important and super helpful to sync... this is why I didn't put the data folder on a different HDD in the system. This method may also work for slower HDDs also, but you will probably need to wait a lot longer for best chances. (full node requires SSD)

Now let's make sure we back up our old keystore files and any other data we can't afford to lose to a few places such as Desktop and maybe a backup folder on another hard drive. We will be completely removing any remnants of old wallets from the AppData folder and if you delete your wallet by accident there is no going back!

If you already have been tinkering with geth.. let's make sure we get rid of the old blockchain and associated database. Navigate to your current geth folder in the command line and run the following command:

geth removedb

answer "y" when prompted and exit the window once it finishes. Depending on the size of your blockchain this may take a minute or so.

Either use the default control panel uninstaller to remove old installations of Ethereum or Mist or any other Ethereum client/wallet you don't want to use anymore. Again.. make sure you back up any accounts before doing this. I like using Revo Uninstaller to be sure to get every last trace that I can off the system. Once uninstall process is over check and make sure the folders in your Users/YOURUSER/AppData related to Ethereum are gone.

Might be a good idea to reboot the machine at this point just to be sure nothing is hanging around in memory or whatnot.

Back in Windows, be sure to test your speed by running a speed test. Just do a google search for one, test your speed, and make sure you are not behind any firewall, VPN, or Proxy that may slow the download and sync process. You can also find "Time to Download" with a quick search as well. After a full sync, my blockchain folder is about 100GB so plug that in along with your speed in the calculator to determine an estimate of how long the entire sync will take. If you run a high-speed cable connection you can get up to 200+mbps in many cases (don't hold me to that.. or anything for that matter).

By default, Ethereum installer seems to like making two folders in your "AppData" User folder. This is hidden by windows explorer by default, so be sure to change explorer to show hidden files and folders.

Users/YOURWINUSERNAME/AppData/Roaming --> /Ethereum Wallet (here on known as "Ethereum Program Folder") /Ethereum (here on known as "Ethereum Data Folder" shows up after you first start geth)

These are the two folders that the client seems to like to use by default. After you first install the package from ethereum.org it will put the program files in the Ethereum Program folder outlined above. The data folder will be created when we start syncing the node.

Now that we extracted the vanilla package and it is in the AppData folder we will navigate to it using windows command prompt in administrator mode. Click the Cortana search thing and type cmd. Right-click cmd and run as administrator.

A new window opens... change directory to your system drive's root (usually C:)

C:\Windows\System32> cd /

C:> _

Open your file explorer and navigate to this folder: C:\Users\YOURUSERNAME\AppData\Roaming\Ethereum Wallet\binaries\Geth\unpacked Right-click the address bar and choose "copy address as text"

Back at your command prompt... we are going to change to that directory by typing cd and pasting that address.

C:> cd C:\Users\YOURUSERNAMEHERE\AppData\Roaming\Ethereum Wallet\binaries\Geth\unpacked

C:\Users\YOURUSERNAMEHERE\AppData\Roaming\Ethereum Wallet\binaries\Geth\unpacked> _

Now we are finally ready to start the sync. Once you start this next command you must wait until it completes most of the chain as I will explain shortly. Make sure you won't need to reboot any time soon in case it takes a little longer than expected. Be to set the data dir correctly, not in the folder you are currently working in. Type the following command (must have 2GB or more RAM):

geth --cache 2048 --syncmode="fast" --datadir="C:\Users\The\AppData\Roaming\Ethereum" console

If you want to you can watch the progress bar from the regular Ethereum wallet link installed on your desktop by launching it after getting geth running in the command line. It should auto-connect to the process running in the command line window.

Hopefully, this will begin pulling the blockchain down pretty fast. After your time estimate from earlier has passed (good idea to add some time) check the status. Remember, this can take a long time depending on your connection. If it completely syncs, congrats your all set.. just import your old keystore to the data dir. But this is where most problems start and people get confused. I got stuck here for a while because it would infinitely download "chaindata" and always be behind about 100 blocks. The program just gets stuck in a loop of "Imported new state entries."

IF you get stuck here like I did (constantly getting stuck on imported state entries and why I made this guide). Thanks to karalabe's post we learn that the node doesn't hang, it just doesn't know how large the state trie is in advance so it keeps on going and going and going until it discovers and downloads the entire thing. The problem is that this would run for me for over 24 hours with no success. You can either try waiting it out for hours and hours, or restart the client, but either way this part will go on for a while.

First, click Launch Application and then quit the GUI if you launched it... Click the command line window that geth is running in and type "exit" and hit enter. The console will be scrolling a lot so it looks like you can't type in commands but they still work.

If you don't have an account yet, maybe make one at this point... not absolutely sure though. If you are using an old account, move the backup keystore to the new keystore in the data folder.

Now that most of the blockchain is in our data folder I had luck by running the following in command line using the "light" sync mode.

geth --cache 2048 --syncmode="light" --datadir="C:\Users\YOURUSER\AppData\Roaming\Ethereum" console

After running this go ahead and attach the GUI by clicking the desktop icon and Launch Application. If you didn't get synced before then this should probably work. The problem is we don't want just a "light" sync... but this was a step I took to get it working so I included it. Uncheck "Sync with light client" in the GUI and quit Ethereum Wallet GUI.

Go back to command window and type exit into the console and hit enter to stop geth.

Now that we did a light sync, let's do a full sync and let it run for about an hour (at 200mbps net speed). I don't know if this helps, but it was a step I did to get the final sync to work and I don't see how it can hurt.

geth --cache 2048 --syncmode="full" --datadir="C:\Users\YOURUSER\AppData\Roaming\Ethereum" console

Open the GUI again and Launch the Application. I waited about an hour or two at this point, even though it didn't seem to make a lot of progress. If the wallet still is not syncing (like mine) after this time lets quit again. Remember quit the wallet GUI first then type exit into the command line console.

I rebooted the computer again at this point. After this, I did another "fast" sync and it finally completed after about a half hour.

geth --cache 2048 --syncmode="fast" --datadir="C:\Users\YOURUSER\AppData\Roaming\Ethereum" console

I now have a fully synced blockchain and can do contracts or whatever. After getting this first sync to go through it should be smooth sailing until the next update. Should never need to launch from the command line anymore.. or have to use Mist or some other wallet when I kinda like using the default program.

I really hope I didn't forget anything that I did to get this to finally work... if you stuck with me through this guide and got it working then great! Again this is my first GitHub post, please let me know if I did anything wrong or posted any sort of misinformation.

Good luck everybody and have fun!

my ETH: 0xD5E0F0e2065E658E09F8131956D4A9e1ac5F3C46

[Chanajit]

😊

[evertonfraga]

Hello everyone, I wanted to share some great news here:

We've launched Mist and Ethereum Wallet v0.11.0, which comes with a major upgrade on how we connect to the blockchain. It should solve the burdens of syncing, so far.

Please head to the download page for links: https://github.com/ethereum/mist/releases/tag/v0.11.0

I am closing this issue, should one of you have any trouble, please open a new one. Thank you!