Snapshots to cut full node bootstrap time 2000x

[homakov]

As of now the blockchain is so long that becoming a full node is a tedious process not just on a desktop machine but on a powerful server too and takes days.

The solution to this has always been on the surface: make a snapshot of current state (chainstate folder) and serve it to new users to start right from that point.

It was commonly rejected as insecure:

https://bitcoin.stackexchange.com/questions/41611/chainstate-database-for-download

Please don't go download a chainstate from a random place on the Internet though. They can introduce arbitrarily many coins assigned to themselves, and your client would accept them

https://eklitzke.org/an-overview-of-bitcoin-utxos

Today you need to download and validate the full Bitcoin blockchain all the way from the genesis block in January 2009 to locally build the chainstate database. This is about 170 GB of data, and it’s a fairly onerous task

In fact, none of that is true. Snapshots do not downgrade the threat model of blockchain bootstrap a single bit: see these articles why: https://medium.com/fairlayer/snapshots-the-simplest-way-to-increase-number-of-full-nodes-3ebf2aaef515 & https://medium.com/fairlayer/3-misconceptions-about-full-nodes-bb03f0d467b7

Here is the offer:

Run a btc node on bitcoin.org and make a gzip of chainstate folder along with blocks (btw, 500 mb tail even in pruning mode? that's insane, ~500 blocks fork! 50 would work just fine). Let any other service to do the same and have interoperable install scripts with them. The install script contains a hash of current code + state and by checking against multiple sources the user gains more security than ever before.

Snapshots can be done once in 24 hours, and must be deterministic so all other independent services serve exact same hash of current state.

This way a new user can get a functioning full node by downloading just 3gb + a few blocks on top. From that point syncing requires 1 minute per each missed day on most consumer laptops. This would dramatically increase number of full nodes without making any compromise to threat model.

Installing from state-1 (genesis state) now becomes an option instead of forced requirement.

Also, number of incorrect wording should be fixed over here:

If you are trying to set up a new node following the instructions below, you will go through the IBD process at the first run, and it may take a considerable amount of time since a new node has to download the entire block chain

Running a Bitcoin full node comes with certain costs and can expose you to certain risks

{{site.text.bitcoin_datadir_gb}} gigabytes of free disk space, accessable at a minimum read/write speed of 100 MB/s.

To be removed as it scares user for no good reason. Running a full node is cheap, fast, takes little space/bandwidth and should be encouraged. Archival mode must become opt-in either, there's no point in storing 150gb chain on a rarely connected laptop.

[harding]

As of now the blockchain is so long that becoming a full node is a tedious process not just on a desktop machine but on a powerful server too and takes days.

On my 7-year old 4-core Xeon server with 16 GB of RAM, it takes two hours. On my 2-year old dual core i7 laptop with 12 GB RAM, it takes 4 hours. In both of these cases, the trick is to increase the -dbcache parameter to Bitcoin Core to about 80% of your typically available RAM. This is only needed for the initial sync; after that, the default value is fine.

by checking against multiple sources the user gains more security than ever before.

This is untrue. Checking a hash of the chainstate against multiple sources requires trusting those sources as well as trusting the code that you're running. Running full validation on the historic block chain means the only thing you're trusting is the code you're running---and the code is provided for inspection (with a reproducible build so you can ensure the exact binary you run derives from the code you audited).

Using someone else's chain state is a clear reduction in security.

btw, 500 mb tail even in pruning mode? that's insane, ~500 blocks fork! 50 would work just fine)

Blocks can be up to 4 megabytes and each time the chainstate is updated an undo file is created to allow rolling back the chainstate in case of a fork. Undo files are smaller than blocks but still consume some disk space and are included in the pruning limit. For expected typical sized blocks under full segwit usage, this basically means the 500 MB allows seemless recovery from a fork of about 144 blocks (1 day), which seems reasonable. Your proposal of 50 MB could theoretically fail on a fork the size of which we saw in the BIP50 consensus failure. (FYI, the longest fork to date was at least 52 blocks.)

This would dramatically increase number of full nodes without making any compromise to threat model.

You've presented no evidence that making the startup costs cheaper will dramatically increase the number of running full nodes.

The threat model goes from being "worry about the software being compromised" to "worry about both the software being compromised AND worry about the UTXO snapshot being compromised". That's worse IMO.

To be removed as it scares user for no good reason. Running a full node is cheap, fast, takes little space/bandwidth and should be encouraged.

This website has more than a dozen pages dedicated to encouraging users to run a full node. However, I think it's the responsibility of anyone encouraging people to run full nodes to also describe the downsides---anything else would be dishonest.

---

A better way to accomplish the same basic thing you're proposing has been suggested: have a wallet start up almost instantly as an SPV wallet so that it's immediately useful and then begin compiling the UTXO state by processing the block chain backwards towards the genesis block. Upon finishing all historic validation, it becomes a full node. Getting the UI around this correct is hard because SPV and full verification provide two different security models, but it's something that's possible and that we may eventually see.

[homakov]

On my 7-year old 4-core Xeon server with 16 GB of RAM, it takes two hours. On my 2-year old dual core i7 laptop with 12 GB RAM, it takes 4 hours

Also, take into account bandwidth issues. With 100Mbit sure, it can get pretty fast.

and the code is provided for inspection (with a reproducible build so you can ensure the exact binary you run derives from the code you audited).

The key difference is do you, personally, introspect the code. If you don't, you gain nothing from state-1 vs state-123456. You trust the server that gave you the code anyway, and since you already trust it, you lost nothing by trusting it with not just code but the state too.

from a fork of about 144 blocks (1 day), which seems reasonable

Didn't know of rollback files, interesting. 1 day sounds OK to me.

You've presented no evidence that making the startup costs cheaper will dramatically increase the number of running full nodes.

I follow the space and well, 2-4 days to bootstrap a node sounds like a deal breaker for 99% of ppl. If you turn that into 5-20 minutes, this changes everything. IMO. Doesn't fix the ugliness of Qt though, but still.

of anyone encouraging people to run full nodes to also describe the downsides---anything else would be dishonest.

And what i'm saying there are no downsides to running a pruning full node that was started from a snapshot. You neither spend much time on bootstrap, nor you are required to store all blocks. Full nodes do not inherently require these 2 things.

A better way to accomplish the same basic thing you're proposing has been suggested

I've seen hybrid SPV proposal. Snapshots provide full-node security from the very install.

---

My primary disagreement lies here:

The threat model goes from being "worry about the software being compromised" to "worry about both the software being compromised AND worry about the UTXO snapshot being compromised". That's worse IMO.

Software and the state inside it are one whole thing. Genesis state is the state that lies as a constant in bitcoin/bitcoin, and since you already trust the server you will be just fine with having the latest state instead of genesis one.

If you are able to examine the node sources, then you can go ahead with state-1. If you're a normal user (let's face it, no one reads all sources before installing), no need to torture yourself with full sync, your threat model doesn't change from that. You already had a root of trust anyway.

[homakov]

Also, if you disagree let's stick to the facts vs feelings.

Please demo a scenario where a user follows some install procedure (visits X runs Y) and compare two threat models where they are served the state-1 and state-lastest versions of software.

Please show how they differ.

[achow101]

NACK

This introduces additional unverifiable trust in bitcoin.org's servers (users must trust that the server is providing the correct chainstate data). By unverifiable, I mean that it is impossible to verify that the chainstate data that you downloaded is actually the correct chainstate data before you load it into Bitcoin Core. The chainstates are not deterministically built and are completely dependent on the node that it came from.

While the vast majority of users do not inspect the code themselves and perform the deterministic builds, a number of people do, and most users tend to trust those people. However, those people cannot also verify that the chainstates provided are correct. This reduces the trust model from "trust servers and independent auditors" to just "trust servers".

[Cobra-Bitcoin]

Security issues aside, do we really want huge numbers of pruned full nodes on the Bitcoin network? Doesn't that change the topography of the network significantly and make it much harder for peers to find archival nodes?

I like the general idea of having something as easy to install and use as an SPV node, but that also gives users the full security benefits of a full node. But it's really not secure to trust servers on the internet to give you chainstate. With the binaries, you can deterministically build them and verify the signatures and at least know that the code has been inspected by lots of eyes.

[homakov]

@achow101 please follow the template offered in "Also, if you disagree let's stick to the facts vs feelings".

I mean that it is impossible to verify

If the user does not verify the source code, he loses nothing by trusting the state either. It's called setting up the root of trust. Afterwards, no trust is needed. Avoiding the root of trust is a very lengthy procedure which I'm not aware if anyone ever tried to avoid.

a number of people do, and most users tend to trust those people

Auditors and anyone interested still can use state-1 version.

You're given two files: "code" and "state". Normal people couldn't care less what "code" contains, so they install and run whatever is inside. Advanced users can verify code and start with state-1. Nobody's threat model got any worse.

Once again, if you see a change in threat model, define it by the steps as offered above. No place for "feelings" here.

@Cobra-Bitcoin

But it's really not secure to trust servers on the internet to give you chainstate.

That's the equivalent argument as 2 others above. You already trust exact same server with binary code.

• You don't verify anything? (99.9%) - get code with last state
• Planning to verify code thoroughly? (0.1%) - get code with state-1, verify code, run from genesis.

If you make the user download from state-1 while in fact the user never verifies the code, you're simply wasting power on nothing. In fact their bitcoind could have had a full node all the time and show the "loader" for 3 days just for fun.

But more importantly I believe we must admit that the second group is not just small but rather non-existant. And embrace an approach with install scripts where you can check 5-10 sources for their code+state snapshot and the more trust-sources you check the higher degree of confidence you get.

Full code and consensus implementation verification takes unfeasibly long and requires unfeasible expertise. It's better to ensure the install process has verification against other servers not just bitcoin.org's.

State-1 and state-lastest must be offered alongside on the same page, with a notice that you only need state-1 if you are planning to inspect the code. State-latest must show hash of code+state so it could be verified against multiple other sources to reduce risk of compromise (which is present right now).

[homakov]

Security issues aside, do we really want huge numbers of pruned full nodes on the Bitcoin network? Doesn't that change the topography of the network significantly and make it much harder for peers to find archival nodes?

In my opinion, if we consider tiny group that verifies 100% of the code and has 100% expertise in consensus implementation is non-existent, then we can safely forget of state-1 and cut-off tail like a lizard once in a while: every 3 or 12 months. This way the nodes that were offline for a year still can sync, but all new nodes are offered to jump onto latest state (with many sources to check against, but still root of trust has to be set up).

First 8 years of Bitcoin history are simply irrelevant to all new users who are not from this tiny non-existent group. Only current state matters to them.

Root of trust is inevitable. How do you know how consensus work? Bitcoin Wiki. Or someone on the Internet. Even when you will verify the code line by line, there are thousands of things you need to trust in/google for, as you didn't know of them earlier.

[achow101]

@homakov In what way did I say anything that was "feelings"?

The current trust model is this:

• Developers write and review code. Other people (who may or may not be other developers) also review that code
• Deterministic builds are performed for each and every release. Hashes of all of the released binaries are GPG signed by multiple people are are freely available to be downloaded from multiple places.
• Other people (developers or not) can perform the same deterministic builds and get binaries whose hashes exactly match those that are released and GPG signed by other people
• Windows and Mac users (the vast majority of users) download binaries which are code signed (a signature is in the binary itself) with a key held by the developers.

In this model, users are only trusting the developers and auditors who also perform the deterministic builds. They are not trusting the server or the person uploading the binaries and hashes to the server. This is because the hashes file is signed by a fixed and known GPG key which can be verified. And because the Windows and Mac binaries are code signed with a fixed and known code signing key. Thus if the server were to tamper with any of the uploaded files, their hashes would not match the hashes in the signed file OR the signature would be invalid.

Under this model, there is no need to trust the person uploading to the server because all of the binaries can be independently verified by auditors who will notify users if something is found to be amiss. Note that this has nothing to do with GPG signatures but rather because of the deterministic build system.

However, once we introduce a chainstate download, we are introducing another level of trust: the person creating the chainstate (note that this is independent of the group "Developers" even if this person is a developer). Because the blockchain chainstate is not built deterministically, users are now trusting that the person who created the chainstate did not create a false chainstate since auditors cannot build the chainstate themselves from their own node and exactly (bit for bit) the same as the chainstate provided. Thus we now introduce another level of trust in the person creating the chainstate, whereas with just the binaries, there is no trust needed in the person creating the binaries.

---

In general, UTXO sync degrades the security of the network. When users no longer verify the full blockchain, they become open to attack from someone who provides a fake UTXO set. And just because users don't need/don't want to verify all of the blockchain data does not mean that we should force everyone to not do that when they first sync.an attacker to produce a fake UTXO set for people to use.

[mjamin]

If I, as someone who is unable to review the code himself, already trust developers + code reviewers, wouldn't it be possible to hardcode a checkpoint block hash into the client? Make the chain up to that point effectively part of the source code. How exactly does the trustmodel change there?

Edit: This would be an optional shortcut, obviously. The software always needs to be able to do a full verification of the chain. Maybe it shouldn't be the default behaviour, though.

Should've read @achow101's post first :)

[homakov]

In what way did I say anything that was "feelings"?

Everything in your last comment still falls under "feelings".

I proposed a clear step-by-step argument above:

Please demo a scenario where a user follows some install procedure (visits X runs Y) and compare two threat models where they are served the state-1 and state-lastest versions of software.

That would show a factual proof. Again: how a group of users that does NOT verify code benefits from state-1 sync?

And just because users don't need/don't want to verify all of the blockchain data does not mean that we should force everyone

I never said to force everyone but to provide an option. Me, and millions of other users are ready to explicitly trust the server of software with the chainstate, as they already do with the code itself. That's called root of trust and that's how all software worked for ages. This is not forced on users, that's their choice.

Code signing for Mac/Win adds nothing to the table. Signed code could just as easily contain the hash of state-latest over state-1, and be cross-signed by same groups of people. They only requirement for them is to have a working node to produce the latest snapshot.

[kek-coin]

Everything in your last comment still falls under "feelings".

Just because you feel like his arguments fall under "feelings" doesn't make it true. All his arguments are logical, security analysis is not "feelings". So I would suggest you stop strawmanning his arguments as "feelings", and actually read what he has to say, as his arguments are cogent and coherent.

Besides, your "scenario demo" thing that you're trying to railroad this discussion into would not constitute "factual proof" but "hypothetical anecdotal evidence".

[homakov]

as his arguments are cogent and coherent

Just as mine. If code is not verified, no point in syncing from genesis. Simple as that.

[romankoblov]

Because the blockchain chainstate is not built deterministically, users are now trusting that the person who created the chainstate did not create a false chainstate since auditors cannot build the chainstate themselves from their own node and exactly (bit for bit) the same as the chainstate provided.

Why not? It is pretty easy to create deterministic chainstate, since there always same UTXO at specific block height. And everybody who has running node can verify that specific chainstate is same as it should be.

---

BTW, deterministic verified chainstate snapshots even more secure than current approach, because there is possible malicious version of client which will work correctly with current consensus (with sync from genesis), but will produce malicious UTXO at specific block height.

[harding]

@homakov I don't think you're reading what @achow101 is saying. He said the chainstate is not built deterministically. You can verify his yourself by running the gettxoutsetinfo RPC on two different nodes. Note the hash_serialized_2 field:

It's possible to create a deterministic commitment to the chainstate but Bitcoin Core doesn't currently support that feature, and so your proposal doesn't work right now. (It also has all the problems described above which you've tried to ignore by labeling them as feelings, but at least if we can agree it doesn't work right now, we can close this issue until someone does make it workable.)

[homakov]

He said the chainstate is not built deterministically

That's bad but it only makes the "verify chainstate against other sources" part not possible.

It is still possible to implement the feature to download latest code+state without verifying with other sources (because their hash would be different) as long as the user opts-in into that.

My statement is as long as the user intentionally doesn't verify any hashes, GPG sigs or whatnot, there is no need to do any sync when you establish root of trust. And there are plenty of users like that.

I'm able to setup a service like that where you can setup full node in couple of minutes, but there needs a reference for this option in official guides. Otherwise guys you're just shooting into your foot for a made up reason.

[mjamin]

If you don't want to properly verify the chainstate, why bother with a fullnode at all?

[harding]

there is no need to do any sync when you establish root of trust.

Echoing @mjamin's comment, there's no need to do any validation at all when you establish a root of trust! If you trust Coinbase, you can just let them handle your coins and skip all this nitty-gritty validation stuff altogether.

Maybe one day validating the chainstate will be so burdensome that shipping with some chainstate bootstrap mechanism will be required, but I don't think we're anywhere near there yet (mostly thanks to conservative block sizes and years of work on validation optimization)---and I think that (if that day ever comes) we're much better off implementing a highly conservative mechanism into full nodes directly rather than making it easy for users to lose money by trusting random websites that can be hacked, corrupted, seized, mimicked (e.g. phising), or redirected (e.g. BGP plus SSL cert attacks).

Anyway, I'm finished with this conversation and will be unsubscribing from this issue. Thanks for the pleasant debate.

[homakov]

If you don't want to properly verify the chainstate, why bother with a fullnode at all?

Because it doesn't make sense to verify the chain with unverified code. Most users don't verify code, that's a fact. Those who do can run from state-1, no one is forcing snapshots on them.

If you trust Coinbase, you can just let them handle your coins and skip all this nitty-gritty validation stuff altogether.

There's huge difference between trust-on-first-use and trust-on-every-use. Snapshot requires trust only at the bootstrap.

but I don't think we're anywhere near there yet

Days of sync and 150Gb of bandwidth... falls under "burdensome" to me.

years of work on validation optimization

that's especially painful for me to watch as all optimizations go in vein. Each day can be synced in 1 minute, yet only real enthusiasts run full nodes on desktops (in reality absolutely anyone can with snapshots).

by trusting random websites that can be hacked, corrupted, seized, mimicked (e.g. phising), or redirected (e.g. BGP plus SSL cert attacks).

Right now any of that can happen to bitcoin.org, how are users protected from that? (given that practically no one verifies any sigs).

[mjamin]

Days of think falls under "burdensome" to me.

Dual core i7 laptop, 12 GB RAM -> 4 hours.

Right now any of that can happen to bitcoin.org

Sure, but users can opt-into a high level of protection by actually checking the provided sigs. This isn't possible in the same way with the chainstate as it's non-deterministic and there is no single sig to verify.

[homakov]

Dual core i7 laptop, 12 GB RAM -> 4 hours.

World average bandwidth is 8 mbit. That's 40+ hours, the very minimum.

Let's split my offer in two:

• Verify with multiple sources (not possible because non-det. chainstte)
• Trust just 1 source (possible right now)

Why not explain users the risks and offer the second option temporarily until chainstate gets deterministic? Why push them to light clients and worse custodians when this solution only requires trust on setup vs trust the validators (light clients) or trust-every-second (custodians).

A very fair tradeoff to me. I would use it and I understand the risks.

[wbnns]

Closing this due to lack of agreement and/or support.

@homakov Thanks for spending the time to open this issue and share your idea.