Discussion of a wild idea: Multiple Genesis Blocks, Archival Node

[Technologov]

I think we may be able to reduce a block chain size dramatically, by re-generating automatically a new genesis block every year, (or every X blocks), and keeping the balance of the previous block chain, last block of it. Theoretically it will solve the hard disk space issues, And will reduce network bandwidth requirements for setting a new Full node. It will not solve new block propagation delays.

Objective: My primary concern is that with on-chain scaling, our blockchain can grow into a multi-petabyte size, over a few decades, and both network capacity and hard disk space demands will out-grow our ability to sponsor those. (even with incentivized Masternodes)

Concept: 2016 [block 0]...[block n] --- the last block's balance could transfer into a new genesis block 2017. 2017 [block 0]

Balance (all utxo) of each address are copied into a new Masterblock as one transaction. Masterblocks are created in a deterministic fashion, so every node can see it and reproduce the correctness of a Masterblock.

Afterwards Mining should start build up on the new chain. Just the history of transactions will be lost.

There could be more types of nodes: Archival node, keeping all history of all the previous block chains, and the Full Node, keeping only the current block chain.

In this scenario, only block explorers and scientists will need to keep an Archival Node. It is not be needed for payments and verification at all.

This idea, if it works, will mean that Dash will beat Visa x 10 transactions easily, with a goal to break 1 billion transactions per day, in a decentralized way.

NOTE1: Blockchain pruning, as is done in Bitcoin has very bad side effects: namely inability to start new nodes from them and doing full rescan. My approach doesn't have such downsides. https://news.bitcoin.com/pros-and-cons-on-bitcoin-block-pruning/

04.Mar.2017 -Technologov

[Technologov]

Clarification 01: New Term: Let's rename our so-called new Genesis Block into a Masterblock. (for clarity)

Clarification 02: New Masterblock must be generated in a deterministic way, to be reproducible by all miners and Full Nodes.

Clarification 03: tiny dust amounts can be distributed to the last 1000 miners. Let's define 'dust' as transaction outputs below the median fees for the last 100 blocks. (this removes a bunch of bloat from the blockchain)

Clarification 04: Old chain doesn't get removed immediately, but only after Y blocks (say after 1 month). This means that new nodes connecting to the network during this period can download both new chain and old chain, can verify the new Masterblock by recalculating it from the last year's chain.

Clarification 05: Q: What do new nodes do when they come online after this one month has passed? How do they verify the correct chain from scratch? A: Consensus must work as follow:

1. Same as nowadays (longest chain) AND
2. current timestamp (of year 2017) - proof of work must include timestamp. (if it finds a very long chain, but with old timestamp like 2015 or 2016, it gets rejected; future timestamps also get rejected)

TODO: Research new attack vector: NTP protocol / wrong time. (For now my idea requires the administrator to manually setup clocks on critical production systems, such as payment processors. Maybe we can solve this problem later.) -- Perhaps we can solve it via an idea of cryptographic time -- a separate block-chain for time-keeping, with the same X11-proof-of-work and same difficulty as the main-chain, that doesn't reset on each Masterblock -- it's continuous. It should be merge-mined together with the primary block chain. Basically Segregated Time - 'SegTime'.

NOTE: New Inconvenience: My idea, as written now, will invalidate masternode tx-outputs, forcing people to restart Masternodes. In the meanwhile MNs can function as Full Nodes. (But perhaps there is an elegant solution to this problem too... -- something like an extended block, that will be written into the Masterblock, and will list all previous transactions with 1000.0 DASH in unspent tx outputs as valid.)

================ Okay, Let's do some maths: 1 GB block each 2.5 min = 4 GB in 10 min (equivalent of a 4 GB block in Bitcoin). This will allow us for 1 billion tx/day. In Dash it would equal to 576 blocks/day x 1 GB/block = 576 GB/day. (In Bitcoin it would equal to 144 blocks/day x 4 GB/block = 576 GB/day.)

It's 576 GB/day x 365 (year) = 210 TB-per-year.

Without my idea we will grow into a multi-petabyte territory in 5 years. Will be hard, even with incentivized Masternodes. With my idea, it would take only 18 Hard Disks (okay 20 HDDs, with RAID6) to keep an entire block chain (that's assuming big 12 TB HDDs; that both Seagate and Western Digital started producing this year).

Block propagation ? It takes only 8 seconds on a Google Fiber (1 Gbit/s Internet) -- so I believe it's very much possible and feasible to grow with on-chain transaction scalability.

[akhavr]

1. How fresh new client would determine if he's fed with a valid masterblock
2. If the validity of a masterblock would be confirmed by masternodes, how fresh new client would determine which masternodes to trust?

[Technologov]

It's already answered in C04 (Clarification 04) and C05.

[taw00]

I think there are some ideas swirling around the bitcoin world on this topic. Part of the challenge is that certain types of transactions have to be honored somehow: Future dated, etc.

[sidhujag]

You may want to look into directed acyclical Merkle DAGs because they essentially use the last transaction which is a hash of all the previous hash's as the validity of any future transactions, essentially a blockchain-less system that tries to emulate a chain of transactions to do the same thing.

[Technologov]

Is there any alt coin that implement said "Directed acyclical Merkle DAGs" for tx scalability?

[sidhujag]

Iota and to lesser extent byteball afaik.. its still unproven however read their paper and critique it

[Technologov]

Taw00: maybe we can make a special section in the MB with special transactions, and list them as 'valid' - Masternode tx with 1000.0 DASH output, future dated, multisig, etc... it needs some though.

[Technologov]

Opinion from Bitcoin people: https://bitcointalk.org/index.php?topic=1851463.0

spartacusrex said: (April 03, 2017)

Like.

BUT. Could be easier. Don't need a Master Block.

1) Throw away EVERYTHING except the Block-header for blocks older than 1 day / (..or the longest re-org imaginable ). The block header is the ONLY bit you really need - as it has the POW. Also it links back to the previous block header, so we have our 'Proof Chain' back to the Genesis block. (Which would be tiny). But still packed with ridiculous levels of POW-er..

2) You need to store the deterministic UTXO root hash somewhere in the block header. Changes for every block obviously. But everyone would agree on it.

3) That's it.

As with your design we lose the TXN history - but we keep the POW chain that led us to the present day UTXO.

So.. doing the Maths.. it would be about :

576GB (as you calculated for 1 billion txn a day)+ UTXO database size (orders of magnitude smaller than 0.576TB) + the proof chain size (tiny).. forever.

Whether there would be any point to keeping the TXN history - is debatable..

[ghost]

this seems like a really inefficient way to prune down to UTXO set. The masterblock would be huge - nodes can just prune if they want. There are no real savings here, and pruning spent tx on all nodes will cause other issues an lose history e.g. metadata in spent tx.

[Technologov]

Standard blockchain pruning, as done in Bitcoin disallows people to start new nodes from pruned nodes.

[ghost]

yes but new nodes synching from a masterblock are gaining the same info as they would from a pruned node. in fact less.

the main issues are i have are:

1. this is just an ugly way to use blocks - the network has a certain ebb and flow with blocks gradually adding new tx to the chain every 2.5 minutes - shoving everything in a gigantic block once a year is not within the design rationale for such a system - apart from needing an absolutely huge block (an entire year of UTXO in one block) that needs to be propagated and verified, your adding this huge focal point where if that block is attacked the network is stuck

2. to do this we have to be ok with pruning everything except UTXO. the reason this is optional is there are TX types and metadata that are really needed. Moving that problem to archive nodes just reduces the security, and how will they be incentivized etc?

Sorry to be a wet rag, just think this needs a lot more thought and deeper understanding of the architecture. Maybe one of the core devs can answer better.

Andy

EDIT: @Technologov thanks for submitting this anyway, maybe you should write a paper on it if you want to present the full details, above is just initial impression from flicking through the details and not sure what the other devs think.

[Technologov]

1. It will increase delay of the Dash network once a year. Ugly ? Yes, a little bit. But it is better than having a multi-petabyte storage array, that even incentivized Masternodes will be unable to afford. Server-grade hardware is not powerful enough to push a Petabyte. It will push us into Mainframe level (Super-computer) territory where costs are upward of $10 million dollars-a-pop.

2. I think special transactions (such as time-locked utxo, masternode tx with 1000.0 DASH output and multisig) should be put into a special "extended space" inside a Masterblock, and special transactions must be marked as valid as long as an address has the required balance. Or simply re-create similar type of transaction in every Masterblock.

3. Archival Nodes are not required for network operation, at all. (only recommended for block explorers, but not required)

========= After all: We have several years to figure out a long-term scalability strategy. But it needs to be solved before Dash, or any crypto-currency for that matter, could become mass adopted by a billion users. My proposal is one possible solution, maybe even not the best, and I urge everyone to look for solutions, because this problem is critical. But Masterblocks idea is a solution worth keeping in mind, if all else fails. And my solution has a realistic chance of working on standard server-grade hardware of $10,000.

[ghost]

1. Ok but right now the highest load we know works is 1MB per block every 10 minutes (excluding Dash testnet which test ok at 4x that and pretty sure we can push that far higher). How big would the masterblock be? Could be 1GB for the year on 10GB chain growth? How is that possibly going to be propagated / verified in a timely / secure fashion with continued processing of existing TX load, assuming we remove any kind of size limit? It's also a huge attack surface at a given time that can be targeted. The MB size also scales with the average block size so it would get larger as blocks get larger

2. But then we need to discern which spent tx to add to the masterblock making it larger

There are more elegant solutions being researched in the space like sharding (although sceptical on that too). Basically, don't see any elegant solutions to this yet. Lot of solutions in the space try to improve on Bitcoin efficiency but truth is the core architecture is about as efficient you can get without some radical redesign (from what we know today at least).

To make this viable, i think you'd need to first show that a masterblock is feasible in itself - size, propagation, verification time, security with such a larger block at predesignated height. but i'm pretty sure you will find that it's not possible within the architectural limits IIRC.

[joezippy]

Hello,

I know I'm new here, but I might have something you want to think about, as you consider moving forward with the concept of "more types of nodes: Archival node, keeping all history of all the previous block chains, and the Full Node, keeping only the current block chain." (@Technologov)...

The idea (more details in the forum post) has to do with speeding up the mixing of coins for Private Send functionality, essentially by flagging transactions on the chain, as already "mixed / private" previously... Maybe this is a "PrivateSend node" or just identified on the "Archival node" as type private... Regardless... The idea also suggests moving the private send / mixing functionality from the wallet endpoints, to the masternodes which would allow all wallets to PrivateSend, not just the QT in the future....

Anyway, here is the thread on the dash.org forum... It should probably be moved out of the "General Discussion" to the "Development Tech Discussion" for more developer visibility, but I can't do that... :)

https://www.dash.org/forum/threads/masternode-private-send-mixing.14238/

Let me know if you have questions about the ideas... I'm not at the same level as you all are (yet), but I'm on the same Dash team!

Thanks for your time... Cheers!

[joshafest]

The idea is good!! Can anyone add features and create pull here ? I mean can anyone submit his work by creating pull request or it is only done by team ?

[Technologov]

Anyone can submit a patch, that will be reviewed by the Dash Core team.

[joezippy]

@joshafest ... Just for clarification is it my idea you think, is good for resolving current issues w/ mixing and private send or the bigger topic @Technologov is proposing... If it's mine, are you considering implementing this pseudo-code?

// So I might be way off base, but somewhere in the dApp lives a 'write' function 
// and somewhere their lives a 'read'... If that is true... Maybe something like this 
// could be done...

public read(addr) {
   // psAddressRegistry contains all address that are private 
   if (addr in psAddressReqistry) {
       addr.trans.obfuscate.sender()
       addr.trans.obfuscate.receiver()
   }
}

public write(addr) {
   addr.trans.do-what-you-do-today()
}

Wishing I had the skills to implement it myself... Just not there anymore... I'm hopeful you can get it done @joshafest ! :)

Cheers!

[Technologov]

joezippy: Private send is NOT part of my idea. This idea -- "Masterblocks" -- is squarely targeted at scalability problems, not privacy.

You should create a separate issue on github.

[joezippy]

@Technologov not trying to hijack the thread, just clarify @joshafest comment... Cheers!

[joshafest]

What I think is .. accept the last checkpoint block as initial block with height and treat it as it was genesis block ... Problems

• one who have coins on much earlier blocks, have no effect of this solution as he has to download whole chain.
• One who want to import private key, have problems.

[ghost]

@Technologov read https://github.com/cyberdust/wiki/wiki/White-Paper#8-blockchain-compression

[Technologov]

I have updated the specs: https://docs.google.com/document/d/1usnNigTJw6cpone_xrcIMmzxF7U1MFrjb3y7aTsbDx0/edit?usp=sharing

[nmarley]

Good forum topic discussion, but not feasible as an issue to be solved. Closing here, please open a thread at dash.org/forum for further discussion.