tradecraftio / tradecraft

Tradecraft integration/staging tree https://tradecraft.io/download
Other
13 stars 9 forks source link

Merge Mining in Tradecraft without Forward Blocks #84

Closed maaku closed 4 years ago

maaku commented 4 years ago

This pull request implements so-called auxiliary proof-of-work (AKA "merge-mining") using SHA256d in a bitcoin-compatible construction, as a soft-fork. Once activated, it allows miners to simultaneously work on both Bitcoin and Freicoin/Tradecraft blocks at the same time, finding proof-of-work solutions that satisfy one or the other, or both chains.

The only previously described soft-fork proof-of-work change is the Forward Blocks proposal. This is a simplification of the ideas presented therein, which achieves the soft-fork proof-of-work change in a way that can be deployed more quickly, but is still compatible with future deployment of Forward Blocks.

The basic idea is that we require each new block to satisfy two proof of work challenges in series: first the auxiliary proof-of-work (merge-mining) challenge, which can be shared across multiple chains including Bitcoin, then the existing (native) header challenge which specific to our chain.

There will be a transition period during which the difficulty of the native header adjusts down and the auxiliary difficulty adjusts up. During this period the overall proof-of-work function will not be progress-free, but the transition period will be short (a few months) and the benefits that come with merged mining well worth it. At the end, the native/compatibility headers will be at or near difficulty 1, which is effectively progress-free, or practically no limit at all once the protocol-cleanup fork activates.

The parent chain contains a commitment to the Tradecraft block at a fixed location. The native chain likewise commits to the auxiliary proof-of-work in a fixed location as well. The block header structure is modified to contain the information needed to validate these commitments.

The merge mining commitment in the parent chain is placed in the last 36 bytes of the last output of the last transaction. This helps to minimize the size of the auxiliary header's Merkle proof. The commitment format is a 32-byte commitment hash, a 4-byte commitment identifier, and the transaction's nLockTime field. Note that no lock_height field is present, so Bitcoin can act as the parent chain, but the commitment identifier is chosen so that Tradecraft cannot. This allows merge mining with Bitcoin and prevents Tradecraft blocks from containing such a commitment.

Note: since Bitcoin does not have consensus-enforced block-final transactions, and since every Bitcoin transaction requires an input, to merge-mine Tradecraft blocks a miner must have at least one input they control.

The commitment structure includes a hash of a pool-selected secret value, which is used for block-withholding protection. The pool can allow auxiliary difficulty to be split across two challenges, one which is calculated in the usual way that is compatible with Bitcoin, and the other using the block-withholding secret. This serves two purposes: (1) a miner which submits a share cannot predict whether it will be a valid block or not, and therefore cannot grief a pool through block withholding, and (2) it decouples the finding of Tradecraft blocks from Bitcoin blocks, since a share that meets the necessary work threshold for both chains isn't necessarily accepted by both.

Once merged mining is activated, every block has two proof-of-work targets: the native difficulty and the auxiliary (merge mined) difficulty. And unlike namecoin, every block MUST have an auxiliary header: merge mining is not optional on a block-by-block basis.

Both the native and auxiliary difficulty adjustment algorithms separately track observed hash rate according to the block timestamps and their expected block arrival predictors, except that the auxiliary difficulty adjustment algorithm targets a longer inter-block interval: 15 minutes / 900 seconds, which results in 4 blocks per hour, or 96 blocks per day.

Once the auxiliary proof is satisfied, a proof then needs to be found for the native header. The miner who finds this proof is allowed to change the coinbase's scriptSig or nSequence field, and/or the header's nNonce and nVersion fields. Note that the native header's nTime field MUST be the same as the auxiliary header's, so nTime-rolling isn't permitted with the native header.


This proposal is now ready for merging.

kallewoof commented 4 years ago

Only been reading description so far, but

The merge mining commitment in the parent chain is placed in the last 33 bytes of the last output of the last transaction.

Why use this, when there's an extendable witness commitment structure in place already? (Esp. if you incorporate the signet witness commitment extension system I've already written..)

You literally only have to come up with a new 4 byte header that differs from segwit and signet and use it as is..

Note: since Bitcoin does not have consensus-enforced block-final transactions, and since every Bitcoin transaction requires an input, to merge-mine Tradecraft blocks a miner must have at least one input they control.

This would not be an isssue if you used the existing witness commitment, as it is in the coinbase, which all miners have..

I guess it's an issue because you'd have to support two separate segwits in your code?

a miner which submits a share cannot predict whether it will be a valid block or not, and therefore cannot grief a pool through block withholding

...wow? That's a great idea! Is that used elsewhere or is this something you just whipped up? I haven't heard about that before.

Speaking of namecoin btw, is it possible to merge mine both freicoin and namecoin simultaneously?

The right structure for this is probably a prefix-tree, with the genesis block hash as the chain identifier.

This sounds like you can just grind prefixes, but I may be misunderstanding something (alsdo depends on length of prefixes ofc)

maaku commented 4 years ago

Why use this, when there's an extendable witness commitment structure in place already?

The witness commitment is in the coinbase, and for blocks with a non-power-of-2 number of transactions the path to the block-final transaction is shorter. For example a block with 1025 transactions has 10 hashes in the path from root to coinbase, but only one hash on the path to the last transaction, because it is an unbalanced Merkle tree. The vast majority of the size of an auxiliary block header is the Merkle tree proof--it can be up to 800 bytes just for those hashes.

If the block-final transaction is used instead, then even without the miner doing anything special we should expect a couple of hashes shaved off the proof on average, because real blocks are not going to have power-of-2 number of transactions, which is non-trivial in aggregate. An aware miner could even get it down smaller by picking transaction based on size near the end of the block, to get an optimal number of transactions for small proof size.

This is the same reason Tradecraft places the segwit commitment in the block-final position. Since bitcoin uses the coinbase instead for miner commitments, the block-final area is free for us to use.

(Esp. if you incorporate the signet witness commitment extension system I've already written..)

To be honest that slipped under my radar. I should take a look at the work you've done.

kallewoof commented 4 years ago

FWIW, https://github.com/bitcoin/bitcoin/blob/9a6245c0e9f980700bbb11f179710eacefdce12f/src/signet.cpp#L56-L118

maaku commented 4 years ago

[Having an output to spend] would not be an isssue if you used the existing witness commitment, as it is in the coinbase, which all miners have..

This is true, and it is the single largest disadvantage of using block-final outputs for the commitment. In Tradecraft we solved this by having it be a consensus rules that there be a certain type of anyone-can-spend output available for each block to use as the input to its block-final transaction. The same thing could be done on bitcoin, eventually. Until then, it is rather simple to simply make a zero-valued output with a key the pool controls.

maaku commented 4 years ago

[The block witholding secret is] a great idea! Is that used elsewhere or is this something you just whipped up? I haven't heard about that before.

The general concept is an old idea. Meni Rosenfeld described it here in 2011:

https://bitcoil.co.il/pool_analysis.pdf

Here's some discussion about it from earlier in that year:

https://bitcointalk.org/index.php?topic=6577

What he described was a hard-fork change though. I swear that at some point someone (maybe Luke-Jr?) outlined how it could be done as a soft-fork, but a quick google is not finding anything. Maybe that part of it is new? I'm hesitant to take credit until I've looked into it further.

In any case the specifics of how it is implemented are certainly unique to this proposal, since the design is tightly integrated with how the auxiliary proof-of-work is constructed. The idea of using the block-withholding secret to also randomize the arrival or merge-mining blocks was first introduced with the Forward Blocks proposal, I believe, where this trick is used to make sure that each shard chain finds blocks independently of the others. With the scheme name coin uses, a "lucky" block would satisfy ALL proofs of work, and that would be bad for the network since there would be a new block to process simultaneously on each chain. The ideal is that they are independent of each other, so that you don't get a propagation delay due to all the daemons processing their blocks simultaneously.

(This is in fact a key point of the argument for Forward Blocks achieving higher throughput with multiple independent, loosely-coordinated shard chains.)

Speaking of namecoin btw, is it possible to merge mine both freicoin and namecoin simultaneously?

Yes. And rootstock, and other merge-mined chains. The commitment mechanisms do not conflict, so you can have a single bitcoin block that merge-mines commitments for freicoin, namecoin, rootstock, etc.

maaku commented 4 years ago

This sounds like you can just grind prefixes, but I may be misunderstanding something (alsdo depends on length of prefixes ofc)

You could grind some bits of the prefix, though I don't know what it would gain an attacker.

The idea is that the path through the tree is fixed to be something immutable. Using the hash of the genesis block for the chain is just an obvious immutable way of identifying a chain. However the links would be "compressed" so that the tree only branches at the point where there are two or more entries which shared prefix bits.

There's a couple descriptions of this idea out there on the net. Here's one: http://diyhpl.us/wiki/transcripts/stanford-blockchain-conference/2019/urkel-trees/

kallewoof commented 4 years ago

Thanks for the responses. Concept LGTM

kallewoof commented 4 years ago

Weirdly, I get

configure: error: Could not link against boost_filesystem !

when running ./configure. I do have libboost-filesystem-dev installed (Ubuntu 20.04)..

maaku commented 4 years ago

FWIW I exclusively develop by doing make -C depends and then ./configure —prefix=$(pwd)/depends/x86_64*.

It should work with the system installed libraries though, so I’ll treat this as a bug.

On Jul 13, 2020, at 5:38 PM, kallewoof notifications@github.com wrote:

Weirdly, I get

configure: error: Could not link against boost_filesystem ! when running ./configure. I do have libboost-filesystem-dev installed (Ubuntu 20.04)..

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tradecraftio/tradecraft/pull/84#issuecomment-657900115, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAQ4IQV5UFV2SMS26FWBELR3OSILANCNFSM4OXAAINA.

kallewoof commented 4 years ago

A bit spammy there as I figured out what was going on. The two suggestions are the end results so feel free to ignore the above comment(s).

kallewoof commented 4 years ago

I'm gonna stop here until you look at my comments, as github UI tends to fall apart when there's too much "stuff". Lemme know when you feel like more review is welcome.

maaku commented 4 years ago

I haven't finished addressing all your comments yet, but please feel free to review more.

maaku commented 4 years ago

This pull request is now feature-complete! I added a Merkle trie structure for the commitment, so that an indefinite number of values can be committed to in the optimal location of the bitcoin block-final transaction.