Disabling ASICBOOST

[jgarzik]

A patented mining chip hardware feature "ASICBOOST" has been the subject of debate and controversy in the community.

This issue is raised for the WG to consider testing protocol/software changes that ban/disable/render ineffective this hardware optimization.

Quoting [with permission] one chip maker, Jihan Wu of Bitmain: "Asicboost is being repetitively mentioned in the reddit. Btc1 can take a very clear stance to help to ban it if community emotion desire it"

Suggested next steps for this issue:

• Reach out to other chip makers
• Develop technical strategy to render AB inert
  • Develop starting-point technical implementation
• Review, test, query WG & wider community
• In terms of internal WG priorities, this is after milestone 1 alpha.

--

Status: Unscoped. Process Note: This is outside the scope of the SegWit2x charter; post-Milestone1 material. This means that the SegWit2X WG, as a matter of policy, will (1) 1st pursue the originally stated goal and milestones, and then (2) 2nd consider other issues that might require additional rounds of agreement among the 70+ working group members.

[gmaxwell]

This is outside the scope of the SegWit2x charter; post-Milestone1 material.

So to be clear about what you've written between the lines: you have decided that impeding covert asicboost would be a violation of the "SegWit2x charter" and so you will assure that covert asicboost continues to function in your modified version of segwit and HF as people have been alleging you would do?

This will require further departure and incompatibility with the segwit proposal.

[hoffmabc]

Where does it say that it's in violation of the charter? It's outside the scope @gmaxwell.

[gmaxwell]

I am asking a simple question: Will the SW version here block transaction commitment incompatible covert asicboost, as BIP141 does for blocks where it is used, or will it be modified to continue to permit it because blocking asicboost is "post-Milestone1 material." and "require(s) additional rounds of agreement among the 70+ working group members"?

[hoffmabc]

I personally feel that the agreement should be amended to include the banning of ASICBOOST. I'm not really confident at all that others who have agreed to the proposal would disagree either. Jihan has expressly said he supports the removal. @jgarzik has tried to keep the scope here in this work group limited to what was expressly agreed upon so there would need to be an updated effort to get agreement on the new scope.

[gmaxwell]

The agreement as stands already bans segwit incompatible covert asicboost: It says the signers will immediate deploy segwit. And segwit breaks the sqrt optimization for covert asicboost.

So while sounding positive I believe this issue is stating an explicit intent to do exactly the opposite.

[hoffmabc]

I don't understand what your issue is then @gmaxwell. If the agreement requires activation of segwit, which in turn accomplishes this, then your entire point is moot. How is this an intent to avoid it. It sounds like there's probably just some confusion of language.

[gmaxwell]

Because this states that " This is outside the scope of the SegWit2x charter". This sounds like it is reinterperting the charter to say that it should implement an asicboost compatible HF segwit; rather than segwit.

If the intent is to not be incompatible with asicboost no separate "outside the scope" action is required, all that really needs be done is to activate segwit.

I guess we should be happy to see the unequivocal confirmation that Jeff Garzik's actions are being directed in private by Jihan Wu; so thanks for that.

[JaredR26]

So to be clear about what you've written between the lines: you have decided that impeding covert asicboost would be a violation of the "SegWit2x charter" and so you will assure that covert asicboost continues to function in your modified version of segwit and HF as people have been alleging you would do?

That isn't what he said. He meant anything extra that we would like to add. As I understand it from your email, segwit only makes covert asicboost more difficult, and doesn't affect overt asicboost. Jihan appears to be signaling a willingness to work together to block all forms completely as an olive branch.

I don't think anyone is suggesting segwit should change in any way except discussing signaling and activation, and fulfilling the 2mb lock-in.

[JaredR26]

Because this states that " This is outside the scope of the SegWit2x charter". This sounds like it is reinterperting the charter to say that it should implement an asicboost compatible HF segwit; rather than segwit.

If that is indeed the case, I would imagine the charter to be dead in the water. Shit, I'd be completely opposed (not that that counts for much but I do love the compromise). Give some time for him to clarify that this isn't the case?

[jgarzik]

When working group members raise an issue, we want to record the issue and make sure it is published, and not forgotten.

At the same time, the 1st priority is a crystal-clear focus on segwit2x WG delivering per agreement "segwit AND 2M" All other issues by default sorted into the 2nd priority bin, and might be unreviewed, flawed, or simply unable to attain widespread agreement.

Standard point of process: Being open and transparent about issues raised within the WG.

[h0jeZvgoxFepBQ2C]

@jgarzik Does this mean the prevention of covert asicboost is not included (which segwit in its current form prevents by default) in the agreement?

And therefore has to be removed (because you have to remove it by purpose if it should not be part of the agreement)?

PLEASE PROVIDE A CLEAR ANSWER TO THIS QUESTION OR THIS WHOLE AGREEMENT-PROCESS IS DEAD RIGHT NOW

[jgarzik]

@lichtamberg No; again, it's a point of process: This detail was not discussed in any way, positive or negative, inclusive or exclusive. Anything that was not discussed is, by definition, A New Issue To Raise And Discuss. Which is what was done here.

[hmsln]

@jgarzik Can you confirm to us that the parts of the current Segwit implementation that might hamper/prevent Asicboost will stay the same in whatever implementation is finally released by the Working Group?

[cdelargy]

To echo @hoffmabc:

the agreement should be amended to include the banning of ASICBOOST

Such an amendment would do a great deal to increase the legitimacy of this effort.

[JaredR26]

fuck this shit, I'm out of this nonsense and I encourage everyone to leave this bullshit.

He said it wasn't discussed and therefore not going to happen; I.e., segwit won't change on that point (based on any plans discussed to date) and therefore will still block asicboost as it does today.

@jgarzik your words are implying that there's a possibility segwit may be changed to ALLOW covert asicboost. Maybe it would be fair to say "no discussions on that point have happened to date, and there are no plans or intentions to modify this aspect of segwit." That'd allow you to remain as neutral as possible without implying that such a thing was intended, discussed or requested by any party. Correct?

[petertodd]

@jgarzik Along the lines of of @gmaxwell's points, who is paying you to do this work?

[Cobra-Bitcoin]

@jgarzik I don't understand why you have to use such weasel words and be so pedantic. Just answer @hmsln's question with a simple yes/no answer. This isn't difficult. If you keep being suspicious, people will assume that you're going to alter Segwit in strange ways to preserve Asicboost for the financial benefit of a certain person...

[paOol]

@petertodd why does that matter if we can see the code ?

are you going to reject an asicboost patch depending on who is funding it?

[Saicere]

My understanding is that SegWit2x will activate the existing Segwit deployment in BIP141, which already disables covert ASICBOOST. When you talk about addressing it, are you referring to overt ASICBOOST?

[JaredR26]

@jgarzik Along the lines of of @gmaxwell's points, who is paying you to do this work?

To be honest, if it doesn't matter that many core developers are working for or involved with Blockstream (one of the major frequent accusations from the pro-BU crowd), why would it matter who was paying him?

If you're going to use the same tactics as the other side does we're never going to get anywhere because everyone opposed is going to demand the same thing of you, and equally distrust your answers.

I would assume that the WG has an agreement to split the costs between the participants equally or proportionally; It wouldn't seem fair or right for only one side to cover any costs, if @jgarzik is even getting paid.

[jgarzik]

The WG agreed to "segwit AND 2M", and that what we are focused on delivering.

To the extent that current segwit disables asicboost, or not, that remains unchanged and unmodified.

This issue asks the question, therefore, do additional changes need to be added, to further ban/disable/render inert asicboost.

[Saicere]

To the extent that current segwit disables asicboost, or not, that remains unchanged and unmodified.

Thank you. That is all everyone wanted to know.

[Vaesper]

@jgarzik thank you for responding in unambiguous terms.

[hmsln]

@jgarzik Thanks, this is the clear answer we were waiting for.

[JaredR26]

@jgarzik I don't understand why you have to use such weasel words and be so pedantic. Just answer @hmsln's question with a simple yes/no answer. This isn't difficult. If you keep being suspicious, people will assume that you're going to alter Segwit in strange ways to preserve Asicboost for the financial benefit of a certain person...

I don't think he's doing it intentionally which is why we keep pressing the issue. I think he's trying to remain as neutral as possible and not get dragged down into the craphole.

Yep, finally he answered. Thanks @jgarzik

[JaredR26]

It's a known issue, and it's already solved in the floated version of Segwit.

Not in all forms(overt), and not in absolute terms.

I genuinely think this was Jihan's (+others?) attempt to extend an olive branch. The distrust runs so deep that it came across as a potential sneaky trojan horse. :( Not that there isn't good reason for concern, just highlights how critical clarity is with all wording and communication on such a contentious issue

[hoffmabc]

I'm curious who's dime you're on @petertodd while you're participating here. All associated companies have agreed to dedicate resources and time to this work group independently.

[Cobra-Bitcoin]

To the extent that current segwit disables asicboost, or not, that remains unchanged and unmodified.

Cheers mate. I've archived this. Good to know you're still committed to blocking Asicboost. There will be hell to pay if you ever backtrack on this.

[petertodd]

@hoffmabc No one - I'm not receiving any funding to do Bitcoin Core work at all right now.

[Vaesper]

@petertodd how about btc1 work, clearly this is not Bitcoin Core?

[hoffmabc]

It's not really relevant at all @petertodd, it was rhetorical. The code is open source. Go ahead and review any work products you'd like.

[gmaxwell]

To the extent that current segwit disables asicboost, or not, that remains unchanged and unmodified.

Good, but then I'm confused why PR#6 is still open which breaks compatibility with segwit. Is it the intention to be compatible or not?

The blog post you made the other day seemed to be unambiguously saying compatibility but the code continues to say another (incompatible).

[JohanShandor]

Just to clarify, since the DCG2017 agreement is not specific but does provide what SegWit code will be based upon:

"We agree to immediately support the following parallel upgrades to the bitcoin protocol, which will be deployed simultaneously and based on the original Segwit2Mb proposal:"

Under the Segwit2Mb proposal by Sergio:

"Segwit2Mb combines segwit as it is today in Bitcoin 0.14+ with a 2MB block size hard-fork activated ONLY if segwit activates (95% of miners signaling), but at a fixed future date."

So, any proposed SegWit code within this DCG2017 agreement code must be in accordance with the Bitcoin 0.14's version of SegWit. In this context, COVERT ASICBoost is disabled by default and the terms of the signed 2017 agreement, and the only issue that should be discussed is disabling OVERT ASICBoost. Using SegWit code that is not consistent with the Bitcoin 0.14 version of SegWit is outside of the scope of the signed agreement.

[earonesty]

@jgarzik In your opinion, is the "COOP" proposal good enough to satisfy the WG? At least some devs and users are on board with this one: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-May/014460.html. I'd rather see most of the spoonet code in a hard fork, it was at least tested a bit and a lot of the ideas render it safer. It would be cool if the first hard fork of bitcoin ever used the best available practices.

[wds4]

If any suspicion remains within the community that SegWit2X will preserve ASICBOOST, then more members of the community will rally behind the UASF, i.e. BIP148 which could disable ASICBOOST as early as August 1. If SegWit2X disables ASICBOOST, it will probably take longer than BIP148 to do so. So Jihan's olive branch could be an attempt to extend his ASICBOOST advantage for as long as he can, by reducing support for BIP148. And you can't blame him for doing what is in his economic self interest (even if he is being disingenuous about his reasons). But you also can't blame the community for being highly suspicious that the (perhaps unspoken?) agreement underlying DCG2017 is for Jihan to support SegWit, and in exchange, he somehow at the end of the day ends up keeping his ASICBOOST advantage. If the WG wants community support, it needs to be made overwhelmingly, unambiguously clear that this is not the case. If Jeff can convince Greg and Todd ... I think that will be good enough for everybody else.

[hoffmabc]

I'm really curious if anyone can point me to the proof that ASICBOOST is being actively taken advantage of. I think this issue has been fairly crystal clear that the intention is to eliminate the ASICBOOST advantage entirely. Do we need a pact signed in blood?

[JaredR26]

So Jihan's olive branch could be an attempt to extend his ASICBOOST advantage for as long as he can, by reducing support for BIP148.

Have u heard? The moon landing was faked by the CIA and hollywood!

I mean, it is POSSIBLE.

But you also can't blame the community for being highly suspicious that the (perhaps unspoken?) agreement underlying DCG2017 is for Jihan to support SegWit

I don't blame them for being suspicious just like I don't blame Jihan (+others) for being suspicious that the segwit2mb could wind up being segwit and no HF.

But the suspicions on either side help no one when we are trying to compromise. And I think, finally, the majority across every subgroup realizes that the need to compromise is approaching criticality.

he somehow at the end of the day ends up keeping his ASICBOOST advantage. If the WG wants community support, it needs to be made overwhelmingly, unambiguously clear that this is not the case. If Jeff can convince Greg and Todd ... I think that will be good enough for everybody else.

110% agreed. The WG and in particular Jihan+co should commit 100% unequivocally to the goal of lauching segwit soon and blocking asicboost forever, with no word games or tricks. In exchange, the primary developers from core should commit 100% to attempting to deliver the smoothest 2mb hardfork possible either 6 months later or as soon as is feasibly safe, with no shenanigans or trickery.

Maybe after we get through this everyone can be friends again. Maybe.

[hoffmabc]

I don't blame them for being suspicious just like I don't blame Jihan (+others) for being suspicious that the segwit2mb could wind up being segwit and no HF.

To be honest after several years of this back and forth I fully expect this is the outcome of this agreement. I don't see really any incentive in anyone fulfilling a HF block size increase once they get the segwit activate that they want.

Maybe after we get through this everyone can be friends again. Maybe.

I wouldn't have shown my face at the Blockstream event at Consensus if I didn't respect the hard work everyone is doing and the intelligent folks working on all of this with Bitcoin's best interest at heart. Let's remain focused on delivering what all these parties involved and not let this devolve into yet another place to moan about the scaling struggles.

[jgarzik]

Please let's keep the discussion focused, specific and technical.

Are there additional changes, over and above what is currently in 0.14.1, that should be considered?

[hoffmabc]

It almost makes me think this issue is moot unless there's something we need to do beyond activating segway to solve the ASICBOOST issue.

[cdelargy]

Blocks which contain witness commitments effectively disable covert (but not overt) ASICBOOST, however blocks which do not contain witness data can be constructed with an advantage using covert and overt ASICBOOST.

@gmaxwell describes a mechanism (not present in Bitcoin 0.14+) to require blocks to contain a witness commitment that would inhibit covert ASICBOOST use as well as a change to nVersion validation that would also disable overt ASICBOOST at https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-April/013996.html

[drunkenclam]

@jgarzik: Core just tagged 0.14.2 RC1. Maybe consider rebasing your PR's when it's final.

[chjj]

Blocks which contain witness commitments effectively disable covert (but not overt) ASICBOOST

Given the title of this thread, and since nobody explicitly pointed it out, I'd like to mention that merkle commitments do not prevent covert asicboost. Greg's claim in his proposal is that the merkle optimization is what makes asicboost so worthwhile. That involves creating a bunch of left & right subtrees by swapping the leaves, and hashing the roots of different permutations of subtrees together to efficiently find collisions.

• Does a commitment disable the optimization Greg is talking about? Yes.
• Does that hurt perf of generating merkle collisions? Slightly, since now it's an extra dozen or so hashes per attempt to hash up the left side of the tree for coinbase grinding (not too hard for ASICs).
• Does it hurt profits enough to dissuade usage of covert asicboost? Probably not. If that's the case, why did the patent also suggest the coinbase-grinding method?

Disabling only one optimization inside of a patented optimization doesn't do much to prevent centralization here.

I'm really curious if anyone can point me to the proof that ASICBOOST is being actively taken advantage of.

I'd rather hear whether the optimization for calculating the merkle collisions is being taken advantage of. So far there's been no evidence other than some claims of reverse engineering an ASIC (guessing that's not easy to do without tens of millions of dollars of highly sophisticated equipment handy, so I'd treat that claim with skepticism).

Doing this would involve creating a custom mining protocol which serves all leaves to the miner (stratum only serves the steps up the left side of the tree, it also doesn't let you submit back a different merkle tree), writing a custom driver which handles this payload and resubmits a new merkle tree when the block is found, and then implementation of the collision algorithm in hardware.

I imagine it's also awkward to do since it's already not that easy to just swap stuff around in a bitcoin merkle tree, given that transactions have to be sorted in dependency order (and I see nested transaction chains get mined all the time that are 25 txs deep). I guess it seems doable if you're mining txs without any dependents.

Not sure, decide for yourself based on all the hurdles I guess. This seems like a very roundabout way of doing it when you could just grind the coinbase without all the technical overhead.

---

This issue asks the question, therefore, do additional changes need to be added, to further ban/disable/render inert asicboost.

I think we should just require the wtxid commitment on every block, just like the ML proposal says. It doesn't actually prevent asicboost, but we should do it anyway. This was a throwaway political move by Greg. We should include it to stop the drama and move on from this.

[kek-coin]

This was a throwaway political move by Greg. We should include it to stop the drama and move on from this.

Under the BTC1 principles I will assume you are not intentionally stirring up drama, merely doing it out of ignorance towards your own actions, so I will just point out that throwing around accusations of bad faith does not comply with the guidelines.

[JaredR26]

Given the title of this thread, and since nobody explicitly pointed it out, I'd like to mention that merkle commitments do not prevent covert asicboost. Greg's claim in his proposal is that the merkle optimization is what makes asicboost so worthwhile.

This is true, but it does not represent a good argument why asicboost should not be disabled. In fact, it makes it all the more important that coinbase-grinding asicboost gets disabled. Since this would require a hardfork and client upgrade, the right time to do this is when the +2mb hardfork is added. Asicboost may help individual miners in some ways but it provides effectively no benefits for Bitcoin with the potential to cause not only problems for segwit, but many unpredictable problems in the future due to its complex and very narrow uses.

Since witness commitments are optional under segwit, segwit alone does not block covert asicboost. Further, the wtx commitment requirement proposal doesn't address asicboost coinbase string grinding as you mentioned, it only prevents asicboost from discouraging segwit's activation.

Side note: It also strikes me that since BIP141 was supposed to have its witness transaction data moved out of the coinbase string in a future hardfork, the +2mb hardfork might be a good time to do exactly that.

So far there's been no evidence other than some claims of reverse engineering an ASIC (guessing that's not easy to do without tens of millions of dollars of highly sophisticated equipment handy, so I'd treat that claim with skepticism).

It has been confirmed by Bitmain and independently that both the S7 chips and the S9 chips supported midstate inputs. Bitmain also stated that they tested midstate mining on the testnet, but found it to be impractical on the "production network". FYI, It isn't particularly hard to verify on the asics themselves since the midstate input has to be represented on the pinout charts, which can be verified against the inputs on the physical chip. Moreover, the chinese ASICBOOST patent was submitted within days of the S7 chip's verification, which appears to be the first one to support midstate inputs, and this was prior to the announcement/publication of ASICBOOST as we know it (though the U.S. patents were filed and would have been searchable by that time).

I'd rather hear whether the optimization for calculating the merkle collisions is being taken advantage of.

I put substantial time into looking into this and was unable to find any evidence showing that the merkle tree swap approach is being done. While it would be impossible to tell for certain from any given block, merkle tree swaps could be detected statistically so long as the general initial ordering of transactions is in sorted descending by tx fee per byte. All major pools I examined do follow that same initial general ordering. A merkle-swapping process as described by Greg could be detected by identifying blocks where the the merkle sub-trees are primarily in order at the lowest levels, but increasingly out of order at the top of the tree, and this is what I primarily searched for. I failed to find any proof or even any suspicious indicators.

Coinbase grinding on the other hand could possibly be in use. If identification of its use were possible, it would (probably) involve deep manual analysis of the coinbase strings suspected looking for anomalies and changes over time, possibly also on testnet given Bitmain's statements.

Doing this would involve creating a custom mining protocol which serves all leaves to the miner (stratum only serves the steps up the left side of the tree, it also doesn't let you submit back a different merkle tree), writing a custom driver which handles this payload and resubmits a new merkle tree when the block is found, and then implementation of the collision algorithm in hardware.

These are largely flat costs. The collision algorithm wouldn't need to be implemented on hardware, though it might need to be for coinbase grinding to be effective. Writing a custom driver is already something a chip manufacturer has to do.

Not sure, decide for yourself based on all the hurdles I guess. This seems like a very roundabout way of doing it when you could just grind the coinbase without all the technical overhead.

This is what we should be blocking. Greg suggested using the rightmost bits of the timestamp as a hash of the first 64 bytes, but this (and any!) approach would completely upend the chip design market, as the optimal chip circuitry post-activation will be substantially different from the optimal chip circuitry pre-activation. The best approach would be to discuss with the miner's chip design teams to ensure that the change does not have a drastic impact on any of the existing chips or chips in the pipeline. I'm not sure how exactly to do this - Perhaps if the merkle root hash was shifted 12 bytes to the right (part of the witness commitment hash?) so that the merkle root was split with 16 bytes on each side. This would still require version checking. Another idea would be to split each hash(merkle, prevblock, witness) equally with 16 bytes in each of chunk, but simultaneously give miners a much much larger nonce space (making extranonce be no longer required).

If occurs to me that if the block header were to go over 128 bytes, every major miner would object and refuse the fork, as I would be surprised if any chips in use today don't explicitly optimize for two-chunk input. This means asics may always present a perverse incentive that may sometimes go against the best technical solutions.

@kek-coin is right though, it isn't helpful to accuse Greg of bad faith motivations; From a technical perspective, ASICBOOST can and should be solved to prevent large miners from edging out smaller miners more than they already do, and also to prevent its perverse incentives from interfering with future progress.

[jgarzik]

Revisiting the basic issue - do we have a usable patch or at least method, beyond that which already comes for free with SegWit?

[theslagter]

Why do you ask?

[mpatc]

It seems everyone here is taking for granted that ASICBOOST is patented, which is a wrong assumption. It is patent-pending. No patent has been awarded, anywhere. Abstract ideas are not patentable, and to me at least, that's what ASICBOOST is, so it's not clear that a patent ever will be awarded.

Even if the patent was granted, how would the authors enforce it? It seems like a lot time spent on what isn't an issue now, and only may become a relatively minor, short-term issue in the future, since patents are of a limited duration. A waste of time.

Further, I'm not quite clear on how this 'exploit' is different from an optimization, aside from the fact that 2 guys are trying to claim a monopoly over its use. It sets a troubling precedent, in that anyone can find a great way to improve bitcoin, file a patent, and cause everyone in the community to scramble engineer a way to avoid using that improvement, which seems a bit perverse to me.

Aside from the patent issue, are there valid reasons why ASICBOOST needs to be addressed? Aside from the fact that it can be used 'covertly', which I'm not sure is even an issue. Pool operators, like exchanges, require trust, and are therefore outside the scope of a trustless protocol like bitcoin. Anyone who doesn't trust their pool operator or chip manufacturer will remain free to start their own pool or make their own ASIC.

[JaredR26]

Asicboost provides no benefits for the ecosytem, and can provide an adverse incentive to discourage improvements to the protocol that are financially motivated rather than technically sound.

Specifically, as an "optimization" it only works under a specific set of constraints. Those constraints are ones that just happen to be in place (a minimal number of merkle root hash bits fall in the second 64-byte section) with the existing block header structure; They are not actually a part of Bitcoin's specifications, are not intended, do not provide benefits for Bitcoin, and are not guaranteed by future versions of the protocol.

There's no reason to allow it to cause these problems, regardless of patents or anything else. The best path forward would be to coordinate with the manufacturers so that the asicboost block implementation has the smallest negative impact possible on their existing chip designs, while still making midstates infeasible for use. Most likely this means rearranging the bytes of the block header in the hardfork or in a subsequent hardfork.

[ghost]

segwit2x will fail if you implement that, miners will back out