[Consensus] Problems with difficulty drop consensus rule

[sanket1729]

The difficulty drop consensus rule creates an interesting scenario as follows:

Consider the following simple scenario: 1) Miners wait for difficulty to down significantly low (they can mine BTC meanwhile) 2) They mine 2016 blocks very fast. 3) Back to step 1

With a stable hash rate (stable referring to amount of hash power such that 2016 blocks are mined in 2 weeks), miners can wait for some periods of 12 hours and then mine. Assuming stable hash rate, some basic calculations I did showed that it is possible mine 2016 blocks in 6.3 days while maintaining the same difficulty across periods. I think the problem is even worse because the potential hash power(BTC + BCH) which can mine bitcoin cash is atleast 5 times higher than stable hash power.

I think such a faster rate of coin generation is problematic as it would lead to faster halving(less than 2 years), which leads to faster rise of transaction fees. This leads to the very problem for which bitcoin cash was created to solve. Furthermore, faster generation of coins will also lead faster into unexplored lands of comparable transaction fees overcome block reward incentive.

This is a big problem because if 95% hash power can meetup at a place, it is quite easy to collude for this profit as it fits perfectly in the consensus rules for Bitcoin Cash. Miners are also not necessarily bound to Bitcoin Cash, since they have main bitcoin to fall back on.

A simple solution would be remove the rule by adding a Soft Fork.

I am sure that that the scenario has been considered before designing REQ-7 of UAHF spec, but I feel these consequences do not justify the rationale.

I intended this for a mailing list discussion, but I can't seem to references to it in bitcoinabc.org or bitcoincash.org

[ThomasdenH]

Since we can expect difficulty jumps, at least in the near future, I think it is best to move to a algorithm like the one on Ethereum. That will allow for hashrate drops but also pointless to manipulate.

[sanket1729]

To add to this, Miners can easily manipulate timestamps in history to produce fake difficulty drops while continously mining blocks.

As an example,

1) They create blocks 2016 blocks with timestamps at t+1 sec,... t +2016 secs. 2) Difficulty bumps up 4 times. (difficulty can bump up maximum 4 times) 3) Miners mine blocks at t + 2016 sec + 2 *k hours(k=1,2,3,4,5,6 ...) till the difficulty goes significantly low again. They have gained this time in "real world" by manipulating timestamps in "blockchain time".

Fast Mining is also highlighted by the last period of 2016 blocks which should have lasted 2 weeks but was very short(3 days?)

[1BitcoinOrBust]

Removing EDA entirely and reducing the retarget interval to something like 1, 6 or 12 blocks (down from 2016) should mitigate the stability issue.

[zanza321]

Seems like this could be addressed by included an EDA adjustment in an upward direction also. So sudden influx of hashrate will trigger increase in EDA also. This way the target 2 weeks = 2016 blocks will hold steady in the long term. With the current scenario, the EDA is activating outside of an "emergency," so it seems to not be the original intention.

[ThomasdenH]

The best solution is to have one calculation that just works instead of patching current limitations. I think Bitcoin Cash is able to HF to make these changes, so it's best to do it right. Making difficulty adjustments gradual instead of being dependent on a 2016 block period seems like the simplest and most elegant solution. By adding more EDA's you risk creating an even more severe oscillation or other unpredictable behaviour.

[ser81]

Miners follow the profit and the profitability depends on the price and difficulty. The price can change very quickly (global crypto-market attracts big money as we know :) ). In order to balance the profitability quickly the difficulty has to ajust quickly as well. The solution has to include the reduction of a target interwal to a small amount of blocks (down from 2016). Small block target number means stable system. I am new to GitHub and have only registered to participate in your discussion. How will the decision be made whether to make the appropriate adjustment or not?

[DesWurstes]

How about a better difficulty adjustment system? (eg. DarkGravityWave or digishield)

[mpatc]

I think the adjustment system should reflect the old as much as possible, so it retains as many of the same properties as (pre-fullblock era) bitcoin as possible. But it should be updated.

The 2016-block average has seemed to work well as a basis to set target, and minimized opportunities to game the system, however it is a little too coarse. A better way to do it, while keeping the same properties, would be retain the 2016 block average, but adjust the target on a more regular basis, perhaps every 144 blocks.

This would make the network a little more responsive to changes in hashpower, while retaining as many of the good properties of the original difficulty adjustment as possible.

(This would be post-HF to remove EDA, once the price/hashpower disequilibrium quiets down.)

[TierNolan]

The purpose of the EDA is to prevent the chain going dead when miners switch away. Even with a 144 block update, the system would react very slowly to a sudden loss of hashing power.

The EDA itself takes half a day.

The EDA messes up the actually block rate calculation too.

The calculation assumes constant difficulty since the last update.

https://github.com/Bitcoin-ABC/bitcoin-abc/blob/a46e63c2bb3d189beb846a8d34a0d33fd563fa7b/src/pow.cpp#L13

In summary:

diff = old_diff * (2 weeks) / (interval)

but old diff is the difficulty after EDA.

A better scheme would be to calculate the total POW over the adjustment period and the total time over the adjustment period.

time_interval = previous.time - previous_2016.time
work_done = previous.chain_work - previous_2016.chain_work
Target = ((time_interval) * (2^256 / (work done))) / (2 weeks in seconds)

(You would also need to limit the changes to X4 and /4)

I think EDA should be kept for emergency changes.

If at least one chain has step changes and miners hyper transfer, then it doesn't seem like there is much of a reasonable way to keep things stable.

[mpatc]

I think if the EDA was kept, should be added to stop the flood blocks that occur whenever it kicks in and miners switch their hash.

A way to do this would be something like

const CBlockIndex *pindex24 = pindexPrev->GetAncestor(nHeight - 25);
    assert(pindex24);
int64_t mtp24blocks =
        pindexPrev->GetMedianTimePast() - pindex24->GetMedianTimePast();
if (mtp24blocks > 3600) {
        return tooQuickMakeItHarder;
    }

to check from whenever the EDA is triggered, until the next regular difficulty adjustment.

This would check if the last 24 blocks happened within an hour, and then do something (increase difficulty by an amount, perhaps a 'reset' by recalculating the last 2016 blocks and using that.

Edit: I'd probably make the check smaller, something like 12 or 18 blocks/hr. But you get the idea.

[sanket1729]

I did some simulations. It showed that you could mine 2016 blocks in about 2 days maintaining difficulty across periods under the assumption that old_diff is difficulty before main adjustment period.

I think the change you recommend should be added, but that won't solve the issue

[mpatc]

A more 'draconian' implementation might solve it? For example, whenever EDA is triggered, adjust difficulty if last block was sooner than 1 hr from 6 block, until next regular adjustment period. It would make it harder to game. This would make the kind of yo-yo effect we're seeing less likely.

[ThomasdenH]

How about adjusting the difficulty every block based on the previous 2016 block average? That way a sudden change in hash power should produce a gradual change in difficulty. It also means every 2016 blocks the difficulty is the same as it would be in the old system.

[mpatc]

@ThomasdenH that approach seems riskier, since it means miners would constantly be mining at a different difficulty level, so this could:

• create more orphan blocks, since it'll be more likely different nodes mine at different diffs.
• change the behavior of on/off decisions of miners based on profitability. (Could be good or bad)
• incentivize block timestamp manipulation that will help profitability.
• open up an DDOS attack surface to get miners mining at different diffs.

I could be wrong about any or all of those, however a more conservative approach just seems less likely to screw with market forces I don't think anyone fully understands. A big chance of unintended consequences, in other words.

[TierNolan]

create more orphan blocks, since it'll be more likely different nodes mine at different diffs.

The difficulty of a block still depends on the parent (and the rest of the chain).

There could be problems with forks that are more than 1 block long, but those are rare anyway.

change the behavior of on/off decisions of miners based on profitability. (Could be good or bad)

I think good, since they would switch to and from the chain more gradually.

incentivize block timestamp manipulation that will help profitability

I think this balances out.

If it is a problem MTP could be used instead.

open up an DDOS attack surface to get miners mining at different diffs.

The total chain work is what matters anyway. Two miners mining on the same parent will have the same difficulty anyway.

Again using MTP would help here.

[ThomasdenH]

Well, every block has a unique and well-defined history. So there will always be consensus about whether a particular block has the correct difficulty. There is no extra chance of orphaning since the validity of a block is independent of external factors.

The profitability could very well change, but it would only become more gradual. For example, currently an oscillating effect could occur:

• Period 1: High difficulty, low profitability.
• After 2016 blocks a downward adjustment occurs.
• Period 2: Low difficulty, higher profitability, more miners.
• After 2016 blocks an upward adjustment occurs.
• Repeat.

Calculating every block would mitigate this and make profitability more predictable.

incentivize block timestamp manipulation that will help profitability.

Possibly.

There isn't any difference in the DDOS attack surface since every miner with the same history agrees about the difficulty of the next block. If they aren't mining on top of the same block they will produce blocks incompatible with the other chain anyway.

[mpatc]

Every block has a well defined history, however each node makes it's own calculation each time the target is changed. Assuming the chance of error is > 0, having each node calculate difficulty each block increaces the chance of error. Each block is a new opportunity for a new error, instead of every 2 weeks.

[ThomasdenH]

The chance of error is exactly 0 because the calculation is only based on information in previous blocks.

[stacksmith]

The timestamp manipulation could also lead to an attack against the emergency mode kicking in, as witnessed in the blockchain (lost the link...) I believe someone pushed the timestamp to an earlier position, so that the difference between 6 relevant blocks was 11 hours and 59 minutes.

[deadalnix]

This problem is not really specific to this adjustment algorithm. It happens for all coins that share the same mining infrastructure and is very well known for GPU mined coins. It did not happen on Bitcoin before because Bitcoin has ASIC and no other SHA256 coins come even close to it.

This is what has changed. BCC is big enough to create that mess. The wild swing will stop either when both chains HF to some smoother difficulty adjustment, or one of the chain dies.

[ghost]

I agree with Deadalnix. The EDA is not really broken, but it's causing this issue because of the situation and relationship BTC and BCH share. Because at one point the profitability for BCH is higher than that of BTC's it makes perfect sense for miners to start mining BCH and liquidate for BTC. However, this drives the price down obviously.

The issue is the wild swing of the network hashrate from one network to the other. Bitcoin Cash needs to be able to scale the difficulty back up when needed for cases like these.

[ghost]

Also, Deadalnix I apologize for the message from Reddit, I should have checked here first. You obviously have a grasp on the situation.

Has anyone generated any ideas toward a solution? Is a hard fork absolutely necessary?

[sanket1729]

@deadalnix ,

This problem is not really specific to this adjustment algorithm.

I disagree. With this specific algorithm, faster inflation rate would lead to faster halvings which would lead to higher fees.

The wild swing will stop either when both chains HF to some smoother difficulty adjustment, or one of the chain dies.

As mentioned above, this is way bigger issue for BCH than for BTC. If both chains are to survive, BCH would be highly inflationary and this should be fixed. As mentioned in the issue OP, a soft fork to remove EDA can be deployed. This should atleast fix the faster coin generation issue, but could the chances of BCH chain dying.

[brian2001]

@deadalnix

Thanks a lot for your fantastic work on Bitcoin ABC / Cash. In my view this was needed to save the real Bitcoin. As well EDA was necessary to make sure BCH survive.

EDA indeed seems to lead to a little bit of inflation and eventually inflation manipulation.

What do you think would be the best way to mitigate this ? more frequent difficulty adjustments (every 128 blocks rather than every 2016 blocks ?) or/and maybe symetric EDA ?

@sanket1729

1) EDA is very important and clever because it ensures that the chain is saved even when the hashrate drops, you probably need to acknowledge that.

2) I disagree with what you say

You say that : 1> "faster inflation lead to faster halving " yes as always, for Bitcoin and other cryptos emission curves. you probably mean that the EDA is asymmetric and faster inflation periods are not symmetrically compensated with lower inflation periods of similar length.

2> "faster halvings which would lead to higher fees" faster halving doesn't lead to higher fees. Fee is voluntary, not miner driven (Higher fee is triggered by full blocks, nothing else, should come when global adoption arrive)

My take I'm against EDA removal, but I agree we need to work to fix the inflation manipulation byproduct.

[ghost]

Remove EDA, it is not necessary anymore. It was necessary in the early days of BCH to ensure the network could survive the UAHF, and it has. Unless there is another soft fork option available, I think EDA is unnecessary now.

[mpatc]

There have been 25 blocks in the last hour, there really should be some kind of brake to prevent this kind of thing and readjust both ways. We're seeing the fact that EDA is one way being gamed by miners as we speak.

[jzcjca00]

A huge amount of time has been invested on developing difficulty adjustment algorithms for the altcoins that respond quickly to changes in hashpower without over-correcting and inducing oscillations.

Rather than try to reinvent the wheel, I suggest that we replace both the regular and emergency difficulty adjustment algorithms in Bitcoin Cash with one of these existing algorithms:

• Dark Gravity Well (DGW) -- Used in DASH.
• Digishield V3 -- Used in Zcash.
• Multi-Interval Difficulty Adjustment System (MIDAS).

[zanza321]

Agreed, lets convert some time tested and well researched adjustments from other alts. Trying to keep 10 minute block time average. There are lots of alt-coins out there, lets take advantage of their research and benefit from them.

I would like at least to keep in the EDA function such that, in case of WW3, major real world even that the hash rate crashed 99%+, the chain would still be able to recover without needing to make any changes. Honey Badger!

[jzcjca00]

The three algorithms I listed already have the equivalent of our EDA as an integral part of them, as opposed to being a separate piece of code. They render our EDA code unnecessary, and leaving it in would just open up opportunities for abuse.

[brian2001]

jzcjca00 all this sounds good, is there a BCH foundation or coding team that can officially decide of this ? and push this ? Should we setup a governance model ?

[jzcjca00]

@brian2001, I am just an ordinary Bitcoin fan, who happens to have a degree in Computer Science and 40+ years of experience in IT. I am not a Bitcoin developer. I saw an issue, did some research, stumbled across this discussion, and reported what I have found so far.

I'm guessing that there's some group who discuss the short and long-term road map for Bitcoin Cash, but I don't know where those discussions take place, and I have not been invited to join.

[ftrader]

@jzcjca00 : There is discussion on Bitcoin Cash roadmap is taking place on https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-ml which is open for anyone to to join. Not enough people are aware of this yet.

There is also a thread on the difficulty adjustment problem there, see for example

https://lists.linuxfoundation.org/pipermail/bitcoin-ml/2017-August/000136.html

Some people have been submitting proposals for changes affecting Bitcoin Cash as a whole via the UAHF spec website, which is also fine (e.g. https://github.com/Bitcoin-UAHF/spec/pull/20). Though it's probably more effective to raise the issues directly on the mailing list (bitcoin-ml) because devs from the initial implementations are all on there.

Some ABC devs monitor other forums related to issues/support as well, such as BTCfork slack, where we've also been having some discussion on the EDA topic. Again, that's also open to anyone to join ( https://btcforks.signup.team/ ).

[brian2001]

@jzcjca00

You seems to be able to add a lot of value to BCH, is there a way to have an official BCH dev team (similar to BTC but without the blockstream compromission ?)

I'm as well a Bitcoin Crypto-Currency fan, with IT-Finance background for around 20 years.

[maaku]

Why not switch to merge mining? Then miners can secure both chains without any trouble with giant difficulty swings.

[deadalnix]

Good idea @maaku . Please raise a BIP and see if that's possible for core to adopt.

[maaku]

No changes to Bitcoin Core are required for Bitcoin ABC to adopt merge mining.

[deadalnix]

I think bitcoin core will have to adapt to be merge mined within the BCC chain. I thank you in advance for proposing it and salute your effort. I hope the core team will do the right thing.

[maaku]

Oh come on. You know very well Bitcoin Core isn't going to adopt any hard forks anytime soon, whereas Bitcoin ABC seems to have multiple threads on its mailing list and issue tracker seriously considering further hard forks in the not so distant future. This is one of them.

I was genuinely offering a suggestion that I thought would help solve your problem and make your chain long-lasting without causing any further strife or aggravation here or in the wider crypto currency community.

I was even going to offer to help write it if it was something this community wanted. But I don't think I'm welcome here, so goodbye and good luck.

[DesWurstes]

Can't we just switch to DigiShield or DarkGravWave? They're tested.

You can find more info about them here: https://github.com/zcash/zcash/issues/147#issuecomment-200331246 (Take a look!)

and this: https://github.com/zcash/zcash/issues/147#issuecomment-221534222 (MUST take a look!)

Can someone post these to mailing list?

[TierNolan]

Has there been any altcoins which try to lock the coins per minute given out and decouple that from difficulty?

The coin payout per block and the difficulty set the profitability. They could both be tweaked independently.

[h0jeZvgoxFepBQ2C]

Why don't we just restore the normal core difficulty adjustment?

[zanza321]

@lichtamberg It will require a HF anyway to make a change. Original BTC difficulty algo is kind of "dumb," meaning it was made a long time ago. I think it can be improved on quite a bit. Have a true "emergency" difficulty adjustment is a good feature to have. Think if certain governments decided to ban Bitcoin mining, or WW3, etc..., this makes it resilient. The problem we see now with BCC is its being "gamed" and in a long term time period, there is much more than 2016 blocks per 2 week period (1 block per 10 minutes) because of this.

[h0jeZvgoxFepBQ2C]

Hm but are these long difficulty adjustment times not by purpose? See this: https://petertodd.org/assets/commitments/52ccc4802bd563076cbd25ec4c1ba88152098cb6aa356ba644c9e79a24182da5.txt

[is55555]

I really don't see how this is controversial at all. The original PoW adjustment is just not very good anyway and it's there simply because a HF is required to change it. If you're hard-forking, then definitely adopt a better targeting system.

On top of that - but this is more debatable - a PoW change would be a good idea, so BCH doesn't have to compete with the strongest PoW chain in the world.

[DesWurstes]

If we change the POW, BitMain will never be on BCC side.

[is55555]

And does that make any difference? They are not committing as things stand. I think that ship has sailed, but as I said this is more debatable. What's sure is that a targeting change is quite urgent. Might also fix other issues if you're hard-forking anyway and a PoW change is something to consider.

[DesWurstes]

Why not just replace the difficulty system with a more modern one? They're simpler (than changing the POW), tested, and don't need to be debated.

[is55555]

Yes, definitely this has to be done. But since a HF is going to happen, it would be a waste not to consider other improvements.

[zanza321]

The only person who spoke of POW change, is LukeJR who is hostile to miners. BCC is friendly to miners, POW change is not urgent issue to consider at this point in time, maybe in the future.

Next HF, hopefully it will be soon ? Will include Difficutly adjustment optimization, and I hope Malleability Fix, just so BCC can be on the same track for LN as BTC (which leaves less reason for people and miners to stick to BTC).

[DesWurstes]

I believe we should choose a difficulty adjustment system first. The most stable one looks like DigiShield? (Stable blue line = stable algorithm) (Implementations: https://github.com/zcash/zcash/issues/147#issuecomment-202741851)