This discusses the start-up and MTP delay problems in Digishield (e.g. Zcash). It works good, but it has two problems that arguably make it not a lot better than an SMA with N=75. "Better" means that for a given level of stability during constant hashrate, it responds faster to sudden increases in hashrate. With MTP and 16%/32% removed, and safe timestamp handling in place, it easily beats SMAs. With N=17 and a tempering factor of 4x in the normal Digishield, an SMA with N=17x4=68 has the same stabilit but slower response.
Problem 1: POW Limits
There is a +16% to -32% "nPowMaxAdjust" in Digishield v3 that caused me to screw up Bitcoin Gold's new difficulty algorithm as I reported here and here. They were going to use Zcash's Digishield v3 code directly, but I recommended a switching to a Simple Moving Average (SMA) with N=30 to let it respond faster. By making it respond faster, a big problem with the +16% / 32% limits was exposed. Further investigation revealed the limits do not help Digishield in any way and double the number of blocks it takes Digishield to reach the correct difficulty after startup from 260 to 500.
If previous difficulty is accidentally low, a hash attack is motivated. If the accidentally low difficulty (which will occur often in faster-responding difficulty algorithms) is lower than average D by more than the 16% limit, then difficulty is limited to increase to the 16% for 1 block, but then it can't rise very fast anymore because the limit is based on the avg D, not the previous D. The 16% limit (which is a actually 19% limit when expressed as 1/(1-0.16)) frequently placed a 0.5% to 1.5% change per block in BTG's previous SMA N=30 algorithm, which I expected to be 2x faster. But as a result of the limit, it was 5x slower. Digishield isn't experiencing a problem because the trigger is never activated. The limits are [sarcastically speaking] good code as long as they are never used.
No timespan limits should be used in difficulty algorithms because it opens up a catastrophic attack I discovered and detail in my timestamp attacks article. Most coins are affected, including BTC, LTC, Dash, and BCH. The attack results in unlimited number of blocks in 3x the difficulty window, but requires enough hashpower to perform a 51% selfish mining attack.
The following is what was intended and should have been coded more directly. If this had been the code, there would not have been any problem. The difference is that the "correct" code below limits the per-block change in difficulty (as I assume everyone expects the limits to mean) instead of limiting the timespan changes, which unexpectedly makes a big difference. Limiting timespan limits the average of the past difficulties, not the previous difficulty. Limiting timespan was inherited from the 4x and 1/4 limits in BTC, but in BTC a limit on timespan is a limit on previous difficulty & avergae of past difficulty because the difficulty is not changing every block.
if (next_D > previous_D/(1-0.16) ) next_D = previous_D/(1-0.16);
if (next_D < previous_D/1.32 ) next_D = previous_D/1.32;
The following compares BTG's actual difficulty to what it would have been if it was a simple SMA with N=30 (as I was expecting). The semi-straight lines are where the limits are being hit. Notice they start out flat (horizontal) where only 0.5% rise per block was being allowed when we desperately needed the full 16% to be allowed. The green peaks would not have gone so high if it were the difficulty.
It causes a problem at startup: Digishield coins take 500 or more blocks to get to the correct difficulty, literally giving them away. This giveaway causes the difficulty to then overshoot, causing delays while it gets back down to the correct level. In the Zcash example below you can see it did not overshoot which is because they had a slow start period where the reward was really small. They did the slow start at least in because they were warned by another coin that difficulty would cause a problem at start up.
To prevent the problem delete all the following code:
if (nActualTimespan < params.MinActualTimespan())
nActualTimespan = params.MinActualTimespan();
if (nActualTimespan > params.MaxActualTimespan())
nActualTimespan = params.MaxActualTimespan();
The 16% / 32% limits are only 1/2 the cause of the startup problem. Even without the limits, the 4x "tempering" in Digishield causes 260 block delays at startup where SMA and LWMA cause 140 and 60 block delays respectively, as shown in the image below which shows a 1000x increase in hashrate for 400 blocks (all the red). Interestingly, these delays are the same for any > 50x hashrate increase.
Problem 2: Delay in responding (MTP)
Most Digishield v3 implementations do not get data from the most recent blocks, but begin the averaging at the MTP, which is typically 6 blocks in the past. This is ostensibly done to prevent timestamp manipulation of the difficulty. However, there are several different methods of dealing with bad timestamps that will not inject this delay.
These charts show hashrate jumps on Digishield coins when difficulty accidentally falls low. But I think there is a distinct excess of oscillations that is being caused by the MTP delay.
Asymmetry
The lack of symmetry (+16 / -32 instead of +16 / -16 limits) has made things worse for Bitcoin Gold. BCH EDA also had a lack of symmetry where it could rise faster than fall which led to oscillations and too many blocks being issued. BTG also has the ugly oscillations and 3% too many blocks are being issued. The oscillations are obvious, but are not causing terrible delays. Although the 16/32 limits are not close to what's needed and meant by "asymmetry", at least the avg solvetime would have been correct if they were symmetrical, if not fewer oscillations.
Useful Tempering
The Digishield v3 code has tempering that only seem to be only slightly better than using SMA with N=68.
Simplifying the math this is:
nActualTimespan = 0.75*params.AveragingWindowTimespan() + 0.25*ActualTimespan
The params.AveragingWindowTimespan() is just a constant N*TargetSolvetime. This equation takes only recent data into account, but it includes some skepticism about changing from the past.
Summary
Digishield v3 code is effectively:
# Digishield v3 difficulty algorithm
N=17
# ST = solvetime
T=<Target Solvetime>
# Step 1: Insert a harmful delay in your difficulty's response.
sumST = [last - first timestamp, N blocks apart, delayed 6 blocks) ]
# Step 2: Do tempering that actually helps
sumST = [ 0.75*T*N + 0.25*sumST ]
# Step 3: Insert a POW limit to do the exact opposite of what you want.
sumST = (1 - 0.16)*T*N if sumST < (1-0.16)*T*N
# Step 4: Make it asymmetrical so that if your POW limits are utilized, it will cause
# oscillations and make your average solvetime too low.
sumST = 1.32*T*N if sumST > 1.32*T*N # disastrous asymmetry waiting to happen
next_D = sum(past N D's) * T / sumST
Once the bad stuff is removed, the resulting Tempered SMA works better than an SMA with N=65.
This discusses the start-up and MTP delay problems in Digishield (e.g. Zcash). It works good, but it has two problems that arguably make it not a lot better than an SMA with N=75. "Better" means that for a given level of stability during constant hashrate, it responds faster to sudden increases in hashrate. With MTP and 16%/32% removed, and safe timestamp handling in place, it easily beats SMAs. With N=17 and a tempering factor of 4x in the normal Digishield, an SMA with N=17x4=68 has the same stabilit but slower response.
Problem 1: POW Limits
There is a +16% to -32% "nPowMaxAdjust" in Digishield v3 that caused me to screw up Bitcoin Gold's new difficulty algorithm as I reported here and here. They were going to use Zcash's Digishield v3 code directly, but I recommended a switching to a Simple Moving Average (SMA) with N=30 to let it respond faster. By making it respond faster, a big problem with the +16% / 32% limits was exposed. Further investigation revealed the limits do not help Digishield in any way and double the number of blocks it takes Digishield to reach the correct difficulty after startup from 260 to 500.
If previous difficulty is accidentally low, a hash attack is motivated. If the accidentally low difficulty (which will occur often in faster-responding difficulty algorithms) is lower than average D by more than the 16% limit, then difficulty is limited to increase to the 16% for 1 block, but then it can't rise very fast anymore because the limit is based on the avg D, not the previous D. The 16% limit (which is a actually 19% limit when expressed as 1/(1-0.16)) frequently placed a 0.5% to 1.5% change per block in BTG's previous SMA N=30 algorithm, which I expected to be 2x faster. But as a result of the limit, it was 5x slower. Digishield isn't experiencing a problem because the trigger is never activated. The limits are [sarcastically speaking] good code as long as they are never used.
No timespan limits should be used in difficulty algorithms because it opens up a catastrophic attack I discovered and detail in my timestamp attacks article. Most coins are affected, including BTC, LTC, Dash, and BCH. The attack results in unlimited number of blocks in 3x the difficulty window, but requires enough hashpower to perform a 51% selfish mining attack.
The following is what was intended and should have been coded more directly. If this had been the code, there would not have been any problem. The difference is that the "correct" code below limits the per-block change in difficulty (as I assume everyone expects the limits to mean) instead of limiting the timespan changes, which unexpectedly makes a big difference. Limiting timespan limits the average of the past difficulties, not the previous difficulty. Limiting timespan was inherited from the 4x and 1/4 limits in BTC, but in BTC a limit on timespan is a limit on previous difficulty & avergae of past difficulty because the difficulty is not changing every block.
The following compares BTG's actual difficulty to what it would have been if it was a simple SMA with N=30 (as I was expecting). The semi-straight lines are where the limits are being hit. Notice they start out flat (horizontal) where only 0.5% rise per block was being allowed when we desperately needed the full 16% to be allowed. The green peaks would not have gone so high if it were the difficulty.
It causes a problem at startup: Digishield coins take 500 or more blocks to get to the correct difficulty, literally giving them away. This giveaway causes the difficulty to then overshoot, causing delays while it gets back down to the correct level. In the Zcash example below you can see it did not overshoot which is because they had a slow start period where the reward was really small. They did the slow start at least in because they were warned by another coin that difficulty would cause a problem at start up.
To prevent the problem delete all the following code:
The 16% / 32% limits are only 1/2 the cause of the startup problem. Even without the limits, the 4x "tempering" in Digishield causes 260 block delays at startup where SMA and LWMA cause 140 and 60 block delays respectively, as shown in the image below which shows a 1000x increase in hashrate for 400 blocks (all the red). Interestingly, these delays are the same for any > 50x hashrate increase.
Problem 2: Delay in responding (MTP)
Most Digishield v3 implementations do not get data from the most recent blocks, but begin the averaging at the MTP, which is typically 6 blocks in the past. This is ostensibly done to prevent timestamp manipulation of the difficulty. However, there are several different methods of dealing with bad timestamps that will not inject this delay.
These charts show hashrate jumps on Digishield coins when difficulty accidentally falls low. But I think there is a distinct excess of oscillations that is being caused by the MTP delay.
Asymmetry
The lack of symmetry (+16 / -32 instead of +16 / -16 limits) has made things worse for Bitcoin Gold. BCH EDA also had a lack of symmetry where it could rise faster than fall which led to oscillations and too many blocks being issued. BTG also has the ugly oscillations and 3% too many blocks are being issued. The oscillations are obvious, but are not causing terrible delays. Although the 16/32 limits are not close to what's needed and meant by "asymmetry", at least the avg solvetime would have been correct if they were symmetrical, if not fewer oscillations.
Useful Tempering
The Digishield v3 code has tempering that only seem to be only slightly better than using SMA with N=68.
Simplifying the math this is:
nActualTimespan = 0.75*params.AveragingWindowTimespan() + 0.25*ActualTimespanTheparams.AveragingWindowTimespan()is just a constant N*TargetSolvetime. This equation takes only recent data into account, but it includes some skepticism about changing from the past.Summary
Digishield v3 code is effectively:
Once the bad stuff is removed, the resulting Tempered SMA works better than an SMA with N=65.