My current estimate of the best N based on the target solvetime for LWMA is:
T = 600, N=45
T = 300, N=55
T = 150, N=70
T = 75, N=85
Medium size coins and coins who are confident their POW will not be subject to on-off mining like NiceHash should double these numbers. The largest coins should triple these numbers. The choice is a balance between wanting a fast speed of response by lower N to keep up with network hashrate changes and not wanting random variation caused by lower N to encourage hashrate changes. For example, BTG with N=45 and T=600 will see a typical random variation of 1/SQRT(45) = 15% every 45 blocks, which is 7.4 hours. 15% is probably 5x more than the typical price+fee changes they will see every 7.5 hours, so I've probably recommended a value too low for them because with their unique equihash POW, they would probably not see 5x or 10x changes in hashrate 5x per day, which is what the N=45 would try to protect.
The above 45 should be changed to 10 for if you're using my termpered SMA (like digishield). For SMA make it 34. For EMA make it 20. This will give all 4 algorithms the same speed of response to a hash attack that is 3x to 30x your baseline hashrate after about 20 blocks. They differ in how soon they rise after an attack begins (some start slow like SMA but then catch up and pass others that were initially ahead like Digishield) while other are more stable, but have more post-attack delays (digishield verses EMA).
More recent comment from me:
Q: "What should N be for T=300?"
A: "N=50 to 60. I have no solid ground or intuition for choosing N, except that it should be in the N=45 to N=90 range for T=240 or thereabouts. A coin that is the largest for a gven POW might use N=200 to have a really smooth difficulty with little ill effect. So for the typical micro to small coins we have, I say N = 50 might be good, but sticking N=60 makes it more comparable to existing LWMA's. and LWMA-2 may completely change this......maybe all N's need to rise 50% to 100% now that it can rise so fast.
But I would say just stick with N=60."
Introduction / background
The choice of N for the size of an averaging window in a difficulty algorithm is more important than the algorithm (if the algorithm is at least as good as a simple moving average).
Choosing N tries to achieve a balance between a fast response to price (and other causes of hash rate changes) and minimizing accidental variation that accidentally attracts or discourages hashrate with an incorrect difficulty. Smaller N gives faster response but higher variation. The speed at which the difficulty can keep up with price changes is proportional to N*T. The accidental variation in D in block time is proportional to 1/SQRT(N). Approximately, we want to minimize some function like this:
A/SQRT(N) + B*N*T
i.e.: variation + slowness
where A and B are some unknown scaling factors that relate the relative importance of 1/SQRT(N) and N*T. It needs to be determined by watching the results of a lot of live coins. The left side does not contain time, but it should because the number of instances of variation in real time depend on T. But deciding how to include it is complicated as shown further below. My estimate is that it's a 1/x^0.33 function instead of 1/x^0.5 near the optimal points, and degrades to 1/x as too much variation starts becoming really attractive to opportunistic miners.
So stability increases as 1/N^0.33 while speed decreases linearly with N, so there's a mathematical advantage to be biased towards a low N. But if the stability causes the variation to go above about +/- 20% of the correct difficulty, droves of miners will suddenly be coming online and offline whenever that event occurs, amplifying instability (but not necessarily causing bad oscillations: some coins are satisfied with N=17). On the other hand, if it's responding fast enough, fewer miners are going to bother switching coins to "attack" the accidentally low hashrate. The chart below estimates the effect I see in live coin delays and hash attacks. I had to use 1/N^0.66 to get the left side of the curve.
It appears very likely that a specific N can be chosen to be good all reasonable target solvetimes (60 to 600). There's a plateau in the above graph that suggests this.
Miners seek best price/difficulty ratio. If the price/difficulty ratio drops by 25%, there is typically a 300% increase in hashrate (in the coins I follow) until the difficulty catches up. So we want to respond fast enough to keep up with price changes, but we don't want to accidentally go 25% too low because to a miner that's the same thing as a 25% price increase.
BTW if the price and hash rate of a coin changed in "block time" instead of "real time", then changing T would have no effect on the difficulty algorithm's math or performance.
Live coins with different N and T values
If the above gives an accurate idea of how we should change N relative to T, then we still need a good starting point as determined by watching live coins.
N=320 is definitely too large for coins with T=240 seconds (target solvetime) as seen in Monero clones. BCH's N=144 with T=600 is not performing as well as other coins in terms of delays. Zcash clones with T=150 seconds are doing pretty darn good with Digishield v3 and could do better if they employ my modifications to it. Digishield v3 with N=17 is like an N=60 algorithm. Two or four times a day both Zcash and Hush see a 3x increase in hashrate for 20 blocks. Masari with T=120 and N=60 my WHM is doing the best of the 7 difficulties I follow. Compared to Zcash and Hush, it gets 1/2 as many of these "hash attacks" per day ("attack" = avg 11 solvetimes < 1/3 T) and 1/3 as many delays (delay = avg of 11 solvetimes > 2xT).
720 x 120 and 300 x 240 = disaster (Karbowanek, Masari, Sumokoin, et al)
144 x 600 = not very good (BCH)
60ish x 150 = pretty good (Zash, HUSH)
60 x 120 = super awesome (Masari)
17 x 600 = not bad (if BTG's had not been screwed by bad POW limits)
17 x 240 = fairly bad (Sumokoin, Karbowanec)
17 x 120 = disastrous (Masari)
A good algo seems to have about N=60 and T=150 which will have a 25% change in difficulty from random variation once every day, but price does not typically change near that much. So there seems to be a strong bias towards keeping a faster response than price change alone requires. Maybe it's partly because 10% to 15% of hash power can come and go even with a constant price/difficulty ratio and it's good for the algorithm to respond quickly to those changes, especially if they coincide with a 10% change in price a couple of times per week. But a bigger reason we seem to need algos with more accidental variation is that they are able to correct themselves quicker, so the higher variability is not so harmful.
Changing N when T changes
Once we know the ideal N and T for a particular algorithm, we should choose a larger N for a coin that has a smaller T, and vice versa. Since there are more sample points in real time for a smaller T, we can have a better functioning algorithm. With a smaller T we can choose a faster response, less random variation, or a smaller improvement in both.
Options if T is made smaller: (N will be larger)
Less random variation in real time, same response rate:
new_N = old_N x old_T / new_T
where T=target solvetime.
Faster response in real time, but same random variation in real time
new_N = old_N x (old_T / new_T)^0.33
(see footnote on the source of 0.33)
A little bit better response rate and a little less random variation
new_N = old_N x (old_T / new_T)^0.66
This gives N=1.5x higher when T is 1/2.
Options if T is made larger (N will need to be smaller)
Use the same equations above, but replace "less", "faster", and "better" with "more", slower", and "worse". Larger T means fewer data points per day to estimate the network hashrate, so the algorithm is necessarily not going to be as good.
Foot note:
Source of my 0.33 above:
For a given algorithm and level of probability, difficulty randomly varies as +/- k*StdDev/SQRT(N) in every group of N blocks. k is a constant that depends on the algorithm and on the value of standard deviation. StdDev=1.96 means 95% of difficulty values will be within that range. k is about 1.1 to 1.6 depending on StdDev and the algorithm. It's not precise because D is not a Gaussian random variable even if hash rate is constant. I'll assume k=1 here because I'm looking for a trend that will give me a power factor, not a precise result (because I don't know A and B anyway).
Let's assume I want to keep random variation down to one 25% accident every 5 days because I do not expect price variation to be more than that, and I sort of want the two variations about equal in an effort to balance them (this is like assuming A=B=1, but I believe A<B). There will be 5 x 24 x 3600 / (NxT) sets of N blocks. For a single set, a 25% variation will approximately obey the equation:
0.25 = StdDev/SQRT(N)
I want an N that has a 25% variation in difficulty in 5 days when the hash rate is constant.
With T=600, there are 5 days x 24 x 3600 / (Nx600) = 720/ N sets of N blocks. I will first guess the ideal N is 60, so there will be 720/60 = 12 sets of N blocks in 5 days. I want a 1/12 chance of having a 25% variation in each set of N blocks. A chart tells me a 1/12 two tailed probability has a StdDev of 1.53. Now I solve:
0.25 = 1.53/SQRT(N) => N=37
The true N I seek is about half way between my N=60 guess and the N=37. I try again with (37+60) /2 = 49. 49/720 = 6.8%. Chart gives StdDev =1.83 for 6.8%.
0.25 = 1.83/SQRT(N) => N=54
So it seems N=(49+54)/2 = 52 is a pretty good choice for T=600 and 25% variation once in 5 days.
Repeating the same process for T=300 and T=150 gives N=64 and 78. These number follow the trend N=52*(600/T)^0.31
Repeated for a different set of conditions gave a power of 0.33.
My current estimate of the best N based on the target solvetime for LWMA is: T = 600, N=45 T = 300, N=55 T = 150, N=70 T = 75, N=85
Medium size coins and coins who are confident their POW will not be subject to on-off mining like NiceHash should double these numbers. The largest coins should triple these numbers. The choice is a balance between wanting a fast speed of response by lower N to keep up with network hashrate changes and not wanting random variation caused by lower N to encourage hashrate changes. For example, BTG with N=45 and T=600 will see a typical random variation of 1/SQRT(45) = 15% every 45 blocks, which is 7.4 hours. 15% is probably 5x more than the typical price+fee changes they will see every 7.5 hours, so I've probably recommended a value too low for them because with their unique equihash POW, they would probably not see 5x or 10x changes in hashrate 5x per day, which is what the N=45 would try to protect.
The above 45 should be changed to 10 for if you're using my termpered SMA (like digishield). For SMA make it 34. For EMA make it 20. This will give all 4 algorithms the same speed of response to a hash attack that is 3x to 30x your baseline hashrate after about 20 blocks. They differ in how soon they rise after an attack begins (some start slow like SMA but then catch up and pass others that were initially ahead like Digishield) while other are more stable, but have more post-attack delays (digishield verses EMA).
More recent comment from me: Q: "What should N be for T=300?" A: "N=50 to 60. I have no solid ground or intuition for choosing N, except that it should be in the N=45 to N=90 range for T=240 or thereabouts. A coin that is the largest for a gven POW might use N=200 to have a really smooth difficulty with little ill effect. So for the typical micro to small coins we have, I say N = 50 might be good, but sticking N=60 makes it more comparable to existing LWMA's. and LWMA-2 may completely change this......maybe all N's need to rise 50% to 100% now that it can rise so fast. But I would say just stick with N=60."
Introduction / background The choice of N for the size of an averaging window in a difficulty algorithm is more important than the algorithm (if the algorithm is at least as good as a simple moving average).
Choosing N tries to achieve a balance between a fast response to price (and other causes of hash rate changes) and minimizing accidental variation that accidentally attracts or discourages hashrate with an incorrect difficulty. Smaller N gives faster response but higher variation. The speed at which the difficulty can keep up with price changes is proportional to N*T. The accidental variation in D in block time is proportional to 1/SQRT(N). Approximately, we want to minimize some function like this:
A/SQRT(N) + B*N*Ti.e.: variation + slownesswhere A and B are some unknown scaling factors that relate the relative importance of 1/SQRT(N) and N*T. It needs to be determined by watching the results of a lot of live coins. The left side does not contain time, but it should because the number of instances of variation in real time depend on T. But deciding how to include it is complicated as shown further below. My estimate is that it's a 1/x^0.33 function instead of 1/x^0.5 near the optimal points, and degrades to 1/x as too much variation starts becoming really attractive to opportunistic miners.
So stability increases as 1/N^0.33 while speed decreases linearly with N, so there's a mathematical advantage to be biased towards a low N. But if the stability causes the variation to go above about +/- 20% of the correct difficulty, droves of miners will suddenly be coming online and offline whenever that event occurs, amplifying instability (but not necessarily causing bad oscillations: some coins are satisfied with N=17). On the other hand, if it's responding fast enough, fewer miners are going to bother switching coins to "attack" the accidentally low hashrate. The chart below estimates the effect I see in live coin delays and hash attacks. I had to use 1/N^0.66 to get the left side of the curve.
It appears very likely that a specific N can be chosen to be good all reasonable target solvetimes (60 to 600). There's a plateau in the above graph that suggests this.
Miners seek best price/difficulty ratio. If the price/difficulty ratio drops by 25%, there is typically a 300% increase in hashrate (in the coins I follow) until the difficulty catches up. So we want to respond fast enough to keep up with price changes, but we don't want to accidentally go 25% too low because to a miner that's the same thing as a 25% price increase.
BTW if the price and hash rate of a coin changed in "block time" instead of "real time", then changing T would have no effect on the difficulty algorithm's math or performance.
Live coins with different N and T values If the above gives an accurate idea of how we should change N relative to T, then we still need a good starting point as determined by watching live coins.
N=320 is definitely too large for coins with T=240 seconds (target solvetime) as seen in Monero clones. BCH's N=144 with T=600 is not performing as well as other coins in terms of delays. Zcash clones with T=150 seconds are doing pretty darn good with Digishield v3 and could do better if they employ my modifications to it. Digishield v3 with N=17 is like an N=60 algorithm. Two or four times a day both Zcash and Hush see a 3x increase in hashrate for 20 blocks. Masari with T=120 and N=60 my WHM is doing the best of the 7 difficulties I follow. Compared to Zcash and Hush, it gets 1/2 as many of these "hash attacks" per day ("attack" = avg 11 solvetimes < 1/3 T) and 1/3 as many delays (delay = avg of 11 solvetimes > 2xT).
720 x 120 and 300 x 240 = disaster (Karbowanek, Masari, Sumokoin, et al) 144 x 600 = not very good (BCH) 60ish x 150 = pretty good (Zash, HUSH) 60 x 120 = super awesome (Masari) 17 x 600 = not bad (if BTG's had not been screwed by bad POW limits) 17 x 240 = fairly bad (Sumokoin, Karbowanec) 17 x 120 = disastrous (Masari)
A good algo seems to have about N=60 and T=150 which will have a 25% change in difficulty from random variation once every day, but price does not typically change near that much. So there seems to be a strong bias towards keeping a faster response than price change alone requires. Maybe it's partly because 10% to 15% of hash power can come and go even with a constant price/difficulty ratio and it's good for the algorithm to respond quickly to those changes, especially if they coincide with a 10% change in price a couple of times per week. But a bigger reason we seem to need algos with more accidental variation is that they are able to correct themselves quicker, so the higher variability is not so harmful.
Changing N when T changes Once we know the ideal N and T for a particular algorithm, we should choose a larger N for a coin that has a smaller T, and vice versa. Since there are more sample points in real time for a smaller T, we can have a better functioning algorithm. With a smaller T we can choose a faster response, less random variation, or a smaller improvement in both.
Options if T is made smaller: (N will be larger)
Less random variation in real time, same response rate: new_N = old_N x old_T / new_T where T=target solvetime.
Faster response in real time, but same random variation in real time new_N = old_N x (old_T / new_T)^0.33 (see footnote on the source of 0.33)
A little bit better response rate and a little less random variation new_N = old_N x (old_T / new_T)^0.66 This gives N=1.5x higher when T is 1/2.
Options if T is made larger (N will need to be smaller) Use the same equations above, but replace "less", "faster", and "better" with "more", slower", and "worse". Larger T means fewer data points per day to estimate the network hashrate, so the algorithm is necessarily not going to be as good.
Foot note: Source of my 0.33 above:
For a given algorithm and level of probability, difficulty randomly varies as +/- k*StdDev/SQRT(N) in every group of N blocks. k is a constant that depends on the algorithm and on the value of standard deviation. StdDev=1.96 means 95% of difficulty values will be within that range. k is about 1.1 to 1.6 depending on StdDev and the algorithm. It's not precise because D is not a Gaussian random variable even if hash rate is constant. I'll assume k=1 here because I'm looking for a trend that will give me a power factor, not a precise result (because I don't know A and B anyway).
Let's assume I want to keep random variation down to one 25% accident every 5 days because I do not expect price variation to be more than that, and I sort of want the two variations about equal in an effort to balance them (this is like assuming A=B=1, but I believe A<B). There will be 5 x 24 x 3600 / (NxT) sets of N blocks. For a single set, a 25% variation will approximately obey the equation:
0.25 = StdDev/SQRT(N)I want an N that has a 25% variation in difficulty in 5 days when the hash rate is constant.
With T=600, there are 5 days x 24 x 3600 / (Nx600) = 720/ N sets of N blocks. I will first guess the ideal N is 60, so there will be 720/60 = 12 sets of N blocks in 5 days. I want a 1/12 chance of having a 25% variation in each set of N blocks. A chart tells me a 1/12 two tailed probability has a StdDev of 1.53. Now I solve:
0.25 = 1.53/SQRT(N) => N=37The true N I seek is about half way between my N=60 guess and the N=37. I try again with (37+60) /2 = 49. 49/720 = 6.8%. Chart gives StdDev =1.83 for 6.8%.
0.25 = 1.83/SQRT(N) => N=54So it seems N=(49+54)/2 = 52 is a pretty good choice for T=600 and 25% variation once in 5 days.
Repeating the same process for T=300 and T=150 gives N=64 and 78. These number follow the trend
N=52*(600/T)^0.31
Repeated for a different set of conditions gave a power of 0.33.