I made this article long and complicated, so let me try to summarize "how to identify the best algorithm":
Normalize their "N" parameter until they all have the same response to 2x to 5x attacks that last from 10 to 30 blocks (very typical of what Zcash and other coins see). I average the values they have at the end of the step function to make sure it is about the same value. Any of them can be faster or slower based on the N. Speed comes at a price of stability. You get speed proportional to 1/N and stability with SQRT(N). Then, to check overall performance, I model real attacks that being when difficulty is 0.7 to 0.95 of the baseline difficulty. The attacks can be 2x to 25x hashrate which occurs in small coins. The attack lasts until the difficulty reaches 1.1 to 5x the baseline difficulty. I have metrics that measure delays and "blocks stolen at low difficulty", which ranks the results. This turns out to be the same as the winning algo having the lowest root-mean-square of the error between where the difficulty should be based on the hash rate and where the difficulty is. And for a given speed of response, the winner can be identified apparently just as well by just choosing the one with the lowest standard deviation when the hash rate is constant (since it drops lower less often, it will attract fewer hash attacks and therefore score better).
The reverse process for objectively determining the best algorithm can be used: Adjust their N parameter until the competing algorithms have the same StdDev when the hash rate is constant. The one with the fastest response to the "hash attacks" you expect is the winner, which is LWMA with Digishield again tying for 2nd place with EMA and clearly beating SMAs. The reason SMA is not good is because it is getting the avg hashrate as it was N/2 blocks in the past, while EMA and LWMA are estimating current HR. Digishield is not quite as good because it's tempering it's calculation of 17/2 blocks in the past, but a lot better than SMA because an equivalent-speed SMA is N = 4x17 = 68 which is 34 block behind.
End edit
This is complicated, but necessarily so.
The biggest problem in identifying the best algorithm is first defining the metric for success. I define it as "the least random variation at a constant hashrate for a given speed of response". "For a given" refers to competing algorithms having a parameter "N" that can be decreased to meet the speed requirement which causes a loss in the stability requirement. A related article to this is the Selecting N article which tries to identify the ideal N to make this balance, once you've identified the best algorithm.
I've checked this method against other methods such as having the fewest 2x target solvetime delays and having the lowest sum of squares error when miner profit motivation is simulated. For those interested in the "least squares" error measurement, I believe we should use:
if ( nD/D > 1 ) { (1-nD/D)^2 } else {(1-D/nD)^2 } where nD=needed difficulty for the given hashrate
instead of the standard (nD-D)^2.
This is for each block where D is what the algorithm generates. Then sum these up and take the SQRT to scale it down. This is an RMS of the error where error is defined in a way that meets our needs.
To repeat, all algorithms have a parameter "N" that increases stability at the expense of its speed of response as you raise the N. When comparing algorithms, the best thing to do is first find their "equivalent ideal N" by checking their response to a sudden increase in hashrate. ( BTW All acceptable algorithms will be symmetrical (equal) in their increase and decrease rate to sudden changes or the target solvetime will average too high (or too low) and oscillations will be present. If your target solvetime is too low, then your algorithm is decreasing faster than it increases. )
To find the equivalent N, you send the algos step functions in hashrate that are some multiple of baseline hashrate and a fixed width apart, and make the steps 2x that far apart for the difficulty to drop back to normal. Their equivalent N will change a little bit for different step functions, so your step function should model the kind of ability you want the algorithms to have. I use 2x to 3x hashrate step function and select a width where the algos have reached about half-way to where they should be for that hashrate (about N/2 for the N of the current champion algo). You average D for the last block at the end of the all the step function "assaults, needing 10,000 block runs to get 100 data points for steps that are 33 blocks wide with 66 block gaps.
For a given N in an SMA, the equivalent speed of response in the other algorithms with step functions N/2 wide are:
WHM = WT = 1.32xN
EMA = 3.3xN using e^(-t/T/N) instead of my e^(-2.5*t/T/N)
Digishield = N/4.25
For wider step functions, SMA needs a larger relative N, and vice versa. But thankfully the relative N's for the others stay closer for different width step functions because they are faster in starting a correction.
After their "equivalent N" is found to standardize the speed of response you want to the type of hash rate changes you expect, you then check their stability during constant hashrate (not during hash rate changes where you want high variation). The algorithm with the lowest standard deviation of the difficulty wins. If you want to increase stability, you have to increase N which will sacrifice the speed you wanted. You would change the equivalent N for the competing algorithms by the same percentage to keep them all in the same terms.
In order of the best algorithms, the standard deviations for the WHM N=60 (which I think is the best for coins with target solvetime T=150) are:
WHM = 0.13
EMA = WT = 0.14
Digishield = 0.15
SMA = 0.16
This seems small but the difference between SMA and WHM is about 3x fewer delays and 20% fewer hash attacks. My charts below are for WHM with N=90 (and the others with their equivalent N).
You will find that all algorithms have too much instability for the speed of response you want. To balance the competing goals to make the best of a bad situation, see the Selecting N article.
If the above process is confusing or suspect, my old method is a good check and it's shown in the last chart below. It's done like this this: The equivalent N as before is found and used. Instead of simply looking as Std Dev, I subject the algos to expected miner profit motivation. Miners want the lowest Price/Difficulty and view a 20% increase in price the same as a 20% decrease in difficulty (or more precisely 1/1.20 = 0833 => 16.66% decrease in difficulty). So the simulation says "during constant hashrate conditions, if difficulty drops below A% of the correct difficulty then a Bx the baseline hashrate "attack" begins, and ends when the difficulty is C% above the correct difficulty. If an algo varies a lot, it gets attacked a lot, but if it varies a lot it is also responding quickly to have fewer blocks "stolen" in each attack. so it's a complex feedback situation that needs test runs and measurement....but it's all simplified because I've already standardized the number of blocks stolen in each attack by standardizing the response speed. This is why all the red bars in last chart are approximately the same width. To be more accurate I comparison, if my miner motivation is correctly modeled, then N should be refined to make the red bars in all algos of equal width, but the further refinement is not important because of a "can't win for losing" effect from the "complex feedback situation". For example, Digishield seems cheated because it has a faster-than expected response (thinner bars) in comparison to how I found its equivalent N. If I increase it's N, then the delay metric gets even worse and that's its main problem compared to the others.
The DWHM-EMA is not easily expressed in terms of the others, but it seems to be the best algorithm. It's not published yet due to complications that make me think it's all the same as WHM and because I already have its cousin published, the Dynamic EMA.
In the charts below, the "delay" and "blocks stolen" percentages are the number of blocks who's average of 11 solvetimes were > 2x target solvetime (delays) or < 1/2 target solvetime ( the difficulty is set too low for the amount of hashing). These are the check on the Std Dev conclusion The number in the charts vary for different runs. I've sorted the charts from best to worst based on more data.
The following ranking is based on 8x delay percentage plus the "blocks stolen" percentage.
WHM-EMA 15.6
WHM 18.3
EMA 20.5
WT 25.4
Digishield 27.9
SMA 36.4
Edit
I made this article long and complicated, so let me try to summarize "how to identify the best algorithm":
Normalize their "N" parameter until they all have the same response to 2x to 5x attacks that last from 10 to 30 blocks (very typical of what Zcash and other coins see). I average the values they have at the end of the step function to make sure it is about the same value. Any of them can be faster or slower based on the N. Speed comes at a price of stability. You get speed proportional to 1/N and stability with SQRT(N). Then, to check overall performance, I model real attacks that being when difficulty is 0.7 to 0.95 of the baseline difficulty. The attacks can be 2x to 25x hashrate which occurs in small coins. The attack lasts until the difficulty reaches 1.1 to 5x the baseline difficulty. I have metrics that measure delays and "blocks stolen at low difficulty", which ranks the results. This turns out to be the same as the winning algo having the lowest root-mean-square of the error between where the difficulty should be based on the hash rate and where the difficulty is. And for a given speed of response, the winner can be identified apparently just as well by just choosing the one with the lowest standard deviation when the hash rate is constant (since it drops lower less often, it will attract fewer hash attacks and therefore score better).
The reverse process for objectively determining the best algorithm can be used: Adjust their N parameter until the competing algorithms have the same StdDev when the hash rate is constant. The one with the fastest response to the "hash attacks" you expect is the winner, which is LWMA with Digishield again tying for 2nd place with EMA and clearly beating SMAs. The reason SMA is not good is because it is getting the avg hashrate as it was N/2 blocks in the past, while EMA and LWMA are estimating current HR. Digishield is not quite as good because it's tempering it's calculation of 17/2 blocks in the past, but a lot better than SMA because an equivalent-speed SMA is N = 4x17 = 68 which is 34 block behind.
End edit
This is complicated, but necessarily so.
The biggest problem in identifying the best algorithm is first defining the metric for success. I define it as "the least random variation at a constant hashrate for a given speed of response". "For a given" refers to competing algorithms having a parameter "N" that can be decreased to meet the speed requirement which causes a loss in the stability requirement. A related article to this is the Selecting N article which tries to identify the ideal N to make this balance, once you've identified the best algorithm.
I've checked this method against other methods such as having the fewest 2x target solvetime delays and having the lowest sum of squares error when miner profit motivation is simulated. For those interested in the "least squares" error measurement, I believe we should use:
if ( nD/D > 1 ) { (1-nD/D)^2 } else {(1-D/nD)^2 }where nD=needed difficulty for the given hashrate instead of the standard(nD-D)^2. This is for each block where D is what the algorithm generates. Then sum these up and take the SQRT to scale it down. This is an RMS of the error where error is defined in a way that meets our needs.To repeat, all algorithms have a parameter "N" that increases stability at the expense of its speed of response as you raise the N. When comparing algorithms, the best thing to do is first find their "equivalent ideal N" by checking their response to a sudden increase in hashrate. ( BTW All acceptable algorithms will be symmetrical (equal) in their increase and decrease rate to sudden changes or the target solvetime will average too high (or too low) and oscillations will be present. If your target solvetime is too low, then your algorithm is decreasing faster than it increases. )
To find the equivalent N, you send the algos step functions in hashrate that are some multiple of baseline hashrate and a fixed width apart, and make the steps 2x that far apart for the difficulty to drop back to normal. Their equivalent N will change a little bit for different step functions, so your step function should model the kind of ability you want the algorithms to have. I use 2x to 3x hashrate step function and select a width where the algos have reached about half-way to where they should be for that hashrate (about N/2 for the N of the current champion algo). You average D for the last block at the end of the all the step function "assaults, needing 10,000 block runs to get 100 data points for steps that are 33 blocks wide with 66 block gaps.
For a given N in an SMA, the equivalent speed of response in the other algorithms with step functions N/2 wide are: WHM = WT = 1.32xN EMA = 3.3xN using e^(-t/T/N) instead of my e^(-2.5*t/T/N) Digishield = N/4.25
For wider step functions, SMA needs a larger relative N, and vice versa. But thankfully the relative N's for the others stay closer for different width step functions because they are faster in starting a correction.
After their "equivalent N" is found to standardize the speed of response you want to the type of hash rate changes you expect, you then check their stability during constant hashrate (not during hash rate changes where you want high variation). The algorithm with the lowest standard deviation of the difficulty wins. If you want to increase stability, you have to increase N which will sacrifice the speed you wanted. You would change the equivalent N for the competing algorithms by the same percentage to keep them all in the same terms.
In order of the best algorithms, the standard deviations for the WHM N=60 (which I think is the best for coins with target solvetime T=150) are: WHM = 0.13 EMA = WT = 0.14 Digishield = 0.15 SMA = 0.16
This seems small but the difference between SMA and WHM is about 3x fewer delays and 20% fewer hash attacks. My charts below are for WHM with N=90 (and the others with their equivalent N).
You will find that all algorithms have too much instability for the speed of response you want. To balance the competing goals to make the best of a bad situation, see the Selecting N article.
If the above process is confusing or suspect, my old method is a good check and it's shown in the last chart below. It's done like this this: The equivalent N as before is found and used. Instead of simply looking as Std Dev, I subject the algos to expected miner profit motivation. Miners want the lowest Price/Difficulty and view a 20% increase in price the same as a 20% decrease in difficulty (or more precisely 1/1.20 = 0833 => 16.66% decrease in difficulty). So the simulation says "during constant hashrate conditions, if difficulty drops below A% of the correct difficulty then a Bx the baseline hashrate "attack" begins, and ends when the difficulty is C% above the correct difficulty. If an algo varies a lot, it gets attacked a lot, but if it varies a lot it is also responding quickly to have fewer blocks "stolen" in each attack. so it's a complex feedback situation that needs test runs and measurement....but it's all simplified because I've already standardized the number of blocks stolen in each attack by standardizing the response speed. This is why all the red bars in last chart are approximately the same width. To be more accurate I comparison, if my miner motivation is correctly modeled, then N should be refined to make the red bars in all algos of equal width, but the further refinement is not important because of a "can't win for losing" effect from the "complex feedback situation". For example, Digishield seems cheated because it has a faster-than expected response (thinner bars) in comparison to how I found its equivalent N. If I increase it's N, then the delay metric gets even worse and that's its main problem compared to the others.
The DWHM-EMA is not easily expressed in terms of the others, but it seems to be the best algorithm. It's not published yet due to complications that make me think it's all the same as WHM and because I already have its cousin published, the Dynamic EMA.
In the charts below, the "delay" and "blocks stolen" percentages are the number of blocks who's average of 11 solvetimes were > 2x target solvetime (delays) or < 1/2 target solvetime ( the difficulty is set too low for the amount of hashing). These are the check on the Std Dev conclusion The number in the charts vary for different runs. I've sorted the charts from best to worst based on more data.
The following ranking is based on 8x delay percentage plus the "blocks stolen" percentage.