EMA for BCH (part 2)

[zawy12]

In a previous issue I discussed this in a different way, received feedback from Mark L, and discussed the possibility of RTT for BCH. This issue discusses everything that came up in a live stream and everything I know of surrounding changing BCH's difficulty algorithm (DA).

My process to find the "best algo" is this: find a window size parameter "N" for each algo that gives them the same standard deviation for constant HR. This indicates they accidentally attract on-off mining at the same rate. Then simulate on-off mining with step response that turns on and off based on a low and high difficulty value. I have "% delayed" and "% blocks stolen" metrics. Delay: add up a delay qty that is (st - 4xT) for each block over 4xT. Divide that by total blocks to get a % delayed. For "% stolen" I divide time-weighted target that the on-off miner sees by that of dedicated miners. It's how much easier his target was, while he was mining. I just now re-tested 5 DAs. Except for SMA-144, the other 4 below are pretty much the same. WT-190 has slightly lower % stolen but slightly higher delays. (WT-144 is what I kept mistakenly calling wtema in the video). Delays and % stolen are usually a tradeoff. I had been penalizing WT-144 in the past by giving it the same N as LWMA when it needs a larger value.

SMA-144 WT-190 (as opposed to wt-144) LWMA -144 ASERT / EMA - 72 [ update: See my 2nd comment below. In Jonathan's code, this value would be called 72 * ln(2) ]

I'll advocate the following for BCH in order of importance.

• ASERT or wtema with N=150 [ correction: this would be 150 * ln(2) in Jonathan / Neil's code ]
• Remove 1/2 timespan limit & do not add anything that limits how fast D rises (to prevent the unlimited blocks exploit).
• FTL 300 seconds with peer time removed or set it to FTL/2 instead of 70 minutes.
• No median of 3.
• 7xT limit on solvetimes to prevent spam attack

There's also sequential timestamps which might be the hardest to do in reality, but important on a theoretical level. It might be possible to enforce sequential timestamps by making MTP=1 instead of 11. Maybe this could prevent bricking of equipment. Sequential stamps (+1 second) must then be able to override local time & FTL. Overriding local time to keep "messages" (blocks) ordered messages is a requirement for all distributed consensus mechanisms. MTP does this indirectly.

Antony Zegers did some EMA code, using bit-shift for division.

EMA is a close enough approximation to ASERT, but here's Mark's tweet on how to do accurate integer-based e^x. https://twitter.com/MarkLundeberg/status/1191831127306031104?s=20

wtema simplifies to:
target = prev_target (1 + st/T/N - 1/N) where N is called the "mean lifetime" in blocks if it were expressed as ASERT, target = prev_target e^(st/T/N - 1/N) The other form of ASERT should give the same results, but the other form of EMA could return a negative difficulty with a large st. [ wtema-144 and asert-144 refer to N = 144 / ln(2) = 208.

If wtema is used: Negative solvetimes (st) are a potential problem for wtema if N is small and delays are long and attacker sends an MTP timestamp. See wt-144 code for how to block them. To be sure to close the exploit, you have to go back 11 blocks. Do not use anything like if (st < 0) { st = 0 } in the DA which allows a small attacker to send difficulty for everyone very low in a few blocks by simply sending timestamps at the MTP.

Std Dev of D per N blocks under constant HR for EMA/ASERT is ~1/SQRT(2*N). This is important for deciding a value for N. If HR increases 10x based on 10% drop in D, you definitely want N > 50 or the DA is motivating on-off mining. On the other hand, being at the edge of motivating on-off mining can reduce the expected variation in D. BTG's Std Dev is about 0.10 when it would have been 0.15 if HR was constant (N=45 LWMA, N=22.5 in ASERT terms). So it's paying a little to on-off miners to keep D more constant. ASERT with N=72 is same as SMA=144 in terms of D variation when HR is constant.

On the other side is wanting a sufficiently fast response to price changes. As a starting point, for ASERT N=100 it takes 6 blocks to rise 5% in response to a 2x increase in HR. This seems sufficiently fast. Std Dev is 1/SQRT(2*100) = 7% per 100 blocks which seems high. 5% accidental changes in D seems to motivate substantial HR changes, so maybe N=200 is better which will have 5% variation per 200 blocks. But this is like LWMA N=400 which would be incredibly smooth and nice if it works but this is 3x larger than my experience. BTG is very happy with LWMA N=45 with Std Dev 10%. Notice that there's bigger bang for the buck with lower N due to the 1/SQRT(N) verses N. You get more speed than you lose stability as you go to lower N. So instead of 200, I would go with 100 or 150.

None of the 60 or so LWMA coins got permanently stuck. About 5% could not get rid of bothersome oscillations. When sending the same random numbers to generate solvetimes, LWMA / EMA / ASERT are almost indistinguishable, so I do not expect them to get stuck either. Most LWMA's are N=60 which has the stability of EMA with N=30. N=100 would have probably been a lot nicer for them, if not N=200 (ASERT N=100). They are mostly T = 120 coins, so they have very fast response, reducing the chances of getting stuck.

Mark was concerned about forwarded timestamps sending difficulty very low which could allow a small attacker to do a spam attack (he would begin with setting a far-forwarded stamp for EMA/ASERT for submission later). To make this a lot harder to so, use st = min(st, 7*600). So long delays will not drop the difficulty as much as "it wants to". Most LWMA's do this as an attempt to reduce oscillations but it may not have helped any.

Combining upper and lower limits on difficulty, timespan, or solvetimes like the 4x and 1/4 in BTC, and the 2 and 1/2 in BCH is what allows unlimited blocks in < 3x the difficulty window. I'm the only one I know of that has described it. A couple of coins following me were the first to see it, losing 5,000 blocks in a couple of hours because I had the hole in LWMA but did not realize it. Some LWMA's had symmetrical limits on st, not just the 7xT upper limit. Removing the 1/2 limit on timespan in BCH will prevent it.

If FTL is dropped, the 70 minute revert from peer to local time rule should be removed (best) or set it to FTL/2 or less. It should be coded as function of FTL. There are a couple of exploits due to Sybil or eclipse attacks on peer time.

A pre-requisite for BFT is that all nodes have a reliable clock that has higher security than the BFT. The clock in POW is each node operator knowing what time it is without needing to consult the network. Any node can unilaterally reject a chain if its time is too far in the future so it's impervious to 99.99999% attacks. (Although current mining on honest chain needs to be > 50% HR of the forwarded cheating chain for the node to continue to reject the cheating chain because time going forward can make the cheating chain become valid.)

[jtoomim]

I've added ASERT to my fork of the kyuupichan difficulty test suite. This test suite allows for testing different algorithms with the same random seed (@zawy12, do I remember correctly that your suite uses a different random seed each time?). Unsurprisingly, ASERT performs basically identically to WTEMA on all metrics. The only differences between the two are small enough to be explained by rounding errors.

http://ml.toom.im:8051/ https://github.com/jtoomim/difficulty/blob/comparator/mining.py#L313

Also, here's Mark Lundeberg's paper about ASERT:

http://toom.im/files/da-asert.pdf

[zawy12]

Recently I have used the same seed to compare better. The code is "test_DAs.cpp" in the code section but it may be difficult to figure out how to use. Some of the mess is because it supports some RTT testing. It's written in terms of difficulty for CN coins.

You're heavy into testing and my 2nd paragraph is important, so I'll repeat it in a different way. The goal in testing is to choose a balance between 1) accidental variation that will not attract on-off mining and 2) speed of response to HR changes. You can't get both because lower variation comes with higher SQRT(N) and speed of response comes with lower N. So to compare algos fairly, you decide on an acceptable variation under constant HR and find the N for each algo that achieves it. Then you compare their the speed of response to step functions. Ability to rise fast means the on-off miner has less excess profits. The ability to come down fast means there will be fewer delays (and helps "dedicated" miner profits). You check speed of rising and speed of falling because some algos are good at one but not the other. You also check avg solvetime because if it's higher than the others, the algo is getting an unfair advantage.

So, I have the following algorithms with their N parameters chosen to give the same Std Dev under constant hashrate. This is in order from worst to best in terms of blocks stolen. Again, only SMA is significantly different from the others. WT wins but at a cost in delays.

SMA-144 (no median of 3) EMA - 72 (wtema form) ASERT - 72 LWMA - 144 WT - 190

Here are charts for the above. This is with 6x HR on-off simulating last years oscillations.

[jtoomim]

Edit/Note: The data presented in this comment are obsolete, due to incorrect window size settings. Accurate data can be found in this comment instead.

Here's some more data from an unpublished version of my sim. I added a proper LWMA. I notice that it takes a 240-block half-window LWMA (480-tap filter) to have the same responsiveness as ASERT-144 or WTEMA-144. ASERT and WTEMA perform identically, so I didn't include both at all N values.

CW-144 is awful, as expected -- steady-hashrate miners earn 13% less by mining BCH instead of BTC. WT-144 and WT-190 (or WHR as you call it?) are mediocre in this test. I'm curious why it performs worse in my testing than in yours. Everything else is pretty competitive, with the EMAs and LWMA at 288 blocks performing slightly better than other window sizes.

Profitability of different mining strategies

Alg	Greedy	Variable	Steady
asert-144	-0.02%	-0.001%	-0.579%
asert-288	0.01%	-0.001%	-0.463%
wtema-288	0.02%	-0.001%	-0.465%
wtema-576	0.06%	0.001%	-0.620%
lwma-190	-0.04%	-0.001%	-0.624%
lwma-240	-0.00%	-0.002%	-0.539%
lwma-288	0.01%	-0.002%	-0.482%
lwma-576	0.03%	-0.001%	-0.515%
wt-144	-0.03%	-0.011%	-2.128%
wt-190	-0.07%	-0.005%	-1.505%
cw-144	0.86%	0.131%	-13.129%

Zoomed:

The WT algos have weird glitchy excursions, like at blocks 7365 and 7422, which are completely absent from the other algorithms.

Parameters: 20k blocks, 300 PH/s steady HR, 10 EH/s variable, 1 EH/s greedy. Greedy threshold 3%.

[jacob-eliosoff]

Three quick comments:

1. For production use, avoiding floating-point math (like e^x) is probably pretty important, to avoid hardware dependencies leading to consensus failures. I understood this to be a major argument for @dgenr8's wtema.
2. The same numerical window parameter (eg 144) implies different responsiveness for EMA-type algos than for simple fixed-window algos, as noted previously. I think the code already accounts for this properly though: eg, mapping asert-"144" to IDEAL_BLOCK_TIME * 208, since 144 / ln(2) ≈ 208. That is, passing a "window" of 208 to asert should yield an algo similarly responsive to a traditional 144-block (1-day) moving avg.
3. As discussed briefly on the DAA livestream, whether to give longer blocks more weight in the weighted avg is important. As best I can recall, it's important that longer blocks not be given more weight, since that would lead to badness like 19 1-minute blocks and one 19-minute block weighted-averaging to 10 minutes.

[zawy12]

I did not realize the numbers in wtema-144 and asert-144 were "half lives" instead of the actual tau/T and alpha_recip inputs.

To compare the DA's fairly, do runs like this: N = alpha_recip = tau/block_time = mean lifetime = half-life / ln(2) ASERT = EMA = N (e.g. N = 288/ln(2), 144/ln(2), or 72/ln(2) ) LWMA = SMA = 2 N (to my surprise they have the same StdDev for a given N) WT = 2 N * 190/144

Once you decide a winner after testing for this particular N, that DA will very likely always be your winner for any consistent change to N which means it will always be your winner.

My LWMA and ASERT runs above show it's very difficult to see any difference between them which indicates this "matching Std Devs" method is good.

The WT glitch you see should go away once you give it this higher N. I just checked my WT (WHR) code and your WT code and they should be the same.

The Weighted Harmonic in these names refers to the calculation on the D/st values which is mathematically the same as Tom's target * time code. LWMA is avg(D) / weighted_mean(st) so I dropped the "WH".

For production use, avoiding floating-point math (like e^x)

Mark's tweet solves the hard step in implementing an integer based e^x, although it may take a little work to figure out how to use what he said in a complete e^x. But as we know, for large N, wtema will be the same.

[zawy12]

Once you settle on a given set of assumptions that lead to a given set of tests, then you can apply the tests to all the algos. You have to also have a metric for success. Mine is that the winning DA is the one with the least % stolen plus % delays under all on-off settings of different sizes of HR "attacks" and different starting / ending percentages of a baseline D. You make an initial guess for N as I define it above. After your tests indicate a winner for that N, you use that DA for different N's. Once you find an optimum N for that DA, there should be no other DA for any other N that will give a better result. This is if you do not change your tests and metric and if my theorizing and your testing are both correct (or equally wrong). But my charts above show LWMA, EMA, and ASERT for N>30 are basically the same thing. WT is detectably a little different.

BTW, Digishield's N in terms of SMA / LWMA needs to be N/4 due to it "diluting" the timespan by 4. The Std Dev confirms this. For SMA N=144, this is Digishield N=36 instead of N=17. Here's the same conditions above to compare ASERT and Digi N=36. "Improved" means the ~6 block MTP delay has been removed to help Digishield perform better. Default Digishield would have been 2x worse for BCH than CW-144. Digishield is an "SMA inside of an EMA" with N=4. target = avg_17_targets * (1 + avg_17_st/T/4 - 1/4) (but the 17 solvetimes here are shifted ~6 blocks in the past which makes the oscillations very consistent)

[jtoomim]

I'm not too worried about the floating point issue. It's a mild annoyance to construct a MacLaurin or Taylor series for e^x or 2^x, but it's definitely doable. @jacob-eliosoff I used float math in my initial python asert implementation out of laziness and the desire to get the algo working quickly, not because I intend that to be how it would finally be implemented in C++. I'll fix it soon-ish.

As I see it, the wtema vs asert debate centers on the following issues:

1. Integer approximations -- easier with wtema than asert, but possible in both
2. Singularity and negative solvetimes -- problematic with wtema, but not asert. wtema likely needs rules restricting solvetimes to be greater than some value (e.g. > -0.5 * 144 * 600 sec for wtema-144)
3. Mathematical and theoretical elegance -- good with both, but better for asert
4. Accumulation of rounding/approximation error -- modest in wtema, but absent in asert due to its absolute nature

Any other considerations that I'm missing?

The WT glitch you see should go away once you give it this higher N. I just checked my WT (WHR) code and your WT code and they should be the same.

Ah, yes, I see the issue. The WT code was treating N as the window size, whereas the LWMA and EMA code was treating N as the half-life or half-window size. Will fix.

[jtoomim]

With fixed N for wt, and sorted by N, and with bold for the winner at any given N:

Profitability of different mining strategies

Alg	Greedy	Variable	Steady
asert-144	-0.02%	-0.001%	-0.579%
lwma-144	-0.07%	-0.000%	-0.927%
wt-144	-0.05%	-0.001%	-0.648%
lwma-190	-0.04%	-0.001%	-0.624%
wt-190	-0.01%	-0.001%	-0.539%
lwma-240	-0.00%	-0.002%	-0.539%
asert-288	0.01%	-0.001%	-0.463%
wtema-288	0.02%	-0.001%	-0.465%
lwma-288	0.01%	-0.002%	-0.482%
wt-288	0.02%	-0.001%	-0.471%
asert-576	0.06%	0.001%	-0.616%
lwma-576	0.03%	-0.001%	-0.515%
wt-576	0.05%	0.001%	-0.597%

Edit: fixed N values a second time

Edit2: I'm going to change this again

[jtoomim]

asert-144: 'tau': (IDEAL_BLOCK_TIME * 208) wtema-144: 'alpha_recip': 208 lwma-144: 'n': 144*2 wt-144: 'block_count': 144*2 * 190 // 144

These tests results are from code that isn't on my repo yet.

You need wt576 * 190/144 = 760

The most recent data includes that.

Edit: I'm changing most of these in a second.

[zawy12]

N = alpha_recip = tau/block_time = 144/ln(2) = 208 N_lwma = N_sma = 2 N = 416 N_wt = 2.64 N = 549

[jacob-eliosoff]

@jtoomim, definitely not criticizing your implementation, but it is strongly preferable that the algo fundamentally not depend on floating-point. Yes there are integer-math approximations, but they add computation steps and, above all, complexity (in the plain English sense), leading to 1. greater risk of bugs/exploitable cases and 2. bikeshedding - there are many possible approximations.

The simpexpi/emai/emai2 algos were my stabs at integer-math approximations of simpexp (aka asert)/ema/ema2, but once I saw Tom's 2-line wtema I much preferred it. I rate simplicity as a top criterion here. (There may be equally simple integer-math algos which approximate asert even better: those to me would be wtema's real competition.)

[jtoomim]

Okay, new settings for N etc:

wt-144: 'block_count': 144*2
lwma-144: 'n': 144*2,
asert-144: 'tau': (IDEAL_BLOCK_TIME * 144),
wtema-144: 'alpha_recip': 144, 

These settings result in the step response for all four algorithms clustering closely for each responsiveness setting:

N_wt = 2.64 * N = 549

This seems wrong. Including the extra 32% (190/144) seems to mess it up. If I use that 32% factor, WT ends up as a clear outlier on the impulse response. I've removed it.

[jacob-eliosoff]

@jtoomim looking good! Apples-to-apples.

[jacob-eliosoff]

I suspect this is immaterial but 576 maps more closely to 831 than 832: 576 / ln(2) = 830.99. For testing purposes you could even maybe pass in floating-point windows just to make sure these diffs aren't distorting the comparisons, dunno...

[jacob-eliosoff]

Also to be very clear, the algo of Tom's I've been praising since I saw it is wtema (eg wtema-144), not wt (wt-144). As I recall I considered wt pretty flawed (also, needlessly messy with its for loop etc) - a little surprised it's not performing worse in these tests.

[jtoomim]

Confirmation times

Algorithm	Block interval (sec)	Confirmation time (sec)
asert-144	599.79	753.62
wtema-144	602.98	757.14
lwma-144	600.36	755.05
wt-144	603.23	761.64
asert-288	599.59	672.23
wtema-288	601.02	673.50
lwma-288	600.17	673.10
wt-288	601.22	676.09
asert-576	599.06	667.28
wtema-576	599.98	661.51
lwma-576	600.03	671.99
wt-576	600.27	673.36
cw-144/2	608.88	1565.14

The confirmation time is the expected amount of time needed to confirm a transaction issued at a random time. This is greater than the average block interval because a random instant is more likely to be in a long block interval than a short one.

Profitability of different mining strategies

Algorithm	Greedy	Variable	Steady
asert-144	-0.043%	0.011%	-0.699%
wtema-144	-0.047%	0.011%	-0.693%
lwma-144	-0.027%	0.010%	-0.720%
wt-144	-0.036%	0.010%	-0.721%
asert-288	-0.007%	0.008%	-0.333%
wtema-288	-0.007%	0.008%	-0.333%
lwma-288	-0.006%	0.008%	-0.348%
wt-288	-0.006%	0.008%	-0.348%
asert-576	0.014%	0.004%	-0.290%
wtema-576	0.016%	0.004%	-0.286%
lwma-576	0.015%	0.004%	-0.305%
wt-576	0.016%	0.004%	-0.305%
cw-144/2	1.245%	0.259%	-10.899%

cw-144/2 is a 144-block window, equivalent in impulse response to lwma-072.

[jtoomim]

@jacob-eliosoff wt-144 was performing poorly until I changed the window size from 144 blocks to 288 blocks. That is, I made "144" the half-window size, just like it is for LWMA and half-life for EMA. With this change, it performs about as well as LWMA and EMA. I'm not seriously considering it as the next BCH DAA. I've mostly only been including it in the analysis because zawy12's analysis showed it to be much better than my initial analysis did, which was a discrepancy that was worth investigating and fixing.

I agree: floating point math is bad. However, the main cost of wtema's use of addition instead of exponentiation is that wtema has a singularity when block_time == -IDEAL_BLOCK_TIME * (alpha_recip - 1). This singularity is an externalized complexity cost. It requires additional lines of code elsewhere to forbid negative solvetimes, or to bound solvetimes within a narrower range. An integer version of asert could avoid that singularity and avoid the need to change the MTP-11 rule. I suspect this would save more lines of code elsewhere than it takes to implement the integer asert approximation.

I'll take a closer look at your simpexpi/emai/emai2 soon.

[zawy12]

Notice asert can't be seen because it's always exactly under wtema, except a little after block 5780 8780 where wtema-144 jumps away from the others (which needs to be investigated).

The "equivalent std dev" numbers I gave are barely to 2 significant figures and since the only difference above is mostly in the in the 3rd significant figure, it's still hard to determine if any is better than the others.

It's a mystery that your lwma and wt need the same N. I have two completely different programs that show N_wt = 190/144 * N_lwma gives the same std dev (constant HR) and look nearly identical to step responses like this:

[jacob-eliosoff]

@jtoomim, your comments on wt-144 and wtema vs asert all make sense to me.

I didn't spend a lot of time on the integer math hacks in simpexpi etc so I wouldn't be surprised if @markblundeberg's approximation is more robust.

The confirmation time is the expected amount of time needed to confirm a transaction issued at a random time. This is greater than the average block interval because a random instant is more likely to be in a long block interval than a short one.

Hmmm. It seems to me that with stable hashrate, the expected confirmation time should be exactly 10 minutes. (You're more likely to start waiting during a long block than a short one; but you also start on average halfway through the block rather than at its beginning, and these two factors are supposed to exactly balance out.) Maybe what this reflects is that when hashrate and thus difficulty adjusts, block times get more likely to be very long (also very short but that's irrelevant), which bumps up confirmation times for the reason you described? So, the more hashrate shenanigans, the higher the expected conf time?

[jtoomim]

with stable hashrate, the expected confirmation time should be exactly 10 minutes. ... the more hashrate shenanigans, the higher the expected conf time?

Yes, that's correct. If I get rid of the switch miners, confirmation times get very close to 600 seconds (+/- 1.2%) for all algorithms. I'm using confirmation time as a summary metric of how much the block intervals deviate from an ideal Poisson process.

[jtoomim]

Notice asert can't be seen because it's always exactly under wtema, except a little after block 5780 8780 where wtema-144 jumps away from the others (which needs to be investigated).

That's probably just the greedy hashrate switching off slightly earlier for wtema-144 than for the other algorithms. Prior to that jump, wtema-144 was ever so slightly lower than lwma-144 and asert-144, which means the revenue ratio could have fallen below the 1.03 threshold for the 1 EH/s of greedy hashrate one block earlier. That, in turn, could have resulted in one or two unusually long block intervals, triggering the following jump in profitability.

[zawy12]

Your metric implies the larger the N, the better. There must be some counter-force to that. There needs to be a metric that balances the need for stability (high N) with the need to respond quickly to BTC/BCH price variation (low N). My intuitive assumption has been that we want to set N so that accidental variation is equal to the expected price variation (both in % per day). If we want accidental variation to be 1/2 of price variation, then we would need N to be 4x larger which would make it 4x slower to respond to price changes.

[jtoomim]

Your metric implies the larger the N, the better.

No, it implies that the optimum N is near or above the high end of the range that was tested.

Confirmation times

Algorithm	Block interval (sec)	Confirmation time (sec)
asert-144	599.80	815.30
asert-288	599.59	687.08
asert-342	599.51	681.79
asert-407	599.41	677.11
asert-484	599.28	680.00
asert-576	599.13	705.52
asert-685	598.94	685.50
asert-815	598.71	749.85
asert-969	598.46	710.08
asert-1152	598.20	748.44
asert-2304	596.89	907.92

Profitability of different mining strategies

Algorithm	Greedy	Variable	Steady
asert-144	-0.107%	0.011%	-0.820%
asert-288	-0.011%	0.005%	-0.361%
asert-342	-0.003%	0.004%	-0.329%
asert-407	0.003%	0.002%	-0.311%
asert-484	0.001%	0.002%	-0.307%
asert-576	0.003%	0.003%	-0.318%
asert-685	0.025%	0.002%	-0.318%
asert-815	0.024%	0.002%	-0.352%
asert-969	0.033%	0.003%	-0.381%
asert-1152	0.036%	0.004%	-0.423%
asert-2304	0.094%	0.007%	-0.724%

[dgenr8]

@jacob-eliosoff I stressed in the livestream the importance of not confounding a blocktime with any prior difficulty other than the one at which that block was mined. The original BTC algo, wtema, ema, wt, and asert all have this property (in the case of asert, it's because no prior difficulties are referenced at all). cw and lwma do not have this property.

@jtoomim Nice to see the step function test. I agree with @zawy12 that both direction of step function should be tested. The magnitude of the profitability step should be at least as extreme as anything ever seen over the space of an hour or so.

[zawy12]

Why is greedy so low? Using last year's miner motivation simulation (6x HR if D is 1.30x baseline HR's needed D and turning off at 1.35x) and if BTC/BCH price is a constant, I see greedy miners with ASERT-484 getting 1% higher target/time (1% more profit) for 1/3 of the blocks (compared to constant miners) and all miners averaging 2.5% profit per block from the slower solvetime. With ASERT-144, the greedy get 1.7% higher target for 1/3 of the blocks but there is not the additional 2.5% lower solvetime being paid out to all miners.

My old LWMA N=60 under these conditions has correct solvetime, but this on-off miner scenario gets 7.7% higher target for 1/3 of the blocks. LWMA is also 1.7% at N=288 (ASERT N=144), so it seems I chose N 1/4 or 1/5 of what he needed to be for a lot coins.

[jtoomim]

Why is greedy so low?

Probably because in my simulation, most of the equilibration is done by the variable category, which will switch hashrate between both chains in a memory-ful and proportional (vs instantaneous all-or-none) fashion. This emulates the behavior of pools like Bitcoin.com, and also emulates the behavior of an aggregation of many different greedy miners with different thresholds. With proportional miners and a good DAA, greedy miners rarely get a chance to turn on even if their threshold is as low as 3% higher profitability, as it is in my simulation. A while ago, rinexc did an analysis of BTC.top's and Unknown's greedy mining strategies, and found them to switch at 3% and 5% higher profitability, respectively.

I spent about 5 hours tweaking the variable hashrate algorithm to try to make it model the behavior of an aggregate of self-interested miners. I don't think I succeeded perfectly, but it works much better than the original kyuupichan variable algorithm, and actually ensures that profitability remains close to 1.0 most of the time, and usually within ±3% of 1.0. But it doesn't prevent the CW oscillations at all.

[jtoomim]

both direction of step function should be tested.

@dgenr8 Both directions are tested by the code. I had just zoomed in on a positive step in the screenshot. Here's a negative step for you:

And here's another one, but with a wider range of ASERT N values so you can taste the rainbow:

I've also pushed my LWMA etc code and updated http://ml.toom.im:8051/, so if you want to look at positive and negative steps in more detail, you can do so there. Or, even better, run a copy locally for better performance. It gets pretty laggy when you have a half dozen algorithms with more than 5k blocks each.

[zawy12]

I can't understand the inputs to your form very well. My simple method has been able to mimic all the oscillations I've seen. I never thought to try modelling the effect of random changes in exchange price. This might be partly because I want to make random D variation equal price variation. So when I model the effects of miner motivation on random D variation, it's same as price variation if D were constant. So I'm missing the effect of the combination.

[jacob-eliosoff]

I don't have a strong sense of the tradeoffs of high N (stability) vs low N (responsiveness). Qualitatively, excessive stability is dangerous because 1. it leads to more "incorrectly cheap/expensive" blocks when hashrate/price moves around; and 2. a plunge in hashrate (eg post-split) could take days/weeks to recover from, if at all, one of the more realistic "chain death" scenarios. Whereas excessive responsiveness leads to noisy swings in difficulty, which 1. may increase frequency of long confirmations when difficulty bumps too high sometimes, and 2. may be easier for miners to game (though all the good algos here should greatly reduce that).

@jacob-eliosoff I stressed in the livestream the importance of not confounding a blocktime with any prior difficulty other than the one at which that block was mined. The original BTC algo, wtema, ema, wt, and asert all have this property (in the case of asert, it's because no prior difficulties are referenced at all). cw and lwma do not have this property.

@dgenr8, was this in response to something I said above? I'd think the nearer an algo is to a pure (memoryless) EMA, the less it should depend on difficulties (or even block times) before the last. So I'm all for avoiding these dependencies when possible.

[dgenr8]

@jacob-eliosoff As you seem to imply, noisy difficulty itself is not a problem, extreme blocktimes are. The tips of the tightrope walker's pole move crazily so that he can remain steady. This is seen with RTT.

You said something about weighting longer times more heavily. I guess it was in reference to something else. Anyway, I agree with you -- that is not justified!

[jacob-eliosoff]

Ah OK yeah that. I just meant that any DAA that calculates a weighted average across blocks mustn't give longer blocks greater weights. One way to think of what EMA-type DAAs do is:

1. Calculate the weighted average block time of recent blocks, weighting more recent blocks more heavily.
2. If the average is > 10 min, decrease difficulty; if it's < 10 min, increase difficulty.

In that weighted average of block times, more recent blocks get greater weight, but longer blocks don't - this is important! That's all I was trying to say.

[jacob-eliosoff]

@jtoomim, I'm struggling with git but if you copy my old exp_int_approx() function, then I think aserti, an integer-math version of asert, could be done as follows. Not sure if this ends up any different from simpexpi.

def next_bits_asert_int_approx(msg, tau):
    blocks_time = states[-1].timestamp - states[0].timestamp
    height_diff = states[-1].height - states[0].height
    orig_target = bits_to_target(states[-0].bits)
    decimals = 9
    scaling = 10**decimals
    next_target = orig_target * exp_int_approx(scaling * (blocks_time - IDEAL_BLOCK_TIME*(height_diff+1)) // tau, decimals) // scaling
    return target_to_bits(next_target)

...

    'aserti-144' : Algo(next_bits_asert_int_approx, {
        'tau': (IDEAL_BLOCK_TIME * 144),
    }),

[zawy12]

Due to the potential of inflation if not getting stuck, I prefer ASERT 144 over 484. Here is the difference in stability under constant hashrate:

Here's kind of the worst case situation for ASERT-484 that I've seen in small coins. This is what will happen if 6x HR becomes interested in it when it drops 5% lower than average, but does not leave until it's 5% above average. It's not likely to happen to BCH under current conditions thanks to its liquidity.

This is a more realistic situation where D drops 2% below avg and the 6x HR does not leave until it's 2% above average. (144 and 484 tie)

Due to BCH's liquidity and these longer averaging windows, step functions are not as applicable as the situations I've modeled in the past. It was literally 20x HR switching on and off. With more liquidity and more diversity of mining pools deciding more gradually to switch, switching on and off modelling is not likely to work as well (it works well for the SMA problem but that's just the DA's fault). In this case we probably need an S curve like the logistic function for "variable level of miner motivation". Jonathan's method for variable HR is:

var_HR_fraction = (1+ VAR_PERCENT/100 - revenue_ratio*VARIABLE_EXP) 50/VAR_PERCENT This allocates 50% of variable HR if revenue_ratio = 1. rev_ratio is BTC/BCH, which is the reciprocal of how I would like to think of it. I will use the reciprocal because I want to think in terms of "desire to switch to BCH from BTC") VAR_PERCENT is sort of the +/- amount of rev_ratio at which it will allocate 0% to 100% of var_HR.

That is hard for me to see when using it, so here's the logistic function in comparison in the plot below. Numbers in parenthesis are the values. The aggressiveness factor is like a combination of VARIABLE_EXP and VAR_PERCENT.

It seems like there should be a channel in the middle where motivation is flatter (everyone is generally happy with current choice and not wanting to switch) instead of being at the maximum slope. I don't see a simpler way than what Jonathan did which is to let the user select how many "blocks of revenue_ratio" to avg (like 1 to 5) before letting it motivate the miner to switch.

It would be good to include the cost basis in revenue_ratio_reciprocal. This allows an automatic increase in aggressiveness if BTC and BCH price gets close to the cost to operate. I'll call this "miner profit ratio" = MPR. MPR = (BCH_reward_per_D - cost_per_D) / (BTC_reward_per_D - cost_per_D) where reward and cost are in the same unit of value (like USD) and per_D is equivalent to per_hash or "times target/2^256". This means that if BTC hashing cost is 1/2 the revenue/D, then 50% higher BCH revenue/D is 2x the profit. If cost_basis is 1/2, then to keep the previous plot, aggressiveness is 3.5 instead of 7.

To summarize:

HR = baseline_HR + variable_HR /[1+e^( -aggressiveness*(avg(MPR) - 1) ]

Where baseline_HR is like BTC top or whoever is willing to stay under harsh conditions.

[jtoomim]

@zawy12 You're missing part of how the variable miners work in my sim: they also have memory. There are three main lines of code:

var_fraction = (high - mean_rev_ratio**params['VARIABLE_EXPONENT']) * scale_fac
memory_frac = states[-1].memory_frac + ((var_fraction-.5) * params['MEMORY_GAIN'])
var_fraction = max(0, min(1, var_fraction + memory_frac))

https://github.com/jtoomim/difficulty/blob/comparator/mining.py#L420

Each block, some proportion of the variable miners will convert into or out of "memory miners", who continue to mine the same coin for an extended period of time. Over a long enough period of time, and assuming 0 random noise or price fluctuations, the memory miners will be enough so that the revenue ratio becomes exactly 1.0.

The memory miners are intended to model miners who look back at the average profitability over the last few blocks, days, or weeks, and manually switch chains based on average profitability of the different chains.

I added this mechanism because without it (i.e. in kyuupichan's original sim), the variable miners would reach an equilibrium other than 1.0, and the exact equilibrium would depend on the total amount of hashrate that the variable miners had access to. With 3 EH/s, the equilibrium might be 0.99, but with 30 EH/s, the equilibrium would shift down to e.g. 0.90.

Because the memory_frac is doing most of the work at equilibrium, the first line is only really important for the short-term responsiveness and for determining the rate at which miners enter or leave memory_frac.

In this case we probably need an S curve like the logistic function for "variable level of miner motivation".

There are some problems with the current formula. The main issue with the current formula is that if the price is low and a small proportion of the variable miners are active (e.g. <5%), they have a tendency to switch off suddenly and entirely. Switching to a different motivation->hashrate mapping curve would address that issue. I was actually thinking of making it an exponential function, not a logistic function, as the asymptotic upper limit for logistic curves doesn't really make sense here. A logistic curve might make it so that there was 99% as much hashrate at BCH/BTC=1.15 as there is at 2.00, which is not realistic. So either a plain exponential function, or a logistic function with an x-axis offset.

Where baseline_HR is like BTC top or whoever is willing to stay under harsh conditions.

Actually, that's not BTC.top's algorithm at all. BTC.top will leave immediately if profitability is low. However, they come back if it has been 1 hour since the last block. So they help cause moderate increases in block intervals, but they actively prevent them from getting excessively long.

I prefer ASERT 144 over 484. Here is the difference in stability under constant hashrate:

Under constant hashrate, diff fluctuation is proportional to 1/sqrt(N). But under variable hashrate, that's not true at all, and fluctuation seems to be proportional to 1/N.

If you have a randomly short block interval, that will increase diff and reduce profitability by an amount proportional to 1/N. With constant hashrate, the next block's interval will be statistically independent, so there's a 50% chance that the next block will also be short and the fluctuation will be increased. But with variable hashrate, the next block's interval is statistically dependent: reduced profitability will quickly chase away hashrate, and cause the next block's interval to tend to be long. This feedback loop means that what matters most in a scenario with variable hashrate is the 1-block noise amount, not the N-block noise amount. It doesn't matter how much noise there is in the entire window; all that matters is the noise from the blocks entering and (for CW) leaving the window. Since the 1-block noise is divided by the window size, the 1-block noise is roughly 1/N.

I suspect this mechanism may be why your testing fails to show an advantage for longer windows/half-lives.

[zawy12]

Under constant hashrate, diff fluctuation is proportional to 1/sqrt(N). But under variable hashrate,

Right. I was just showing how much extra fluctuation we could expect from the lower 144 when revenue_ratio is fairly constant [exchange rate and HR are constant].

A logistic curve might make it so that there was 99% as much hashrate at BCH/BTC=1.15 as there is at 2.00, which is not realistic.

The upper limit is because there is only so much HR that can switch. There is only so much HR in the world.

Actually, that's not BTC.top's algorithm at all.

But the net effect is the same. I meant baseline_HR was kind of figuratively part of what BTC top is supplying.

I think that in my S-curve memory miner will not be needed. The original problem looks like it may have resulted from the lack of symmetry shown in my plot. My S-curve is symmetrical, so it might come out good.

Maybe mine shouldn't have a baseline. By changing aggressiveness, it smoothly transitions from constant miner to variable to greedy.

[jtoomim]

@jacob-eliosoff I want to do an integer exponential approximation that is:

1. Binary, not decimal
2. Based on a simple polynomial, with little to no division
3. Uses 2^x, rather than e^x
4. Uses bitshifting and the 2^x = 2 * 2^(x-1) identity to restrict the function's domain to the [0, 1) interval

Once we add bitshifting to handle out-of-range inputs, I suspect that for any function f(x), it will work well enough for our purposes as long as f(0) == 1, f(1) == 2, and f(x) is monotonically increasing for 0<x<1. It would probably even work if we just used f(x) = 1+x, though I expect we'd get a bit better performance if we were a little more sophisticated (e.g. f(x) = 1+2x/3+x^2/3).

Because ASERT is non-recursive, we don't have to worry about accumulation of error from our approximation. That means our exponential approximation does not need to be particularly accurate; it just needs to be smooth and monotonic. If our approximation is off by no more than 0.3%, that means that the targeted timestamp offset for any given difficulty will be off by at most 0.3% of tau, which doesn't really matter in terms of the mining incentives.

[zawy12]

I suspect this mechanism may be why your testing fails to show an advantage for longer windows/half-lives.

I can choose other miner motivations where longer window is definitely better. Your greedy option should be able to show my examples bad outcomes.

Shorter windows are safer in the sense there can't be a long series of delays, especially if you use clipping to prevent spam attacks. At a high enough N, ASERT will be like an SMA in the sense it's not estimating current HR as much as it' measuring HR as it was in the past. It's better because it does not "ring", directly motivating the miners to come back. My examples show the ASERT oscillation rate is a function of miner revenue_ratio "stickiness". For the 5% stickiness example, they got about 60 blocks and came back in 120 blocks (as shown) to do it again. The hope is that BCH's higher liquidity and diversity of miners will prevent it from being like this. This is common in alt coins who have <10% of the total HR for their POW.

[jtoomim]

I think that in my S-curve memory miner will not be needed.

I tried a simple logistic curve without memory in February. Memory is still is needed.

The issue is that even with a logistic curve, there's no reason why hashrate will adjust until rev_ratio is exactly 1.0. For example, if the price ratio is 1%, and 100% of the SHA256 hashrate is following a logistic curve, then the revenue ratio will stabilize at whatever point on the logistic curve causes 1% of the hashrate to work on BCH. If aggressiveness is 10, that would mean a profitability of about 54%: 1/(1+e**(-10*(.54-1))) == 0.00995. But if only 10% of the SHA256 hashrate is following a logistic curve, then it would stabilize at a profitability of around 78%.

What's needed is a feedback loop with inertia, in which people switch chains and stay based on profitability.

[zawy12]

I tried a simple logistic curve without memory in February. Memory is still is needed.

The issue is that even with a logistic curve, there's no reason why hashrate will adjust until rev_ratio is exactly 1.0.

Why should rev_ratio_reciprocal stabilize to 1? We're seeing 80% to 95% variable HR that has a higher requirement (that is, their definition of rev_ratio_reciprocal = 1 is higher, i.e. they require more BCH) than the constant miners.

Your inertia seems to be what I call stickiness or hysteresis. My idea was to average like I thought you/Neil were doing rev_ratio. My step functions are the logistic function with max aggressiveness. The hysteresis is given by my start and stop points as a fraction of the difficulty that is correct for the constant miners. The bad oscillations occur when both of these factors are above 1.0 instead of being like 0.95 start and 1.05 stop. I mimicked BCH last year with 1.30 start and 1.35 stop. I've seen it several times in coins with bad oscillations in LWMA N=60 (ASERT N=30). They would mine on and off for only 5 to 15 blocks which drove D up about 10% to 20%, then stop for 20 or 30 blocks, then do it again all day (720 blocks). If this occurs with ASERT-484 they will get 50 to 200 blocks at a time followed by really long delays. This is if only 10% of the HR wants the coin at 10% higher difficulty and the other 90% decide to come on & off largely together (high aggressiveness factor with 10% hysteresis). To transition my step functions to an S-curve I need an x-axis offset for the difference between start and stop.

If the 90% variable HR has a higher rev_ratio_recip than constant miners, then the S-curve may need an x-axis offset like you mentioned.

The reasons for the flattening at top and bottom of the S-curve are different, so there's no reason to think what we need should be so nicely symmetrical. But I would not know how to estimate what the top and bottom flattenings should be like and I believe the flattenings like this are in the right direction.

[jacob-eliosoff]

@jtoomim, really all I know about integer-math approximations is what I learned when Googling around for that original exp_int_approx() function, but fwiw that function is easy to adapt to use bits rather than decimals:

def exp_int_approx(x, bits_precision=20):
    """Approximates e**(x / 2**bits_precision) using integer math, returning the answer scaled by the same number of bits as the input.  Eg:
    exp_int_approx(1024,   10) ->    2783       (1024/2**10 = 1; e**1 = 2.718281 =~ 2783/2**10)
    exp_int_approx(3072,   10) ->   20567       (3072/2**10 = 3; e**3 = 20.0855 =~ 20567/2**10)
    exp_int_approx(524288, 20) -> 1728809       (524288/2**20 = 0.5; e**0.5 = 1.6487 =~ 1728809/2**20)"""
    assert type(x) is int, str(type(x))                                             # If we pass in a non-int, something has gone wrong

    scaling, scaling_2 = 2**bits_precision, 2**(2*bits_precision)
    h = max(0, int.bit_length(x) - int.bit_length(scaling) + 4)                     # h = the number of times we halve x before using our fancy approximation
    term1, term2 = 3 * scaling << h, 3 * scaling_2 << (2*h)                         # Terms from the hairy but accurate approximation we're using - see https://math.stackexchange.com/a/56064
    hth_square_root_of_e_x = scaling_2 * ((x + term1)**2 + term2) // ((x - term1)**2 + term2)

    e_x = hth_square_root_of_e_x                                                    # Now just need to square hth_square_root_of_e_x h times, while repeatedly dividing out our scaling factor
    for i in range(h):
        e_x = e_x**2 // scaling_2
    return e_x // scaling                                                           # And finally, we still have one extra scaling factor to divide out.

[zawy12]

@jtoomim You mentioned BTC.top was helping BCH a lot during periods of high difficulty. Back in January I was seeing them as being the smartest on-off miner, getting blocks with lower difficulty than all the other miners and thereby probably helping to cause the oscillations. I teased Jiang about claiming it was Poolin switching for profit and recommended he start mining the high difficulty blocks if he wanted to help.

https://twitter.com/zawy3/status/1223774719838650368?s=20

[markblundeberg]

Regarding binary integer (or perhaps binary fixed point) approximations to exp, here is one I came up with a while ago:

(from: https://twitter.com/MarkLundeberg/status/1191831127306031104 )

The axes are a bit confusing, so to explain: if x is in the interval [0, 1), and we want y = 2^x, then y is in the interval [1,2). We can quantize both these x and y intervals with 1/65536 resolution (16 bits). For any input x outside this interval we can of course separate out this nonfractional part, which produces a simple exponent offset on the resulting y. For estimating exp(x) you just need to map over the x values by multiplying with the approximation of log(2), or better, just forget about exp(x) and think about the protocol in terms of 2^x -- the characteristic relaxation time is now a half-life rather than a 1/e life.

In this case we can do a nice trick which is to perform all our arithmetic with a 2^48 mulitiplier, using 64-bit integers, which is great considering our current architectures love 64 bits. So, this is what the formula in the figure does: takes 16-bit N values (N = 65536 * frac(x)), performs a variety of arithmetic which is purely within 64 bits to yield a 64-bit-precise y value, then round it back to a 16-bit y value.

To be even more concrete, let's say x = 0.5 = 32768/65536, i.e., N = 32768. Then this approximation gives 2^x = 1 + 27317/65536 = 1.41409, which is pretty close to 2^0.5 = 1.41421.

I obtained these coefficients by fitting the curve and minimizing relative error, in floating point, then rounding the resulting coefficients. IIRC I also included constraints that the output must be exact for the ends of the interval (x=0 and x=1), so really there were only two parameters during the fit. As you can see the accuracy is decent (I'd say good enough), at worst ~0.01% off. Since the ends are exact, there is no "discontinuity" in the function as you move from one range to the next, e.g., from x just below 1 to x just above 1. The lack of discontinuity probably doesn't matter but I think it's nice.

Note also that bitcoin nBits sometimes only has 16 bit resolution (when target is near a power of 256) so I think this is entirely appropriate.

(this isn't the only way to do this, but it at least gives an idea of what is reasonable in a very speedy approach)

[markblundeberg]

Just to be clear about the previous comment -- I don't think it's worth worrying about the exact form of the exponential approximation formula until it comes time to really specify.

Also, is 0.01% good enough? Remember the difficulty algo is a feedback algorithm, and every step of the way it naturally recalibrates away any errors that arise (and errors always do arise even now, at very least from the nBits rounding!).

It is of course true that in a relative algo, the errors gradually build up by drifting over time, but this is Brownian drift where the positive and negative movements tend to mostly cancel. If you only have a +-0.01% error per block, this is extremely small drift, especially compared to the current drifts we get using non-exponential algorithm. After 10 years (500k blocks), a random uncorrelated +-0.01% error on each block would only amount to +/-42 seconds of total drift (0.01% 600 seconds sqrt(500k)).

The absolute exponential algorithm does get rid of the drift completely, but, given the drift in the relative version is so small, I think this is an insignificant advantage. If you find the relative exponential version easier to implement, then do that.

[zawy12]

I will be away from computers for about a week so I won't be able to work on this much, but I have this direction in mind. Model 5 miners with equal hashrate and with MPR shifted 1% each based on different  cost basis so that the miner with the lowest MPR is 4% less than the one with the highest. Also assume they have the same stickiness value which is 5 or 6% and centered on their individual MPR. When you add these up they have the classical hysteresis curve which I plan to use to replace my difficulty step functions. MPR in the middle might be the hashrate weighted mean of the miners

[jacob-eliosoff]

I'm still struggling with git but I just pushed an algo asertd (next_bits_asert_discrete()) that approximates asert's exponentiation via discrete integer multiplications:

    Another exponential-decay-based algo that uses integer math instead of exponentiation.  As with asert, we increase difficulty by a fixed amount each time a block is found,
    and decrease it steadily for the passage of time between found blocks.  But here both adjustments are done in integer math, using the principles that:
    1. We can discretize "Decrease difficulty steadily by a factor of 1/e over (say) 1 day", into "Decrease difficulty by exactly 1/e^(1/100) for each 1/100 of a day."
    2. We can closely approximate "difficulty * 1/e^(1/100)", by "(difficulty * 4252231657) >> 32".  (Really, just "difficulty * 99 // 100" would probably do the job too.)

Can someone remind me of good command-line parameters to pass in to mining.py?

@markblundeberg, thanks for your comments, I'll give them a think. My guess is 0.01%, or even 0.1%, does no practical harm. Around 1% is where I'm guessing miners might start to take note, and even then, "blocks coming out 1% too fast/slow on avg" is very different from "miners can make 1% more money by oscillating hashrate". Isn't the historical avg block time for BTC already 9.x minutes just because of hashrate trending upward over time?

(UPDATE: I just checked: in the 10+ years between BTC block 32490 (12 am morning of Jan 1 2010) and this afternoon's block 635596, 603106 blocks were published in ~330377100 seconds for an average of 9 min 8 secs per block.)

FYI, one thing I noticed while testing asertd is that just converting back and forth from target to bits trims target by about 0.001% per round trip.

[jtoomim]

@jacob-eliosoff

Can someone remind me of good command-line parameters to pass in to mining.py?

pip3 install dash
python3 dashdiffsim.py

Then browse to localhost:8050

[jacob-eliosoff]

Thanks!

[jtoomim]

I've implemented a 1st, 2nd, and 3rd-order polynomial approximation of asert. I used my two super simple for the 1st and 2nd-order approximations, and @markblundeberg's 3rd-order approximations in aserti1, aserti2, and aserti3 respectively.

Here's my and Mark's approximatiions and @jacob-eliosoff's discrete asertd-144 facing off against the floating point asert-144 for 5k blocks:

Zoomed in on a point where the price increased 15%:

Here are some stats for a 50k block run:

Confirmation times

Algorithm	Block interval (sec)	Confirmation time (sec)
asert-144	599.80	815.30
asertd-144	599.86	826.16
aserti1-144	599.74	831.64
aserti2-144	599.80	808.34
aserti3-144	599.80	814.09
wtema-144	603.41	819.78
ema2-1d	599.84	811.20
emai-1d	596.29	805.40
emai2-1d	600.06	813.30
simpexp-1d	599.87	814.85
asert-288	599.59	687.08
aserti1-288	599.46	717.23
aserti2-288	599.58	684.28
aserti3-288	599.59	686.71
wtema-288	601.06	685.92
asert-576	599.13	705.52
aserti1-576	598.83	683.83
aserti2-576	599.10	690.47
aserti3-576	599.13	705.43
wtema-576	600.10	693.97

Note the block interval is longer for wtema (especially wtema-144) than for all of the ASERT variants.

Profitability of different mining strategies

Algorithm	Greedy	Variable	Steady
asert-144	-0.107%	0.011%	-0.820%
asertd-144	-0.107%	0.010%	-0.854%
aserti1-144	-0.114%	0.011%	-0.906%
aserti2-144	-0.106%	0.011%	-0.801%
aserti3-144	-0.105%	0.011%	-0.819%
wtema-144	-0.109%	0.011%	-0.811%
ema2-1d	-0.105%	0.010%	-0.812%
emai-1d	-0.100%	0.010%	-0.824%
emai2-1d	-0.106%	0.011%	-0.815%
simpexp-1d	-0.107%	0.011%	-0.820%
asert-288	-0.011%	0.005%	-0.361%
aserti1-288	-0.031%	0.007%	-0.406%
aserti2-288	-0.009%	0.004%	-0.358%
aserti3-288	-0.010%	0.005%	-0.361%
wtema-288	-0.008%	0.004%	-0.358%
asert-576	0.003%	0.003%	-0.318%
aserti1-576	0.006%	0.003%	-0.313%
aserti2-576	0.009%	0.002%	-0.310%
aserti3-576	0.003%	0.003%	-0.318%
wtema-576	0.002%	0.003%	-0.310%

All of the algorithms perform well. aserti1's performance is a bit worse than the rest, but it still does surprisingly well given that aserti1's 2^x ≈ 1 + x approximation has up to 6% error. I think this illustrates how insensitive ASERT is to approximation inaccuracy. However, aserti2 (whose approximation has no more than 0.3% error) is clearly better, and aserti3 (0.01% error) is nearly indistinguishable from floating-point asert. asertd performance is in between aserti1 and aserti2.

Also, it's worth noting that 158 600 ln(2) ≈ 2^16, and 316 600 ln(2) ≈ 2^17, so if we choose one of those two numbers (or a power-of-two multiple) as our tau value, we can avoid dividing by tau entirely.

@jacob-eliosoff, asertd crashes if I try to change the tau value. Haven't checked why.

I've pushed this code to https://github.com/jtoomim/difficulty/tree/comparator and updated http://ml.toom.im:8051.

[jacob-eliosoff]

Thanks @jtoomim. asertd relies on a constant being precalculated and hardcoded for each window size (that's how it avoids the floating-point dependency), but for testing purposes I've pushed a fix that calculates them. asertd-288 etc should work out of the box now.

Speaking of out-of-the-box, dashdiffsim worked perfectly for me the first time I tried it. This is especially pleasant after doing battle with git.

Given how similar performance is for all these exp DAAs compared to the non-exp DAAs (especially cw), we should be choosing between them less based on performance and more based on 1. simplicity/clarity and 2. handling of edge cases.

[dgenr8]

The results look good. My worry however is that we are not emphasizing pathological situations such as a 10x hashrate attack or minimum timestamp attack. These are things that would be addressed by RTT.

A nod at least in this direction would be to model price ratio shocks of 10x (currently we model 1.15x), if only to document the limits of these algos.