Set outlier filtering to 1.1 standard deviation (was/currently 2.2).
Refactor the outlier calculation code, improving layout and logging.
Add new tests for outlier filtering.
Refactor the exchange service tests.
Details:
At the moment, outlier prices are anything outside 2.2 standard deviations from the average. The number 2.2 was initially chosen to cause the least amount of filtering on current valid market prices.
A recent flash crash was caused by a 0 BRL price retrieved from MercadoBitcoin. The 2.2(stdev) is huge, resulting in all prices considered "ok / within range" even when plainly not the case. Example price set (129000, 131000, 0). Average=85966, Stdev=60787. Using 1.1(stdev), 0 would be removed as an outlier.
Here is an example of normal market filtering. The lower/upper bounds calculated from 1.1(stdev) are quite tight, especially when the prices cluster in a tight range.
It seems 2.2(stdev) was too permissive, the aim to avoid filtering an incorrect approach. 1.1 would narrow the accepted range, causing outlier filtering to be more used. This is ok, it tends to more accurate pricing.
Details:
At the moment, outlier prices are anything outside 2.2 standard deviations from the average. The number 2.2 was initially chosen to cause the least amount of filtering on current valid market prices.
A recent flash crash was caused by a 0 BRL price retrieved from MercadoBitcoin. The 2.2(stdev) is huge, resulting in all prices considered "ok / within range" even when plainly not the case. Example price set (129000, 131000, 0). Average=85966, Stdev=60787. Using 1.1(stdev), 0 would be removed as an outlier.
Here is an example of normal market filtering. The lower/upper bounds calculated from 1.1(stdev) are quite tight, especially when the prices cluster in a tight range.
Conclusion:
It seems 2.2(stdev) was too permissive, the aim to avoid filtering an incorrect approach. 1.1 would narrow the accepted range, causing outlier filtering to be more used. This is ok, it tends to more accurate pricing.