bisq-network / bisq-pricenode

GNU General Public License v3.0
6 stars 12 forks source link

Switch outlier price filtering to 1.1 standard deviation. #29

Closed ghost closed 1 year ago

ghost commented 1 year ago

Details:

At the moment, outlier prices are anything outside 2.2 standard deviations from the average. The number 2.2 was initially chosen to cause the least amount of filtering on current valid market prices.

A recent flash crash was caused by a 0 BRL price retrieved from MercadoBitcoin. The 2.2(stdev) is huge, resulting in all prices considered "ok / within range" even when plainly not the case. Example price set (129000, 131000, 0). Average=85966, Stdev=60787. Using 1.1(stdev), 0 would be removed as an outlier.

Here is an example of normal market filtering. The lower/upper bounds calculated from 1.1(stdev) are quite tight, especially when the prices cluster in a tight range.

COINGECKO ZAR outlier price removed:475414.534, lower/upper bounds:485522.0/487355.0, consensus price:486438.5 
KRAKEN DASH outlier price removed:9.78E-4, lower/upper bounds:9.83E-4/9.83E-4, consensus price:9.83E-4 
COINGECKO AUD outlier price removed:39163.826, lower/upper bounds:39262.7/39295.93, consensus price:39278.14333333333 
COINGECKO TRY outlier price removed:676134.357, lower/upper bounds:682530.0/684150.0, consensus price:683089.6666666666 
BINANCE EUR outlier price removed:23453.87, lower/upper bounds:23403.602/23430.0, consensus price:23413.0484 
COINGECKO NGN outlier price removed:1.885220855E7, lower/upper bounds:2.3466372E7/2.3549389E7, consensus price:2.35078805E7 
BTCMARKETS LTC outlier price removed:0.00201256, lower/upper bounds:0.0023397/0.00234333, consensus price:0.002341455 
BITFLYER JPY outlier price removed:3697907.0, lower/upper bounds:3686392.833/3694814.0, consensus price:3689435.611 
BINANCE GBP outlier price removed:20659.73, lower/upper bounds:20107.684/20125.1, consensus price:20115.3468 
IndependentReserve USD outlier price removed:25217.57, lower/upper bounds:25139.15/25168.0, consensus price:25147.6038 
COINBASEPRO ETC outlier price removed:5.87E-4, lower/upper bounds:5.86E-4/5.8601E-4, consensus price:5.860033333333334E-4 
BTCMARKETS ETH outlier price removed:0.06014627, lower/upper bounds:0.06174364/0.06182104, consensus price:0.061761811428571435 
KRAKEN ZEC outlier price removed:9.8E-4, lower/upper bounds:9.8091E-4/9.84E-4, consensus price:9.8297E-4 
BITFINEX XMR outlier price removed:0.005547, lower/upper bounds:0.005561/0.005563, consensus price:0.005562 
COINGECKO BRL outlier price removed:123990.78, lower/upper bounds:124513.84999999/124596.0, consensus price:124554.924999995 

Conclusion:

It seems 2.2(stdev) was too permissive, the aim to avoid filtering an incorrect approach. 1.1 would narrow the accepted range, causing outlier filtering to be more used. This is ok, it tends to more accurate pricing.

gabernard commented 1 year ago

@jmacxx The other PRs caused several merge conflicts. Please resolve these.

ghost commented 1 year ago

@gabernard Merge conflicts resolved.