search

c9s / bbgo

The modern cryptocurrency trading bot framework written in Go.
https://bbgo.finance
GNU Affero General Public License v3.0
1.21k stars 291 forks source link

Backtest repetition gives different results randomly #1007

Open head-and-tail opened 1 year ago

head-and-tail commented 1 year ago

BACK-TEST REPORT shows different results for the same strategy when the strategy is run consecutively several times (same bbgo version and .yaml). Sometimes 1-3 repetitions are sufficient, sometimes more are needed. Number of trades, profit , sharpe ratio etc. vary randomly. This makes backtesting difficult because the results are not reliable and reproducible.

The latest release without the problem is v1.39.2, so releases after v1.40.0 all have the same bug (incl. latest git). I used the slim version for testing. I tested only bollmaker , trendtrader and factorzoo, but bug is probably reproducible with most strategies.

First bad commit:

commit ecc959835a50f471fe2d2a438b53c38358426b50 Author: zenix zenixls2@gmail.com Date: Mon Aug 29 19:46:58 2022 +0900

fix: cache params and kline until next kline 1m appears
zenixls2 commented 1 year ago

it's because of the randomness of golang map on iteration.
kline callbacks now won't follow strict ordering, which is much closer to the real-world case.
To avoid such randomness, we need to use SerialMarketStore instead of MarketDataStore.

zenixls2 commented 1 year ago

If you have exits in your config and use the interval other than 1m, the result is expected to be dynamic and may not be reproducible. If you really need a reliable one without changing code, try to configure using 1m.

Usually using interval 1m with window set to 5 * 21 has similar effect of setting interval 5m and window=21.