bug: different STATISTICS result of the same algorithm & data on windows 10 v.s. Linux Ubuntu 18.04.1

[mw66]

Expected Behavior

same algorithm & data on windows 10 v.s. Linux Ubuntu 18.04.1 should return same STATISTICS results.

Actual Behavior

$ diff win10.txt linux.txt
8c8
< STATISTICS:: Sharpe Ratio 4.756
---
> STATISTICS:: Sharpe Ratio 11.181
17,18c17,18
< STATISTICS:: Information Ratio 0
< STATISTICS:: Tracking Error 0
---
> STATISTICS:: Information Ratio 11.181
> STATISTICS:: Tracking Error 0.322
40c40
< STATISTICS:: OrderListHash -122053106
---
> STATISTICS:: OrderListHash 86766778

Potential Solution

The number '11.181' seems suspicious:

$ grep -w '11.181' linux.txt win10.txt
linux.txt:STATISTICS:: Sharpe Ratio 11.181
linux.txt:STATISTICS:: Information Ratio 11.181

Based on this number, I think the bug is on the Linux side.

Reproducing the Problem

Hard to share algorithm & data.

If you tell me which files & where to debug, I can probably debug it.

System Information

windows 10 v.s. Linux Ubuntu 18.04.1

Checklist

• [x] I have completely filled out this template
• [x] I have confirmed that this issue exists on the current master branch
• [x] I have confirmed that this is not a duplicate issue by searching issues

[Martin-Molinero]

Hi @mingwugmail, the times we have seen this happen it was because of some non-deterministic operation performed by the algorithm which caused different runs to give different results, I'd suggest reviewing the algorithm with this in mind. If I recall correctly 'universe selection' selection was behind a few of these cases. Also might want to double-check both environments are using the same version of Lean, we recently merged a statistics fix https://github.com/QuantConnect/Lean/pull/4248.

[mw66]

Yes, it looks like a code out-of-sync issue. After I did a git pull on Windows 10, now the number on both system agree to 11.181.

But, I still somehow feel there maybe a bug somewhere:

1) before the pull, this number has always be 0: STATISTICS:: Information Ratio 0

and after the pull, it's: STATISTICS:: Information Ratio 11.181

exactly the same as: STATISTICS:: Sharpe Ratio 11.181

How can this be? any explanation?

2) this number alone STATISTICS:: Sharpe Ratio 11.181

seems too good to be true :-)

[Martin-Molinero]

Hi @mingwugmail, I'm glad the issue was just a missing git pull. At https://github.com/QuantConnect/Lean/pull/4248 we merged a fix for AnnualPerformance and TrackingError statistics which also caused a change in the Sharpe Ratios.

STATISTICS:: Sharpe Ratio 11.181

That Sharpe Ratio seems great! It's hard to tell without seeing the algorithm. Maybe it's of a really short backtest period?

Closing this issue since the original Lean issue has been discarded.

[mw66]

Clean check out the latest github Lean, and run the default BasicTemplateFrameworkAlgorithm, the log.txt shows:

20200410 17:34:28.246 Trace:: Debug: Your log was successfully created and can be retrieved from: /home/zhou/project/Lean/trunk/Launcher/bin/Debug/BasicTemplateFrameworkAlgorithm-log.txt
20200410 17:34:28.246 Trace:: BacktestingResultHandler.Run(): Ending Thread...
20200410 17:34:28.578 Trace::
STATISTICS:: Total Trades 3
STATISTICS:: Average Win 0%
STATISTICS:: Average Loss -1.01%
STATISTICS:: Compounding Annual Return 254.782%
STATISTICS:: Drawdown 2.200%
STATISTICS:: Expectancy -1
STATISTICS:: Net Profit 1.632%
STATISTICS:: Sharpe Ratio 8.371
STATISTICS:: Probabilistic Sharpe Ratio 66.555%
STATISTICS:: Loss Rate 100%
STATISTICS:: Win Rate 0%
STATISTICS:: Profit-Loss Ratio 0

I don't think this number is correct: considering these numbers in my quotes:

STATISTICS:: Average Win 0%
STATISTICS:: Average Loss -1.01%
...
STATISTICS:: Loss Rate 100%
STATISTICS:: Win Rate 0%
STATISTICS:: Profit-Loss Ratio 0

then how can:

STATISTICS:: Net Profit 1.632%
STATISTICS:: Sharpe Ratio 8.371

Is there an explanation?