QuantConnect / Lean

Lean Algorithmic Trading Engine by QuantConnect (Python, C#)
https://lean.io
Apache License 2.0
9.51k stars 3.22k forks source link

ProcessSplitSymbols() unexpected situation: IsTradable: False, Active: True #3310

Closed linkliang closed 4 years ago

linkliang commented 5 years ago

Expected Behavior

Exception statement https://github.com/QuantConnect/Lean/blob/master/Engine/AlgorithmManager.cs#L1207 should never be reached.

Actual Behavior

The exception mentioned above is thrown.

Reproducing the Problem

The following algorithm throws this error: https://www.quantconnect.com/terminal/processCache/?request=embedded_backtest_b98256fb456793f5e9932008d2b0e691.html

Checklist

gsalaz98 commented 5 years ago

I've cloned the algorithm to C#, but am not having it fail like it is in python around 2018-03-XX:

/*
 * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
 * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
*/

using System;
using System.Collections.Generic;
using System.Linq;
using QuantConnect.Data;
using QuantConnect.Data.Market;
using QuantConnect.Data.UniverseSelection;
using QuantConnect.Indicators;
using QuantConnect.Interfaces;

namespace QuantConnect.Algorithm.CSharp
{
    public class SplitSymbolExceptionRegressionAlgorithm : QCAlgorithm, IRegressionAlgorithmDefinition
    {
        private Dictionary<Symbol, OrderInfo> _currentSecurity = new Dictionary<Symbol, OrderInfo>();
        private Dictionary<Symbol, SymbolData> _indices = new Dictionary<Symbol, SymbolData>();
        private Dictionary<Symbol, SymbolDataADX> _indicesADX = new Dictionary<Symbol, SymbolDataADX>();
        private SecurityChanges _changes;
        private int _numberOfSymbols;

        /// <summary>
        /// Initialise the data and resolution required, as well as the cash and start-end dates for your algorithm. All algorithms must initialized.
        /// </summary>
        public override void Initialize()
        {
            SetStartDate(2017, 1, 18);
            SetEndDate(2018, 12, 30);
            SetCash(10000);
            _numberOfSymbols = 10;
            UniverseSettings.Resolution = Resolution.Daily;
            UniverseSettings.Leverage = 1;
            AddUniverse(CoarseSelectionFunction);
        }

        public IEnumerable<Symbol> CoarseSelectionFunction(IEnumerable<CoarseFundamental> coarse)
        {
            foreach (var cf in coarse.Where(x => x.Price > 1 && x.DollarVolume > 2500000))
            {
                if (!_indices.ContainsKey(cf.Symbol))
                {
                    _indices[cf.Symbol] = new SymbolData(cf.Symbol);
                }

                _indices[cf.Symbol].Update(cf.EndTime, cf.AdjustedPrice);
            }

            foreach (var index in _indices.Values.Where(x => x.IsAboveEMA && x.IsRsiCondition))
            {
                var symbol = index.Symbol;

                try
                {
                    _indicesADX[symbol] = new SymbolDataADX(index);
                    AddEquity(symbol.Value, Resolution.Daily);
                    UpdateADX(symbol);
                }
                catch (Exception e)
                {
                    Log(e.ToString());
                }
            }

            var filteredAdx = _indicesADX.Values.Where(x => x.AdxCondition && x.AtrCondition);
            var values = filteredAdx.OrderBy(x => x.SymbolData.Rsi).Take(_numberOfSymbols);

            return values.Select(x => x.SymbolData.Symbol);
        }

        public override void OnSecuritiesChanged(SecurityChanges changes)
        {
            _changes = changes;

            foreach (var security in changes.AddedSecurities)
            {
                var symbol = security.Symbol;

                if (!security.IsTradable)
                {
                    continue;
                }

                try
                {
                    var price = security.Close;

                    if (price > 0 && !_currentSecurity.ContainsKey(symbol))
                    {
                        _currentSecurity[symbol] = new OrderInfo(price, 0.0m, price + price * 0.03m, 0);
                    }
                }
                catch (Exception e)
                {
                    Log(e.ToString());
                }
            }
        }

        /// <summary>
        /// OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.
        /// </summary>
        /// <param name="data">Slice object keyed by symbol containing the stock data</param>
        public override void OnData(Slice data)
        {
            var deleteQueue = new List<Symbol>();

            foreach (var symbol in _currentSecurity.Keys)
            {
                var ticker = symbol.Value;
                try
                {
                    if (!data.ContainsKey(symbol))
                    {
                        continue;
                    }

                    var price = data[symbol].Close;

                    if (_currentSecurity[symbol].EntryPrice < price - price * 0.01m &&
                        _currentSecurity[symbol].BarsSinceEntry == 1)
                    {
                        SetHoldings(symbol, 0.1);
                    }
                    else if (price > _currentSecurity[symbol].ProfitPrice ||
                        price < _currentSecurity[symbol].StopPrice ||
                        _currentSecurity[symbol].BarsSinceEntry > 4)
                    {
                        Liquidate(symbol);
                        deleteQueue.Add(symbol);
                        _currentSecurity[symbol].BarsSinceEntry = 0;
                    }
                    else
                    {
                        if (_currentSecurity[symbol].BarsSinceEntry > 2)
                        {
                            UpdateADX(symbol);
                            _currentSecurity[symbol].StopPrice = price - _indicesADX[symbol].Atr.Current.Value / price * 2.5m;
                        }

                        _currentSecurity[symbol].BarsSinceEntry++;
                    }
                }
                catch (Exception e)
                {
                    Log(e.ToString());
                }
            }

            foreach (var symbol in deleteQueue)
            {
                _currentSecurity.Remove(symbol);

                _indices.Remove(symbol);
                _indicesADX.Remove(symbol);

                RemoveSecurity(symbol);
            }
        }

        private void UpdateADX(Symbol symbol)
        {
            try
            {
                AddEquity(symbol.Value, Resolution.Daily);
                var tradeBarHistory = History(symbol, 7, Resolution.Daily);

                foreach (var tradeBar in tradeBarHistory)
                {
                    _indicesADX[symbol].Update(tradeBar);
                }
            }
            catch (Exception e)
            {
                Log(e.ToString());
            }
        }

        private class OrderInfo
        {
            public decimal EntryPrice;
            public decimal StopPrice;
            public decimal ProfitPrice;
            public int BarsSinceEntry;

            public OrderInfo(decimal entryPrice, decimal stopPrice, decimal profitPrice, int barsSinceEntry)
            {
                EntryPrice = entryPrice;
                StopPrice = stopPrice;
                ProfitPrice = profitPrice;
                BarsSinceEntry = barsSinceEntry;
            }
        }

        private class SymbolData
        {
            private decimal _price = 0m;
            private ExponentialMovingAverage _ema;

            public bool IsAboveEMA;
            public bool IsRsiCondition;
            public Symbol Symbol;
            public RelativeStrengthIndex Rsi;

            public SymbolData(Symbol symbol)
            {
                Symbol = symbol;
                Rsi = new RelativeStrengthIndex(3);
                _ema = new ExponentialMovingAverage(150);
            }

            public void Update(DateTime time, decimal value)
            {
                if (_ema.Update(time, value) && Rsi.Update(time, value))
                {
                    _price = value;

                    var ema = _ema.Current.Value;
                    var rsi = Rsi.Current.Value;

                    IsAboveEMA = value > ema;
                    IsRsiCondition = rsi < 30;
                }
            }
        }

        private class SymbolDataADX
        {
            public AverageTrueRange Atr;
            public AverageDirectionalIndex Adx;

            public bool AdxCondition;
            public bool AtrCondition;
            public SymbolData SymbolData;

            public SymbolDataADX(SymbolData data)
            {
                SymbolData = data;
                Atr = new AverageTrueRange(string.Empty, 10);
                Adx = new AverageDirectionalIndex(string.Empty, 7);
            }

            public void Update(TradeBar bar)
            {
                if (Adx.Update(bar) && Atr.Update(bar))
                {
                    var atr = Atr.Current.Value;
                    var adx = Adx.Current.Value;

                    AtrCondition = (atr / bar.Close) > 0.04m;
                    AdxCondition = adx > 45;
                }
            }
        }
    }
}

@mchandschuh (mentioning because you've been assigned) - this might save you some time if you're looking to debug this issue. It's not the best code, but I was looking to do a one-one replica of the failing algorithm to avoid this exact situation from happening.

Edit: I think what's happening is that somewhere along the stack, the Symbol becomes corrupted and contains the SID Base32 in the Symbol.Value. Here's an example error:

20190708 17:32:55.989 Trace:: AlgorithmManager.HandleSplitSymbols(): 3/20/2018 12:00:00 AM - Security split occurred: Split Factor: Split: FISV: 0.5 | 148.4300 Reference Price: 148.4300
20190708 17:33:05.890 Trace:: FactorFile.HasScalingFactors(): Factor file not found: ACIA
20190708 17:33:13.756 Trace:: FactorFile.HasScalingFactors(): Factor file not found: QCP
20190708 17:33:14.610 Trace:: FactorFile.HasScalingFactors(): Factor file not found: MEDP
20190708 17:33:19.121 Trace:: Isolator.ExecuteWithTimeLimit(): Used: 1692, Sample: 1299, App: 2461, CurrentTimeStepElapsed: 00:03.006
20190708 17:33:24.231 Trace:: AlgorithmManager.HandleSplitSymbols(): 3/28/2018 12:00:00 AM - Security split warning: Split Warning: BRO: 0.5 | 51.0400
20190708 17:33:24.231 Trace:: AlgorithmManager.HandleSplitSymbols(): 3/28/2018 12:00:00 AM - Security split warning: Split Warning: ERY U7FBH5GTZQZP: 5 | 9.9100
20190708 17:33:29.261 ERROR:: <>c__DisplayClass9_2.<Run>b__9():  System.Exception: AlgorithmManager.ProcessSplitSymbols(): 3/29/2018 12:00:00 AM - No subscriptions found for ERY U7FBH5GTZQZP, IsTradable: False, Active: True
   at QuantConnect.Lean.Engine.AlgorithmManager.ProcessSplitSymbols(IAlgorithm algorithm, List`1 splitWarnings) in C:\Users\Gerardo\Projects\QuantConnect\Lean\Engine\AlgorithmManager.cs:line 1215
   at QuantConnect.Lean.Engine.AlgorithmManager.Run(AlgorithmNodePacket job, IAlgorithm algorithm, ISynchronizer synchronizer, ITransactionHandler transactions, IResultHandler results, IRealTimeHandler realtime, ILeanManager leanManager, IAlphaHandler alphas, CancellationToken token) in C:\Users\Gerardo\Projects\QuantConnect\Lean\Engine\AlgorithmManager.cs:line 384
   at QuantConnect.Lean.Engine.Engine.<>c__DisplayClass9_2.<Run>b__9() in C:\Users\Gerardo\Projects\QuantConnect\Lean\Engine\Engine.cs:line 335
20190708 17:33:29.443 ERROR:: Engine.HandleAlgorithmError(): Breaking out of parent try catch: System.Exception: AlgorithmManager.ProcessSplitSymbols(): 3/29/2018 12:00:00 AM - No subscriptions found for ERY U7FBH5GTZQZP, IsTradable: False, Active: True
   at QuantConnect.Lean.Engine.AlgorithmManager.ProcessSplitSymbols(IAlgorithm algorithm, List`1 splitWarnings) in C:\Users\Gerardo\Projects\QuantConnect\Lean\Engine\AlgorithmManager.cs:line 1215
   at QuantConnect.Lean.Engine.AlgorithmManager.Run(AlgorithmNodePacket job, IAlgorithm algorithm, ISynchronizer synchronizer, ITransactionHandler transactions, IResultHandler results, IRealTimeHandler realtime, ILeanManager leanManager, IAlphaHandler alphas, CancellationToken token) in C:\Users\Gerardo\Projects\QuantConnect\Lean\Engine\AlgorithmManager.cs:line 384
   at QuantConnect.Lean.Engine.Engine.<>c__DisplayClass9_2.<Run>b__9() in C:\Users\Gerardo\Projects\QuantConnect\Lean\Engine\Engine.cs:line 335

As you can see, on a Security Split Warning, only the Symbol Value is returned. But, for symbol ERY, it contains the SecurityIdentifier Base32 information when it should only be printing ERY, not ERY U7FBH5GTZQZP. Finding the root cause of this corruption will most likely lead to the resolving of the issue

Martin-Molinero commented 4 years ago

This issue is not reproducible anymore with the reported algorithm which finishes succesfully, so closing this issue for now.