Data object empty with dynamic universe selection.

[swisstackle]

Expected Behavior

I expect to be able to use SetHoldings and that SetHoldings can retrieve the current price of the equity asset.

Actual Behavior

The SetHoldings function needs Security.Price to calculate the order quantity from the percentage given. The Security.Price attribute is populated by the Update function which uses the data object.

In my algorithm data.ContainsKey(symbol) returns false for many symbols, which means that the price won't be available to SetHoldings which causes the following error:

Backtest Handled Error: NUE R735QTJ8XC9X: The security does not have an accurate price as it has not yet received a bar of data. Before placing a trade (or using SetHoldings) warm up your algorithm with SetWarmup, or use slice.Contains(symbol) to confirm the Slice object has price before using the data. Data does not necessarily all arrive at the same time so your algorithm should confirm the data is ready before using it. In live trading this can mean you do not have an active subscription to the asset class you're trying to trade. If using custom data make sure you've set the 'Value' property.

This does not just happen with NUE, this happens with other symbols too.

A quick check in the research environment shows me that there is actually data available in the dataset during the timeframe:

            var qb = new QuantBook();
            var symbol = qb.AddEquity("NUE", Resolution.Daily).Symbol;
            var history = qb.History(symbol, new DateTime(2007, 2, 1).AddDays(-185), new DateTime(2007, 2, 1), Resolution.Daily);
            history.FirstOrDefault()

Potential Solution

I do not have a potential solution right now. I am a C# developer but I would have to dig deeper into the codebase. If this turns out to be a bug and not a mistake on my side, I'm willing to help to speed up the process.

Reproducing the Problem

This is the code for my algorithm. The algorithm utilizes a dynamic universe, which means I can't just use SetWarmup in the initialize method.

The algorithm is initialized with a dynamic universe selection as such:

            AddUniverseSelection(
                new ScheduledUniverseSelectionModel(
                DateRules.MonthStart(),
                TimeRules.Midnight,
                GetNewUniverse
            ));

And the error happens in the OnData method in this for loop:

                foreach (var symbol in topStocksNotInvested)
                {
                    if (inverseVolatilities.Keys.Contains(symbol))
                    {
                        decimal targetWeight = inverseVolatilities[symbol] / totalInverseVolatility;
                        if (this.Securities[symbol].IsTradable)
                        {
                            // if (data.ContainsKey(symbol))
                            // {
                            //    actuallyInvested++;
                                SetHoldings(symbol, targetWeight);
                            // }
                        }
                    }
                }

namespace QuantConnect.Algorithm.CSharp
{
    public class SystematicMomentum : QCAlgorithm
    {
        private bool _securitiesChanged = false;

        private List<Symbol> currentSymbols = new List<Symbol>();
        private int nrRemovedSecs = 0;

        private int windowSizeForRegression = 125;

        private int windowSizeForVolatility = 20;

        private int nrOfStocksToHold = 30;

        private int minimumMomentumScore = 40;

        int maxSymbols = 503;

        private SimpleMovingAverage sma;
        private decimal previousSma;

        private string CurrentDate = String.Empty;

        List<Symbol> _universe = new List<Symbol>();
        Dictionary<string, List<(string, double, decimal)>> UniverseData = new Dictionary<string, List<(string, double, decimal)>>();
        public override void Initialize()
        {
            var moreThan40Json = ObjectStore.ReadJson<Dictionary<string, List<(string, double, decimal)>>>("MoreThan40.json");

            //We ordered the symbols by descending standard deviation by accident, so we have to re order it by ascending standard deviation.
            foreach (var old in moreThan40Json)
            {
                var newValue = old.Value.OrderBy(e => e.Item3).ThenByDescending(e => e.Item2).ToList();
                UniverseData.Add(old.Key, newValue);
            }
            Settings.MinimumOrderMarginPortfolioPercentage = 0;

            SetStartDate(2007, 1, 1);
            SetEndDate(2010, 12, 30);
            SetCash(100000);

            UniverseSettings.Asynchronous = true;

            // The LEAN engine will automatically remove the symbols that arent returned by rebalance.
            AddUniverseSelection(
                new ScheduledUniverseSelectionModel(
                DateRules.MonthStart(),
                TimeRules.Midnight,
                GetNewUniverse
            ));
            AddEquity("SPY", Resolution.Daily);
            AddEquity("TLT", Resolution.Daily);

            sma = SMA("SPY", 200, Resolution.Daily);
        }

        private List<Symbol> GetNewUniverse(DateTime dateTime)
        {
            CurrentDate = FindClosestDate(UniverseData.Keys.ToList(), dateTime.ToString("yyyy-MM-dd"));
            var tickers = UniverseData[CurrentDate];
            var symbols = tickers.Select(x =>
                QuantConnect.Symbol.Create(x.Item1, SecurityType.Equity, Market.USA)).ToList();
            symbols = symbols.Take(maxSymbols).ToList();
            currentSymbols = symbols;
            return symbols;
        }

        public override void OnSecuritiesChanged(SecurityChanges changes)
        {
            nrRemovedSecs = 0;
            foreach (var security in changes.RemovedSecurities)
            {
                if (security.Invested)
                {
                    Liquidate(security.Symbol);
                    nrRemovedSecs++;
                }
            }
            _securitiesChanged = true;
        }

        // Since our members data has only random dates as keys, we need to find the date closest to the month start
        private string FindClosestDate(List<string> dates, string y)
        {
            DateTime inputDate = DateTime.Parse(y);
            List<DateTime> dateTimes = dates.Select(date =>
            {
                if (!date.IsNullOrEmpty())
                {
                    return DateTime.Parse(date);
                }
                else
                {
                    Debug($"Found invalid date");
                    return DateTime.Parse(UniverseData.Keys.ToList().FirstOrDefault());
                }
            }
        ).ToList();
            DateTime closestDate = dateTimes.Aggregate((current, next) => Math.Abs((current - inputDate).Ticks) < Math.Abs((next - inputDate).Ticks) ? current : next);
            return closestDate.ToString("yyyy-MM-dd");
        }

        // We will need the OnChange function in order to liquidate removed universe selections.

        public override void OnData(Slice data)
        {
            if (this.IsWarmingUp)
            {
                return;
            }
            // Now that are ineligible equities are removed already, we can just weigh them based on std and trade them
            // Calculate inverse volatility for each symbol
            if (!_securitiesChanged)
            {
                return;
            }
            // the code that you want to measure comes here
            // I need to construct a list of symbols of the top 30 stocks plus the already invested stocks
            var alreadyInvested = currentSymbols.Where(e => Portfolio[e].Invested).ToList();

            var topNr = nrOfStocksToHold - nrRemovedSecs - alreadyInvested.Count;

            // Set target weight for each symbol
            Dictionary<Symbol, decimal> inverseVolatilities = new Dictionary<Symbol, decimal>();
            foreach (var symbol in currentSymbols)
            {
                decimal volatility = UniverseData[CurrentDate].Where(e => e.Item1 == symbol.Value).FirstOrDefault().Item3;

                if (volatility != 0)
                {
                    inverseVolatilities[symbol] = 1 / volatility;
                }
            }

            decimal totalInverseVolatility = inverseVolatilities.Values.Sum();

            List<Symbol> topStocksNotInvested = currentSymbols
            .Where(e =>
            {
                if (Portfolio.Keys.Contains(e))
                {
                    return !Portfolio[e].Invested;
                }
                else
                {
                    return true;
                }
            }
            )
            .OrderBy(symbol =>
            {
                if (inverseVolatilities.Keys.Contains(symbol))
                {
                    return inverseVolatilities[symbol];
                }
                else
                {
                    return 0;
                }
            })
            .Take(topNr)
            .ToList();

            if (previousSma <= sma.Current.Value)
            {
                // sell tlt
                // Place your trading logic here
                if (Portfolio["TLT"].Invested)
                {
                    SetHoldings("TLT", 0);
                    if (data.Time.Date.ToString("MM/dd/yyyy") == "09/01/2009")
                    {
                        var cDate = CurrentDate;
                        var ddata = UniverseData[CurrentDate];
                        var cSymboles = currentSymbols;
                    }
                }
                var actuallyInvested = 0;
                foreach (var symbol in alreadyInvested)
                {
                    if (inverseVolatilities.Keys.Contains(symbol))
                    {
                        decimal targetWeight = inverseVolatilities[symbol] / totalInverseVolatility;
                        if (this.Securities[symbol].IsTradable)
                        {
                            //if (data.ContainsKey(symbol))
                            //{
                            //    actuallyInvested++;

                                SetHoldings(symbol, targetWeight);
                            //}
                        }

                    }
                }

                foreach (var symbol in topStocksNotInvested)
                {
                    if (inverseVolatilities.Keys.Contains(symbol))
                    {
                        decimal targetWeight = inverseVolatilities[symbol] / totalInverseVolatility;
                        if (this.Securities[symbol].IsTradable)
                        {
                            //if (data.ContainsKey(symbol))
                            //{
                            //    actuallyInvested++;
                                SetHoldings(symbol, targetWeight);
                            //}
                        }
                    }
                }
                // Debug($"{data.Time.Date.ToString()}: {actuallyInvested} / {topStocksNotInvested.Count + alreadyInvested.Count} were tradeable.");

            }
            else
            {
                // Invest in TLT
                if (!Portfolio["TLT"].Invested)
                {
                    Liquidate();
                    Debug($"{data.Time.Date}: SMA crossed below {previousSma} vs {sma.Current.Value} and liquidated");
                }

                currentSymbols = new List<Symbol>();
                SetHoldings("TLT", 1);

            }

            previousSma = sma.Current.Value;
            _securitiesChanged = false;
        }
    }
}

System Information

OS is Windows 10 Professional, but this error happens in the cloud. I do not have a local setup of LEAN.

Checklist

• [X ] I have completely filled out this template
• [ X] I have confirmed that this issue exists on the current master branch
• [X ] I have confirmed that this is not a duplicate issue by searching issues
• [X ] I have provided detailed steps to reproduce the issue

[AlexCatarino]

Hi @swisstackle, When the algorithm adds a security, it creates a data subscription that will create the time slices and update the security cache so that the price differs from zero. Your algorithm might select very illiquid stocks so that ContainsKey returns false. You can use GetLastKnowPrices to feed/warm up the security. See Set Security Initializer.

I suggest joining our Discord channel or contacting support for clarification.