Closed heerad closed 4 years ago
Martin-Molinero
commented
4 years ago Hi @heerad ! Could you provide some algorithm/code snippet to reproduce the issue it will help us fix it. If you have the exception message and call stack would also be helpful, thanks!
heerad
commented
4 years ago Here's the stack trace:
2014-05-01 00:00:00 Runtime Error: IndexOutOfRangeException : Index was outside the bounds of the array.
at QuantConnect.Algorithm.Framework.Portfolio.ReturnsSymbolDataExtensions+<>c__DisplayClass0_2.<FormReturnsMatrix>b__12 (System.Double[] s) [0x00000] in <4b69f79636454a7ebbe0a328ba38483d>:0
at System.Linq.Enumerable+SelectEnumerableIterator`2[TSource,TResult].ToArray () [0x0001d] in <92922d9bda2f4e1cba9242d052be6d43>:0
at System.Linq.Enumerable.ToArray[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x0001f] in <92922d9bda2f4e1cba9242d052be6d43>:0
at QuantConnect.Algorithm.Framework.Portfolio.ReturnsSymbolDataExtensions+<>c__DisplayClass0_0.<FormReturnsMatrix>b__11 (System.Int32 d) [0x00024] in <4b69f79636454a7ebbe0a328ba38483d>:0
at System.Linq.Enumerable+SelectIPartitionIterator`2[TSource,TResult].MoveNext () [0x00048] in <92922d9bda2f4e1cba9242d052be6d43>:0
at System.Linq.Enumerable+WhereEnumerableIterator`1[TSource].ToArray () [0x00033] in <92922d9bda2f4e1cba9242d052be6d43>:0
at System.Linq.Enumerable.ToArray[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x0001f] in <92922d9bda2f4e1cba9242d052be6d43>:0
at QuantConnect.Algorithm.Framework.Portfolio.ReturnsSymbolDataExtensions.FormReturnsMatrix (System.Collections.Generic.Dictionary`2[TKey,TValue] symbolData, System.Collections.Generic.IEnumerable`1[T] symbols) [0x001d2] in <4b69f79636454a7ebbe0a328ba38483d>:0
at QuantConnect.Algorithm.Framework.Portfolio.MeanVarianceOptimizationPortfolioConstructionModel.DetermineTargetPercent (System.Collections.Generic.List`1[T] activeInsights) [0x00032] in <4b69f79636454a7ebbe0a328ba38483d>:0
at QuantConnect.Algorithm.Framework.Portfolio.PortfolioConstructionModel.CreateTargets (QuantConnect.Algorithm.QCAlgorithm algorithm, QuantConnect.Algorithm.Framework.Alphas.Insight[] insights) [0x00124] in <aef0ff5ba99f4c94a61c31770edc9d3d>:0
at QuantConnect.Algorithm.QCAlgorithm.ProcessInsights (QuantConnect.Algorithm.Framework.Alphas.Insight[] insights) [0x00007] in <aef0ff5ba99f4c94a61c31770edc9d3d>:0
at QuantConnect.Algorithm.QCAlgorithm.OnFrameworkData (QuantConnect.Data.Slice slice) [0x001a4] in <aef0ff5ba99f4c94a61c31770edc9d3d>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <b0e1ad7573a24fd5a9f2af9595e677e7>:0 IndexOutOfRangeException : Index was outside the bounds of the array.
at QuantConnect.Algorithm.Framework.Portfolio.ReturnsSymbolDataExtensions+<>c__DisplayClass0_2.<FormReturnsMatrix>b__12 (System.Double[] s) [0x00000] in <4b69f79636454a7ebbe0a328ba38483d>:0
at System.Linq.Enumerable+SelectEnumerableIterator`2[TSource,TResult].ToArray () [0x0001d] in <92922d9bda2f4e1cba9242d052be6d43>:0
at System.Linq.Enumerable.ToArray[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x0001f] in <92922d9bda2f4e1cba9242d052be6d43>:0
at QuantConnect.Algorithm.Framework.Portfolio.ReturnsSymbolDataExtensions+<>c__DisplayClass0_0.<FormReturnsMatrix>b__11 (System.Int32 d) [0x00024] in <4b69f79636454a7ebbe0a328ba38483d>:0
at System.Linq.Enumerable+SelectIPartitionIterator`2[TSource,TResult].MoveNext () [0x00048] in <92922d9bda2f4e1cba9242d052be6d43>:0
at System.Linq.Enumerable+WhereEnumerableIterator`1[TSource].ToArray () [0x00033] in <92922d9bda2f4e1cba9242d052be6d43>:0
at System.Linq.Enumerable.ToArray[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x0001f] in <92922d9bda2f4e1cba9242d052be6d43>:0
at QuantConnect.Algorithm.Framework.Portfolio.ReturnsSymbolDataExtensions.FormReturnsMatrix (System.Collections.Generic.Dictionary`2[TKey,TValue] symbolData, System.Collections.Generic.IEnumerable`1[T] symbols) [0x001d2] in <4b69f79636454a7ebbe0a328ba38483d>:0
at QuantConnect.Algorithm.Framework.Portfolio.MeanVarianceOptimizationPortfolioConstructionModel.DetermineTargetPercent (System.Collections.Generic.List`1[T] activeInsights) [0x00032] in <4b69f79636454a7ebbe0a328ba38483d>:0
at QuantConnect.Algorithm.Framework.Portfolio.PortfolioConstructionModel.CreateTargets (QuantConnect.Algorithm.QCAlgorithm algorithm, QuantConnect.Algorithm.Framework.Alphas.Insight[] insights) [0x00124] in <aef0ff5ba99f4c94a61c31770edc9d3d>:0
at QuantConnect.Algorithm.QCAlgorithm.ProcessInsights (QuantConnect.Algorithm.Framework.Alphas.Insight[] insights) [0x00007] in <aef0ff5ba99f4c94a61c31770edc9d3d>:0
at QuantConnect.Algorithm.QCAlgorithm.OnFrameworkData (QuantConnect.Data.Slice slice) [0x001a4] in <aef0ff5ba99f4c94a61c31770edc9d3d>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <b0e1ad7573a24fd5a9f2af9595e677e7>:0 I will circle back if I can generate a very simple algorithm snippet that generates this error!
heerad
commented
4 years ago I've pasted the code you can use to reproduce the error below.
Please note that I'm using an alpha model that produces a random insight direction and magnitude per security, and that I self-defined a universe selection model. This selection model was pasted from the QC500UniverseSelectionModelsource code as of two months ago, and modified to make 500 into a parameter (N). All else is the same.
class FrameworkAlgorithm(QCAlgorithm):
def Initialize(self):
self.SetStartDate(2013, 1, 1)
self.SetCash(1000000)
self.UniverseSettings.Resolution = Resolution.Daily
self.SetUniverseSelection(QCNUniverseSelectionModel(N=100))
self.SetAlpha(RandomAlphaModel())
self.SetPortfolioConstruction(MeanVarianceOptimizationPortfolioConstructionModel())
self.SetRiskManagement(MaximumDrawdownPercentPerSecurity(0.02))
self.SetExecution(ImmediateExecutionModel())
self.SetBrokerageModel(AlphaStreamsBrokerageModel())
class FundamentalUniverseSelectionModel:
'''Provides a base class for defining equity coarse/fine fundamental selection models'''
def __init__(self,
filterFineData,
universeSettings = None,
securityInitializer = None):
'''Initializes a new instance of the FundamentalUniverseSelectionModel class
Args:
filterFineData: True to also filter using fine fundamental data, false to only filter on coarse data
universeSettings: The settings used when adding symbols to the algorithm, specify null to use algorthm.UniverseSettings
securityInitializer: Optional security initializer invoked when creating new securities, specify null to use algorithm.SecurityInitializer'''
self.filterFineData = filterFineData
self.universeSettings = universeSettings
self.securityInitializer = securityInitializer
def CreateUniverses(self, algorithm):
'''Creates a new fundamental universe using this class's selection functions
Args:
algorithm: The algorithm instance to create universes for
Returns:
The universe defined by this model'''
universe = self.CreateCoarseFundamentalUniverse(algorithm)
if self.filterFineData:
universe = FineFundamentalFilteredUniverse(universe, lambda fine: self.SelectFine(algorithm, fine))
return [universe]
def CreateCoarseFundamentalUniverse(self, algorithm):
'''Creates the coarse fundamental universe object.
This is provided to allow more flexibility when creating coarse universe, such as using algorithm.Universe.DollarVolume.Top(5)
Args:
algorithm: The algorithm instance
Returns:
The coarse fundamental universe'''
universeSettings = algorithm.UniverseSettings if self.universeSettings is None else self.universeSettings
securityInitializer = algorithm.SecurityInitializer if self.securityInitializer is None else self.securityInitializer
return CoarseFundamentalUniverse(universeSettings, securityInitializer, lambda coarse: self.FilteredSelectCoarse(algorithm, coarse))
def FilteredSelectCoarse(self, algorithm, coarse):
'''Defines the coarse fundamental selection function.
If we're using fine fundamental selection than exclude symbols without fine data
Args:
algorithm: The algorithm instance
coarse: The coarse fundamental data used to perform filtering
Returns:
An enumerable of symbols passing the filter'''
if self.filterFineData:
coarse = filter(lambda c: c.HasFundamentalData, coarse)
return self.SelectCoarse(algorithm, coarse)
def SelectCoarse(self, algorithm, coarse):
'''Defines the coarse fundamental selection function.
Args:
algorithm: The algorithm instance
coarse: The coarse fundamental data used to perform filtering
Returns:
An enumerable of symbols passing the filter'''
raise NotImplementedError("SelectCoarse must be implemented")
def SelectFine(self, algorithm, fine):
'''Defines the fine fundamental selection function.
Args:
algorithm: The algorithm instance
fine: The fine fundamental data used to perform filtering
Returns:
An enumerable of symbols passing the filter'''
return [f.Symbol for f in fine]
class QCNUniverseSelectionModel(FundamentalUniverseSelectionModel):
'''Defines the QC-N universe as a universe selection model for framework algorithm
For details: https://github.com/QuantConnect/Lean/pull/1663'''
def __init__(self, N = 500, filterFineData = True, universeSettings = None, securityInitializer = None):
'''Initializes a new default instance of the QCNUniverseSelectionModel'''
super().__init__(filterFineData, universeSettings, securityInitializer)
self.numberOfSymbolsCoarse = 1000
self.numberOfSymbolsFine = N
self.dollarVolumeBySymbol = {}
self.lastMonth = -1
def SelectCoarse(self, algorithm, coarse):
'''Performs coarse selection for the QC-N constituents.
The stocks must have fundamental data
The stock must have positive previous-day close price
The stock must have positive volume on the previous trading day'''
if algorithm.Time.month == self.lastMonth:
return Universe.Unchanged
sortedByDollarVolume = sorted([x for x in coarse if x.HasFundamentalData and x.Volume > 0 and x.Price > 0],
key = lambda x: x.DollarVolume, reverse=True)[:self.numberOfSymbolsCoarse]
self.dollarVolumeBySymbol = {x.Symbol:x.DollarVolume for x in sortedByDollarVolume}
# If no security has met the QC500 criteria, the universe is unchanged.
# A new selection will be attempted on the next trading day as self.lastMonth is not updated
if len(self.dollarVolumeBySymbol) == 0:
return Universe.Unchanged
# return the symbol objects our sorted collection
return list(self.dollarVolumeBySymbol.keys())
def SelectFine(self, algorithm, fine):
'''Performs fine selection for the QC-N constituents
The company's headquarter must in the U.S.
The stock must be traded on either the NYSE or NASDAQ
At least half a year since its initial public offering
The stock's market cap must be greater than 500 million'''
sortedBySector = sorted([x for x in fine if x.CompanyReference.CountryId == "USA"
and x.CompanyReference.PrimaryExchangeID in ["NYS","NAS"]
and (algorithm.Time - x.SecurityReference.IPODate).days > 180
and x.MarketCap > 5e8],
key = lambda x: x.CompanyReference.IndustryTemplateCode)
count = len(sortedBySector)
# If no security has met the QC500 criteria, the universe is unchanged.
# A new selection will be attempted on the next trading day as self.lastMonth is not updated
if count == 0:
return Universe.Unchanged
# Update self.lastMonth after all QC500 criteria checks passed
self.lastMonth = algorithm.Time.month
percent = self.numberOfSymbolsFine / count
sortedByDollarVolume = []
from itertools import groupby
from math import ceil
# select stocks with top dollar volume in every single sector
for code, g in groupby(sortedBySector, lambda x: x.CompanyReference.IndustryTemplateCode):
y = sorted(g, key = lambda x: self.dollarVolumeBySymbol[x.Symbol], reverse = True)
c = ceil(len(y) * percent)
sortedByDollarVolume.extend(y[:c])
sortedByDollarVolume = sorted(sortedByDollarVolume, key = lambda x: self.dollarVolumeBySymbol[x.Symbol], reverse=True)
return [x.Symbol for x in sortedByDollarVolume[:self.numberOfSymbolsFine]]
class RandomAlphaModel(AlphaModel):
def __init__(self):
self.resolution = Resolution.Daily
self.insightPeriod = Extensions.ToTimeSpan(self.resolution)
self.symbolData = {}
resolutionString = Extensions.GetEnumString(self.resolution, Resolution)
self.Name = '{}({})'.format(self.__class__.__name__, resolutionString)
def Update(self, algorithm, data):
insights = []
for key, sd in self.symbolData.items():
if sd.Security.Price == 0:
continue
import numpy as np
x = np.random.normal(scale=1e-4)
if x > 0:
direction = InsightDirection.Up
else:
direction = InsightDirection.Down
insight = Insight.Price(sd.Security.Symbol, self.insightPeriod, direction, abs(x))
insights.append(insight)
return insights
def OnSecuritiesChanged(self, algorithm, changes):
for added in changes.AddedSecurities:
self.symbolData[added.Symbol] = SymbolData(algorithm, added, self.resolution)
for removed in changes.RemovedSecurities:
data = self.symbolData.pop(removed.Symbol, None)
if data is not None:
algorithm.SubscriptionManager.RemoveConsolidator(removed.Symbol, data.Consolidator)
class SymbolData:
def __init__(self, algorithm, security, resolution):
self.Security = security
self.Consolidator = algorithm.ResolveConsolidator(security.Symbol, resolution)
Expected Behavior
MeanVarianceOptimizationPortfolioConstructionModelshould work without breakingActual Behavior
Portfolio.ReturnsSymbolDataExtensions.FormReturnsMatrix, which is called byDetermineTargetPercentencounters anIndexOutOfRangeExceptionon aToArraycall.This used to not be an issue, as I ran the exact code without error two months ago.
Checklist
masterbranch