Custom Universe Selection Schedule

[Martin-Molinero]

Expected Behavior

• Users can specify a custom universe selection schedule independent of the underlying data frequency

  Actual Behavior

• Universe selection relies on the data frequency to perform the selection. Lean currently supports ITimeTriggeredUniverse which creates data points based on a schedule but this can't be used with other universe selection data sources. Users currently have to add logic in their selection method to skip selection steps

  Potential Solution

• Implement

  Reproducing the Problem

  N/A

  System Information

  N/A

  Checklist

• [x] I have completely filled out this template
• [x] I have confirmed that this issue exists on the current master branch
• [x] I have confirmed that this is not a duplicate issue by searching issues
• [x] I have provided detailed steps to reproduce the issue

[Martin-Molinero]

Some notes about discussed potential work flow and API changes:

• PR1 AddUniverse(Resolution/TimeSpan) -> AddUniverse(Func<DateTime, DateTime>)
• PR2 AddUniverse(IDateRule) -> AddUniverse(Func<DateTime, DateTime>)
• PR3 AddUniverse(Func<DateTime, DateTime?>) a func that could rely on the algorithms state, determines when selection should happen, if it doesn't know could return null -> tricky because backtest runs in a separate worker thread, ahead of time
  • he needs an underlying data selection feed at what resolution? Load all of the data and perform the filtering in the algo manager loop, so that the algorithm state is right
  • what if its run without a worker thread and directly driven by the algo manager.

[cnaccio]

Any updates/movement on this? This could really help for situations where universe selection is done based on stock behavior when the market opens or during pre-market. i.e. Small cap stocks that run up during pre-market, or stocks that break out of an opening market range during market open. I understand the challenge though, as you'd potentially need to monitor real-time price streams for 8,000 equities.

[NeoSephiroth]

Yes, I really need this one, too

[kbsaravana]

+1 for this ask. I'm also looking for this

[TateLinzel]

I am also looking to use this ability. Can an update be provided? or a target date for completion?

[jaredbroad]

Hi all / @TateLinzel, thank you for the interest in this feature.

This is probably technically possible, but would consume an enormous amount of data - further slowing down LEAN (as @cnaccio said - we'd need to monitor 8k assets prices every minute of the day). We're addressing many challenges now to get the next 10x improvement in speed but will soon be able to expand on this future feature further. Note that its not a simple challenge and we appreciate your patience and support as we continue grinding =)

[dibbydoob]

+1 for me , this would be awesome :) Thanks for your hard work folks.

[jaredbroad]

• Simplifying to only look at longer periods of time (so we can use the same daily data)
• Solve the live trading released at 5am issue?