Consolidators are not working when adding them after program start

michaelsrew commented 7 years ago

Hi, adding consolidators and securities after programm start make a lot of trouble. When testing take a look in the OnData method and the Time.Minute value which should be set.

The following code:
QuantAlgo.txt

/*

QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. */

using System.Collections.Generic; using QuantConnect.Data.Market; using System; using QuantConnect.Indicators; using QuantConnect.Data.Consolidators; using QuantConnect.Data;

namespace QuantConnect.Algorithm.CSharp { ///

/// In this algorithm we demonstrate how to define a universe /// as a combination of use the coarse fundamental data and fine fundamental data ///

public class QuantAlgo : QCAlgorithm { private Dictionary<string, SymbolData> Data = new Dictionary<string, SymbolData>(); bool added = false; public override void Initialize() { AddSecurity(SecurityType.Equity, "SPY", Resolution.Second, true, true); }

    /// <summary>
    /// This is our event handler for our 10 minute trade bar defined above in Initialize(). So each time the consolidator
    /// produces a new 10 minute bar, this function will be called automatically. The 'sender' parameter will be the
    /// instance of the IDataConsolidator that invoked the event, but you'll almost never need that!
    /// </summary>
    private void TenMinuteBarHandler(object sender, TradeBar consolidated)
    {
        Console.Write(consolidated.Symbol.Value);
    }

    private void AddConsolidator(string security)
    {
        Console.WriteLine("consolidator added " + security);
        var tenMinuteConsolidator = new TradeBarConsolidator(TimeSpan.FromMinutes(10));

        // attach our event handler. the event handler is a function that will be called each time we produce
        // a new consolidated piece of data.
        tenMinuteConsolidator.DataConsolidated += TenMinuteBarHandler;

        // this call adds our 10 minute consolidator to the manager to receive updates from the engine
        SubscriptionManager.AddConsolidator(security, tenMinuteConsolidator);
    }

    /// <summary>
    /// OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.
    /// </summary>
    /// <param name="data">TradeBars IDictionary object with your stock data</param>
    public void OnData(TradeBars data)
    {
        if (Time.Minute == 48 && !added) 
        {
            AddSecurity(SecurityType.Equity, "ORIG", Resolution.Minute, true, true);
            AddConsolidator("ORIG");
            AddSecurity(SecurityType.Equity, "REN", Resolution.Minute, true, true);
            AddConsolidator("REN");
            AddSecurity(SecurityType.Equity, "SDRL", Resolution.Minute, true, true);
            AddConsolidator("SDRL");
            AddSecurity(SecurityType.Equity, "CYH", Resolution.Minute, true, true);
            AddConsolidator("CYH");
            added = true;
        }
    }

} } the code above calls the 10 minutes consolidator many times. output:

20161128 20:46:03 Trace:: BrokerageSetupHandler.Setup(): Adding unrequested security: FB 2T 20161128 20:46:03 Trace:: LiveTradingRealTimeHandler.SetupEvents(Equity): Market hours set: Symbol: FB Monday: PreMarket: 04:00:00-09:30:00 | Market: 09:30:00-16:00:00 | PostMarket: 16:00:00-20:00:00 20161128 20:46:03 Trace:: LiveTradingRealTimeHandler.SetupEvents(Equity): Market hours set: Symbol: SPY Monday: PreMarket: 04:00:00-09:30:00 | Market: 09:30:00-16:00:00 | PostMarket: 16:00:00-20:00:00 20161128 20:46:03 Trace:: LiveTradingResultHandler.SendStatusUpdate(): Running 20161128 20:46:04 Trace:: LiveTradingDataFeed.AddSubscription(): Added SPY,SPY,Second,TradeBar 20161128 20:46:04 Trace:: AlgorithmManager.Run(): Begin DataStream - Start: 28.11.2016 00:00:00 Stop: 31.12.2050 00:00:00 20161128 20:46:04 Trace:: InteractiveBrokersBrokerage.GetContractDetails(): clientOnContractDetails event: SPY 20161128 20:46:04 Trace:: LiveTradingDataFeed.AddSubscription(): Added FB,FB,Second,TradeBar 20161128 20:46:04 Trace:: LiveTradingResultHandler.SendStatusUpdate(): Running 20161128 20:46:04 Trace:: Debug: Launching analysis for QuantAlgo with LEAN Engine v2.2.0.2 20161128 20:46:04 Trace:: InteractiveBrokersBrokerage.GetContractDetails(): clientOnContractDetails event: FB 20161128 20:47:04 Trace:: 2016-11-28 20:47:04Z Isolator.ExecuteWithTimeLimit(): Used: 10 20161128 20:48:00 Trace:: LiveTradingDataFeed.AddSubscription(): Added ORIG,ORIG,Minute,TradeBar 20161128 20:48:00 Trace:: InteractiveBrokersBrokerage.GetContractDetails(): clientOnContractDetails event: ORIG 20161128 20:48:00 Trace:: LiveTradingDataFeed.AddSubscription(): Added REN,REN,Minute,TradeBar 20161128 20:48:00 Trace:: InteractiveBrokersBrokerage.GetContractDetails(): clientOnContractDetails event: REN 20161128 20:48:00 Trace:: LiveTradingDataFeed.AddSubscription(): Added SDRL,SDRL,Minute,TradeBar 20161128 20:48:00 Trace:: InteractiveBrokersBrokerage.GetContractDetails(): clientOnContractDetails event: SDRL 20161128 20:48:00 Trace:: LiveTradingDataFeed.AddSubscription(): Added CYH,CYH,Minute,TradeBar 20161128 20:48:00 Trace:: InteractiveBrokersBrokerage.GetContractDetails(): clientOnContractDetails event: CYH 20161128 20:48:00 Trace:: Debug: consolidator added ORIG 20161128 20:48:00 Trace:: Debug: consolidator added REN 20161128 20:48:00 Trace:: Debug: consolidator added SDRL 20161128 20:48:00 Trace:: Debug: consolidator added CYH 20161128 20:48:04 Trace:: 2016-11-28 20:48:04Z Isolator.ExecuteWithTimeLimit(): Used: 12 20161128 20:49:04 Trace:: 2016-11-28 20:49:04Z Isolator.ExecuteWithTimeLimit(): Used: 9 20161128 20:50:00 Trace:: Debug: REN 20161128 20:50:00 Trace:: Debug: SDRL 20161128 20:50:00 Trace:: Debug: CYH 20161128 20:50:00 Trace:: Debug: ORIG 20161128 20:50:01 Trace:: Debug: REN 20161128 20:50:01 Trace:: Debug: SDRL 20161128 20:50:01 Trace:: Debug: CYH 20161128 20:50:01 Trace:: Debug: ORIG 20161128 20:50:02 Trace:: Debug: REN 20161128 20:50:02 Trace:: Debug: SDRL 20161128 20:50:02 Trace:: Debug: CYH 20161128 20:50:02 Trace:: Debug: ORIG 20161128 20:50:03 Trace:: Debug: REN 20161128 20:50:03 Trace:: Debug: SDRL 20161128 20:50:03 Trace:: Debug: CYH 20161128 20:50:03 Trace:: Debug: ORIG 20161128 20:50:04 Trace:: Debug: REN 20161128 20:50:04 Trace:: Debug: SDRL 20161128 20:50:04 Trace:: Debug: CYH 20161128 20:50:04 Trace:: Debug: ORIG 20161128 20:50:04 Trace:: 2016-11-28 20:50:04Z Isolator.ExecuteWithTimeLimit(): Used: 13 20161128 20:50:05 Trace:: Debug: REN 20161128 20:50:05 Trace:: Debug: SDRL 20161128 20:50:05 Trace:: Debug: CYH 20161128 20:50:05 Trace:: Debug: ORIG 20161128 20:50:06 Trace:: Debug: REN 20161128 20:50:06 Trace:: Debug: SDRL 20161128 20:50:06 Trace:: Debug: CYH 20161128 20:50:06 Trace:: Debug: ORIG 20161128 20:50:07 Trace:: Debug: REN 20161128 20:50:07 Trace:: Debug: SDRL 20161128 20:50:07 Trace:: Debug: CYH 20161128 20:50:07 Trace:: Debug: ORIG 20161128 20:50:08 Trace:: Debug: REN 20161128 20:50:08 Trace:: Debug: SDRL 20161128 20:50:08 Trace:: Debug: CYH 20161128 20:50:08 Trace:: Debug: ORIG 20161128 20:50:09 Trace:: Debug: REN 20161128 20:50:09 Trace:: Debug: SDRL 20161128 20:50:09 Trace:: Debug: CYH 20161128 20:50:09 Trace:: Debug: ORIG 20161128 20:50:10 Trace:: Debug: REN 20161128 20:50:10 Trace:: Debug: SDRL 20161128 20:50:10 Trace:: Debug: CYH 20161128 20:50:10 Trace:: Debug: ORIG 20161128 20:50:11 Trace:: Debug: REN 20161128 20:50:11 Trace:: Debug: SDRL 20161128 20:50:11 Trace:: Debug: CYH 20161128 20:50:11 Trace:: Debug: ORIG 20161128 20:50:12 Trace:: Debug: REN 20161128 20:50:12 Trace:: Debug: SDRL 20161128 20:50:12 Trace:: Debug: CYH 20161128 20:50:12 Trace:: Debug: ORIG 20161128 20:50:13 Trace:: Debug: REN 20161128 20:50:13 Trace:: Debug: SDRL 20161128 20:50:13 Trace:: Debug: CYH 20161128 20:50:13 Trace:: Debug: ORIG 20161128 20:50:14 Trace:: Debug: REN 20161128 20:50:14 Trace:: Debug: SDRL 20161128 20:50:14 Trace:: Debug: CYH 20161128 20:50:14 Trace:: Debug: ORIG 20161128 20:50:15 Trace:: Debug: REN 20161128 20:50:15 Trace:: Debug: SDRL 20161128 20:50:15 Trace:: Debug: CYH 20161128 20:50:15 Trace:: Debug: ORIG 20161128 20:50:16 Trace:: Debug: REN 20161128 20:50:16 Trace:: Debug: SDRL 20161128 20:50:16 Trace:: Debug: CYH 20161128 20:50:16 Trace:: Debug: ORIG 20161128 20:50:17 Trace:: Debug: REN 20161128 20:50:17 Trace:: Debug: SDRL 20161128 20:50:17 Trace:: Debug: CYH 20161128 20:50:17 Trace:: Debug: ORIG 20161128 20:50:18 Trace:: Debug: REN 20161128 20:50:18 Trace:: Debug: SDRL 20161128 20:50:18 Trace:: Debug: CYH 20161128 20:50:18 Trace:: Debug: ORIG 20161128 20:50:19 Trace:: Debug: REN 20161128 20:50:19 Trace:: Debug: SDRL 20161128 20:50:19 Trace:: Debug: CYH 20161128 20:50:19 Trace:: Debug: ORIG 20161128 20:50:20 Trace:: Debug: REN 20161128 20:50:20 Trace:: Debug: SDRL 20161128 20:50:20 Trace:: Debug: CYH 20161128 20:50:20 Trace:: Debug: ORIG 20161128 20:50:21 Trace:: Debug: REN 20161128 20:50:21 Trace:: Debug: SDRL 20161128 20:50:21 Trace:: Debug: CYH 20161128 20:50:21 Trace:: Debug: ORIG 20161128 20:50:22 Trace:: Debug: REN 20161128 20:50:22 Trace:: Debug: SDRL 20161128 20:50:22 Trace:: Debug: CYH 20161128 20:50:22 Trace:: Debug: ORIG 20161128 20:50:23 Trace:: Debug: REN 20161128 20:50:23 Trace:: Debug: SDRL 20161128 20:50:23 Trace:: Debug: CYH 20161128 20:50:23 Trace:: Debug: ORIG 20161128 20:50:24 Trace:: Debug: REN 20161128 20:50:24 Trace:: Debug: SDRL 20161128 20:50:24 Trace:: Debug: CYH 20161128 20:50:24 Trace:: Debug: ORIG 20161128 20:50:25 Trace:: Debug: REN 20161128 20:50:25 Trace:: Debug: SDRL 20161128 20:50:25 Trace:: Debug: CYH 20161128 20:50:25 Trace:: Debug: ORIG 20161128 20:50:26 Trace:: Debug: REN 20161128 20:50:26 Trace:: Debug: SDRL 20161128 20:50:26 Trace:: Debug: CYH 20161128 20:50:26 Trace:: Debug: ORIG 20161128 20:50:27 Trace:: Debug: REN 20161128 20:50:27 Trace:: Debug: SDRL 20161128 20:50:27 Trace:: Debug: CYH 20161128 20:50:27 Trace:: Debug: ORIG 20161128 20:50:28 Trace:: Debug: REN 20161128 20:50:28 Trace:: Debug: SDRL 20161128 20:50:28 Trace:: Debug: CYH 20161128 20:50:28 Trace:: Debug: ORIG 20161128 20:50:29 Trace:: Debug: REN 20161128 20:50:29 Trace:: Debug: SDRL 20161128 20:50:29 Trace:: Debug: CYH 20161128 20:50:29 Trace:: Debug: ORIG 20161128 20:50:30 Trace:: Debug: REN 20161128 20:50:30 Trace:: Debug: SDRL 20161128 20:50:30 Trace:: Debug: CYH 20161128 20:50:30 Trace:: Debug: ORIG 20161128 20:50:31 Trace:: Debug: REN 20161128 20:50:31 Trace:: Debug: SDRL 20161128 20:50:31 Trace:: Debug: CYH 20161128 20:50:31 Trace:: Debug: ORIG 20161128 20:50:32 Trace:: Debug: REN 20161128 20:50:32 Trace:: Debug: SDRL 20161128 20:50:32 Trace:: Debug: CYH 20161128 20:50:32 Trace:: Debug: ORIG 20161128 20:50:33 Trace:: Debug: REN 20161128 20:50:33 Trace:: Debug: SDRL 20161128 20:50:33 Trace:: Debug: CYH 20161128 20:50:33 Trace:: Debug: ORIG 20161128 20:50:34 Trace:: Debug: REN 20161128 20:50:34 Trace:: Debug: SDRL 20161128 20:50:34 Trace:: Debug: CYH 20161128 20:50:34 Trace:: Debug: ORIG 20161128 20:50:35 Trace:: Debug: REN 20161128 20:50:35 Trace:: Debug: SDRL 20161128 20:50:35 Trace:: Debug: CYH 20161128 20:50:35 Trace:: Debug: ORIG 20161128 20:50:36 Trace:: Debug: REN 20161128 20:50:36 Trace:: Debug: SDRL 20161128 20:50:36 Trace:: Debug: CYH 20161128 20:50:36 Trace:: Debug: ORIG 20161128 20:50:37 Trace:: Debug: REN 20161128 20:50:37 Trace:: Debug: SDRL 20161128 20:50:37 Trace:: Debug: CYH 20161128 20:50:37 Trace:: Debug: ORIG 20161128 20:50:38 Trace:: Debug: REN 20161128 20:50:38 Trace:: Debug: SDRL 20161128 20:50:38 Trace:: Debug: CYH 20161128 20:50:38 Trace:: Debug: ORIG 20161128 20:50:39 Trace:: Debug: REN 20161128 20:50:39 Trace:: Debug: SDRL 20161128 20:50:39 Trace:: Debug: CYH 20161128 20:50:39 Trace:: Debug: ORIG 20161128 20:50:40 Trace:: Debug: REN 20161128 20:50:40 Trace:: Debug: SDRL 20161128 20:50:40 Trace:: Debug: CYH 20161128 20:50:40 Trace:: Debug: ORIG 20161128 20:50:41 Trace:: Debug: REN 20161128 20:50:41 Trace:: Debug: SDRL 20161128 20:50:41 Trace:: Debug: CYH 20161128 20:50:41 Trace:: Debug: ORIG 20161128 20:50:42 Trace:: Debug: REN 20161128 20:50:42 Trace:: Debug: SDRL 20161128 20:50:42 Trace:: Debug: CYH 20161128 20:50:42 Trace:: Debug: ORIG 20161128 20:50:43 Trace:: Debug: REN 20161128 20:50:43 Trace:: Debug: SDRL 20161128 20:50:43 Trace:: Debug: CYH 20161128 20:50:43 Trace:: Debug: ORIG 20161128 20:50:44 Trace:: Debug: REN 20161128 20:50:44 Trace:: Debug: SDRL 20161128 20:50:44 Trace:: Debug: CYH 20161128 20:50:44 Trace:: Debug: ORIG 20161128 20:50:45 Trace:: Debug: REN 20161128 20:50:45 Trace:: Debug: SDRL 20161128 20:50:45 Trace:: Debug: CYH 20161128 20:50:45 Trace:: Debug: ORIG 20161128 20:50:46 Trace:: Debug: REN 20161128 20:50:46 Trace:: Debug: SDRL 20161128 20:50:46 Trace:: Debug: CYH 20161128 20:50:46 Trace:: Debug: ORIG 20161128 20:50:47 Trace:: Debug: REN 20161128 20:50:47 Trace:: Debug: SDRL 20161128 20:50:47 Trace:: Debug: CYH 20161128 20:50:47 Trace:: Debug: ORIG 20161128 20:50:48 Trace:: Debug: REN 20161128 20:50:48 Trace:: Debug: SDRL 20161128 20:50:48 Trace:: Debug: CYH 20161128 20:50:48 Trace:: Debug: ORIG 20161128 20:50:49 Trace:: Debug: REN 20161128 20:50:49 Trace:: Debug: SDRL 20161128 20:50:49 Trace:: Debug: CYH 20161128 20:50:49 Trace:: Debug: ORIG 20161128 20:50:50 Trace:: Debug: REN 20161128 20:50:50 Trace:: Debug: SDRL 20161128 20:50:50 Trace:: Debug: CYH 20161128 20:50:50 Trace:: Debug: ORIG 20161128 20:50:51 Trace:: Debug: REN 20161128 20:50:51 Trace:: Debug: SDRL 20161128 20:50:51 Trace:: Debug: CYH 20161128 20:50:51 Trace:: Debug: ORIG 20161128 20:50:52 Trace:: Debug: REN 20161128 20:50:52 Trace:: Debug: SDRL 20161128 20:50:52 Trace:: Debug: CYH 20161128 20:50:52 Trace:: Debug: ORIG 20161128 20:50:53 Trace:: Debug: REN 20161128 20:50:53 Trace:: Debug: SDRL 20161128 20:50:53 Trace:: Debug: CYH 20161128 20:50:53 Trace:: Debug: ORIG 20161128 20:50:54 Trace:: Debug: REN 20161128 20:50:54 Trace:: Debug: SDRL 20161128 20:50:54 Trace:: Debug: CYH 20161128 20:50:54 Trace:: Debug: ORIG 20161128 20:50:55 Trace:: Debug: REN 20161128 20:50:55 Trace:: Debug: SDRL 20161128 20:50:55 Trace:: Debug: CYH 20161128 20:50:55 Trace:: Debug: ORIG 20161128 20:50:56 Trace:: Debug: REN 20161128 20:50:56 Trace:: Debug: SDRL 20161128 20:50:56 Trace:: Debug: CYH 20161128 20:50:56 Trace:: Debug: ORIG 20161128 20:50:57 Trace:: Debug: REN 20161128 20:50:57 Trace:: Debug: SDRL 20161128 20:50:57 Trace:: Debug: CYH 20161128 20:50:57 Trace:: Debug: ORIG 20161128 20:50:58 Trace:: Debug: REN 20161128 20:50:58 Trace:: Debug: SDRL 20161128 20:50:58 Trace:: Debug: CYH 20161128 20:50:58 Trace:: Debug: ORIG 20161128 20:50:59 Trace:: Debug: REN 20161128 20:50:59 Trace:: Debug: SDRL 20161128 20:50:59 Trace:: Debug: CYH 20161128 20:50:59 Trace:: Debug: ORIG 20161128 20:51:04 Trace:: 2016-11-28 20:51:04Z Isolator.ExecuteWithTimeLimit(): Used: 7 20161128 20:52:04 Trace:: 2016-11-28 20:52:04Z Isolator.ExecuteWithTimeLimit(): Used: 9

maybe someone can take a look thanks

jaredbroad commented 7 years ago

Looks like its working as expected. The times in the log are your local system time. If you want to see the algorithm time use Time.ToString("u");

Can you point out specifically which logs you think are broken? Please use the file-attach instead of pasting in long logs.

michaelsrew commented 7 years ago

ok, when you take a look on the output it calls the TenMinuteBarHandler for every stock every second: 20161128 20:50:59 Trace:: Debug: REN 20161128 20:50:59 Trace:: Debug: SDRL 20161128 20:50:59 Trace:: Debug: CYH 20161128 20:50:59 Trace:: Debug: ORIG

take a look in the TenMinuteBarHandler method there is a single console.writeline.....but it should be called every 10 minute not every 10 minute 1 minute long and 1 times every second!

you know what i mean. the part with the console.writeline is getting called every 10 minute 1 minute long. thats not good :)

StefanoRaggi commented 7 years ago

@michaelsrew I can confirm the consolidators are emitting once a second after firing at the correct time the first time. The issue is not with the consolidators being called from OnData, but being generated from fill-forward bars. It seems there are also other potential problems using consolidators with fill-forward data (see discussion in PR https://github.com/QuantConnect/Lean/pull/636).

I would suggest for now to use AddSecurity(SecurityType.Equity, "ORIG", Resolution.Minute, false, true); instead of AddSecurity(SecurityType.Equity, "ORIG", Resolution.Minute, true, true);

michaelsrew commented 7 years ago

ok thanks i will check that. i moved the code-block: AddSecurity(SecurityType.Equity, "ORIG", Resolution.Minute, true, true); AddConsolidator("ORIG"); AddSecurity(SecurityType.Equity, "REN", Resolution.Minute, true, true); AddConsolidator("REN"); AddSecurity(SecurityType.Equity, "SDRL", Resolution.Minute, true, true); AddConsolidator("SDRL"); AddSecurity(SecurityType.Equity, "CYH", Resolution.Minute, true, true); AddConsolidator("CYH");

in the "public override void Initialize()" function and it seems to work with that without any problems. thanks i will check your solution.

StefanoRaggi commented 7 years ago

For completeness I tested PR #636 and it fixes the bug (with no changes to the algorithm)

michaelsrew commented 7 years ago

muchas gracias

michaelsrew commented 7 years ago

Hi, i think this issue can be closed. Thanks for fixing.

StefanoRaggi commented 7 years ago

Fixed in PR #643

QuantConnect / Lean

Consolidators are not working when adding them after program start #642