Recieved bar with all values NaN or completely empty

[ghost]

Hi Ran. I run the following code and very often (specially when I have many symbols) I receive bars with all fields NaN. Also, I sometimes receive empty bars as well.

Example output:

---------- This is for instrument: CVS ----------
   high  open  close  volume  low symbol symbol_group asset_class  signal
3   NaN   NaN    NaN     NaN  NaN    NaN          NaN         NaN     NaN
{'T': 3, 'ABBV': 4, 'GOOG': 2, 'MMM': 3, 'CVS': 4, 'MP': 0, 'AXP': 3, 'ABT': 3,
'ALL': 3, 'AMZN': 4, 'BMY': 3, 'AAPL': 4, 'GOOGL': 2, 'AMGN': 4, 'BLK': 1, 'BAC'
: 4, 'COF': 3, 'ACN': 3, 'BIIB': 3, 'BA': 3, 'AIG': 4, 'AGN': 3}
Total number of on_bar called: 68
Total number of non empty bars recieved: 66

Code:

from qtpylib.algo import Algo
import pandas as pd
import threading

class TestStrategy(Algo):
    def on_start(self):
        self.bar_count = dict.fromkeys(self.symbols, 0)
        self.counter = 0
        self.lock = threading.Lock()
        print("Started...")

    def on_bar(self, instrument):
        if not instrument:
            return

        self.lock.acquire()
        bars = instrument.get_bars(28)
        print('-'*10, 'This is for instrument:', instrument.get_symbol(), '-'*10)
        if len(bars.index):
            self.bar_count[instrument] = self.bar_count[instrument] + 1
            print(bars.tail(1))
        self.counter = self.counter + 1
        allbars = 0
        for symbl in self.symbols:
             allbars = self.bar_count[symbl] + allbars
        print(self.bar_count)
        print('Total number of on_bar called:', self.counter)    
        print('Total number of non empty bars recieved:', allbars)
        print('-'*50)
        self.lock.release()

if __name__ == "__main__":
    sp500 = pd.DataFrame.from_csv('../datas/sp500.csv', index_col = None)

    strategy = TestStrategy(
        bar_window = 28,
        #instruments = sp500.iloc[:,0].tolist()[0:98],
        instruments = ['MMM', 'T', 'ABBV', 'ABT', 'ACN', 'AGN', 'ALL', 'GOOG', 'GOOGL', 'MP', 'AMZN', 'AXP', 'AIG', 'AMGN', 'AAPL',
                        'BAC', 'BIIB', 'BLK', 'BA', 'BMY', 'CVS', 'COF'],
        resolution  = "1T" # 1Min bar resolution (Pandas "resample" resolutions)
    )
    strategy.run()

[ranaroussi]

I ran your strategy as-is with the most recent version for 90 minutes and got everything working perfectly...

---------- This is for instrument: ACN ----------
                             open    high     low   close  volume symbol
2016-12-25 12:51:00+00:00  117.59  117.59  117.59  117.59       1    ACN

{'MMM': 61, 'ABBV': 67, 'BMY': 69, 'ALL': 69, 'AAPL': 69, 'AXP': 69, 'AMZN': 69, 
'AIG': 68, 'MP': 68, 'BA': 69, 'BAC': 69, 'BLK': 69, 'COFCVS': 68, 'GOOG': 69,
 'AMGN': 69, 'ACN': 69, 'AGN': 67, 'T': 67, 'ABT': 69, 'GOOGL': 62, 'BIIB': 70}

Total number of on_bar called: 1426
Total number of non empty bars recieved: 1426
Total number of instruments with no bars: 0
--------------------------------------------------

Try upgrading to version 1.5.55 and run this again. Hopefully it'll work this time around :)

[ghost]

Thanks Ran. Any thought on the last comments I posted for issue #48:

With the new update after 3 minutes of receiving data GOOG has no data in place yet:

---------- This is for instrument: ALL ----------
Time passed since start: (3.0, 54.57248899999999)
                             low   open   high  volume  close symbol  \
datetime                                                               
2016-12-25 20:40:00+00:00  74.51  74.51  74.51       0  74.51    ALL   

                          symbol_group asset_class  signal  
datetime                                                    
2016-12-25 20:40:00+00:00          ALL         STK     NaN  
{'AMGN': 3, 'ALL': 4, 'BA': 3, 'AXP': 3, 'AAPL': 3, 'GOOGL': 0, 'AGN': 0, 'ABT': 3, 'GOOG': 0, 'AIG': 3, 'MP': 0, 'CVS': 3, 'AMZN': 3, 'ABBV': 3, 'BAC': 3, 'BIIB': 3, 'COF': 3, 'BMY': 0, 'ACN': 3, 'MMM': 0, 'BLK': 3, 'T': 2}
Total number of on_bar called: 48
Total number of non empty bars recieved: 48
--------------------------------------------------
---------- This is for instrument: T ----------
Time passed since start: (4.0, 5.599064999999996)
                             low   open   high  volume  close symbol  \
datetime                                                               
2016-12-25 20:40:00+00:00  42.65  42.65  42.65       0  42.65      T   

                          symbol_group asset_class  signal  
datetime                                                    
2016-12-25 20:40:00+00:00            T         STK     NaN  
{'AMGN': 3, 'ALL': 4, 'BA': 3, 'AXP': 3, 'AAPL': 3, 'GOOGL': 0, 'AGN': 0, 'ABT': 3, 'GOOG': 0, 'AIG': 3, 'MP': 0, 'CVS': 3, 'AMZN': 3, 'ABBV': 3, 'BAC': 3, 'BIIB': 3, 'COF': 3, 'BMY': 0, 'ACN': 3, 'MMM': 0, 'BLK': 3, 'T': 3}
Total number of on_bar called: 49
Total number of non empty bars recieved: 49

[ranaroussi]

Test using TWS instead of the Gateway and load the relevant symbol on the chart. If you don't see price changes there - you won't see it on QTPyLib...

This happens a lot when testing using the edemo user - you'll often not get market data for all symbols.

[ranaroussi]

Also - the source of this issue as the same one as issue #48, so I'm closing #48 and marking it as duplicate for the time being. Once this is resolved, #48 will be resolved as well.

[ghost]

Thanks, Ran.

I see the data on the TWS. I also used the sample example for IBPY and I am able to get the tick data for the symbols that are not received by qtpylib.

[ranaroussi]

I'm not sure what the problem is :-/ I'm testing this on my 3 machines and I keep seeing all of TWS's data in QTPyLib in perfect sync.

What platform are you using for development? I want to rule out any platform-related issues.

[ghost]

I have the same issue on both my iOS and Windows. On both I have Anaconda 3 installed.

To make sure that I understand what the framework is designed to do please let me know if this statement is correct: If I am subscribed to a ticker data, I should expect to receive a bar for each minute for that ticker.

If this is a correct statement, then I'm confused by the fact that in the following example after 481 minutes of running I only get 110 minute data for AAPL; shouldn't be 480 or a number close to that?:

--------------------------------------------------
Time passed since start: (481.0, 23.600978999998915)
{'AGN': 0, 'BLK': 111, 'AXP': 109, 'AAPL': 110, 'BMY': 0, 'BIIB': 108, 'MP': 0, 'ALL': 121, 'CVS': 107, 'AMGN': 114, 'ABBV': 112, 'COF': 111, 'AIG': 113, 'MMM': 0, 'AMZN': 111, 'ABT': 110, 'BAC': 109, 'GOOG': 0, 'ACN': 108, 'GOOGL': 0, 'T': 117, 'BA': 110}
Total number of on_bar called: 1781
Total number of non empty bars recieved: 1781
--------------------------------------------------

[ranaroussi]

Have you specified a bar_window? Unless you didn't, the defaults is 100 (QTPyLib might store ~10% more bars than requested due to the threaded architecture, so ~110 bars is reasonable)

[ghost]

Here bar_window is 28. The numbers you see in front of each symbol is the number of times a data event has been received for that symbol, not the bar length. So it should be in theory ~480 for each symbol.

[ranaroussi]

I'm thinking this threaded thing is the core problem here... I'm going to add some parameters to limit the number of concurrent threads based on the hardware availability on the machine running the algo.

[ranaroussi]

So... I've set the max number of threads to 1 (single-threaded). You can change this parameter using the max_threads run-time parameter when running your algo:

$ python strategy.py --max_threads [NUMBER]

You should try playing with this parameter and set its value based on your strategy’s needs and your machine's capabilities. I would start with the number of cores you have x 10 and work from there.

Please upgrade to the latest version using

$ pip install qtpylib --upgrade --no-cache-dir

...and let me know if this helps. I really home it would 😄

[ghost]

Thanks Ran for the quick fix. I updated and run as you advised. I get an error :

I guess the tools file was not uodate correctly when you pushed it to server?

(qtpylib) C:\Users\Reza\trading\libs\qtpylib>python blotter.py --max_thread 1 Traceback (most recent call last): File "blotter.py", line 21, in from qtpylib.blotter import Blotter File "C:\Users\Reza\trading\libs\qtpylib\qtpylib\blotter.py", line 64, in ule> threads = tools.read_single_argv("--max_threads") AttributeError: module 'qtpylib.tools' has no attribute 'read_single_argv'

[ranaroussi]

Oops 🤣 ... I forgot to push one of the commits. Please re-upgrade and try again

[ghost]

Thanks, Ran. I still see the same issue. It has been 9 minutes that strategy and blotter are running and almost all symbols are behind. Some are not receiving the data since it is after market time and I'm using the edemo account instead of my paper trading account. But the ones that are live in edemo account are behind as well which tells me the issue is still there.

--------------------------------------------------
---------- This is for instrument: APC ----------
Time passed since start: (9.0, 58.33100000000002)
{'DVN': 4, 'NBL': 5, 'APC': 6, 'MRO': 5, 'EOG': 4, 'OXY': 6, 'PXD': 5, 'HES': 5,
 'AAPL': 4, 'XOM': 5, 'CVX': 4, 'AZN': 5, 'APA': 5, 'IBKR': 0, 'COP': 5, 'HAL':
5, 'SLB': 5}
Total number of on_bar called: 78
Total number of non empty bars recieved: 78
--------------------------------------------------

[ghost]

Hi Ran. I pulled your newest update with (added getPool + no threads if set to <=1 ) now it is working. I'll do more testing with 100+ symbols and let you know how it goes.

Not sure if you celebrate the Gregorian new year, but if you do, happy new year ;)

[ranaroussi]

Although it's a month's delay - I wish you a good year too (well, 10.5 months 😀 at least)