No data from Blotter to Algo

[ErnstTmp]

Hi Ran, thanks again for creating qtpylib, it is great.

I have a strange behaviour: if I create a fresh Anaconda environment, install qtpylib there (with pip install qtpylib --upgrade --no-cache-dir), create the database and run the blotter, the quote data for AAPL is stored in the database. However, if I start a simple strategy - only printing when on_bar() is called, I only get the printout once, so on_bar() is called only once. If I print the bars, they are the first bars when the blotter was started, not the current bars. Then, nothing happens.

If I start the same strategy on an older install (a few weeks) of qtpylib on another machine, the strategy runs fine.

If I remove the line

preload     = "1D", # preload 1 day history when starting

from the instrument definition, it works.

strategy = AAPLStrat(
        instruments = [ ("AAPL", "STK", "SMART", "USD", "", 0.0, "") ], # ib tuples
        resolution  = "1min", # Pandas resolution (use "K" for tick bars)
        tick_window = 20, # no. of ticks to keep
        bar_window  = 5, # no. of bars to keep
        preload     = "1D", # preload 1 day history when starting

The only difference is that TWS runs on the second, older machine where it works. Can this be the reason?

Thank, Ernst

[ranaroussi]

I have a few quick questions to help me better understand the situation:

• Which Python environment/version are you using?
• Are both machines have TWS/IBGW running on the same port?
• Are you using a live, paper or edemo/demouser account?
• Are you testing this during trading hours?

Thanks! Ran

[ErnstTmp]

Hi Ran,

thanks for your help.

1) On both machines, I run Anaconda with python 3.5 that should be up.to-date. conda -V shows conda 3.18.3 pyhon -V shows Python 3.5.2 :: Anaconda 4.0.0 (64-bit)

2) I run TWS only one one computer, the other one connects remotely to this computer. Quotes data is entered into the database on both

3) I am using a paper trading account, but could also test live trading 4) I am trading EUR-USD, and I only run this when quotes are available (I checked this in TWS), FX quotes are available Mo-Fr all the time, except 23:00-23:15 our time each day. I tested on both computers right now - the data is added into the database on both computers.

Does this answer your questions? Thanks & kind regards Ernst

[ErnstTmp]

Hi Ran, I just found out that it works on both machines if I remove the line

preload = "1D", # preload 1 day history when starting

Can this be that this takes so long on one machine that it freezes the program? Thanks and kind regards Ernst

[ranaroussi]

Is the new machine extremely slow / connecting to a remote DB?

Can you please upgrade to the latest version and let me know if the problem was resolved?

[ranaroussi]

Any updates on this or can I close this issue for now?

[ErnstTmp]

Hi Ran,

I am sorry for waiting so long - I will analyse it today afternoon and update the issue.

Thanks a lot for creating qtpylib and for your patience :-) !!

Will you attend NIPS this year - I am going to Barcelona tomorrow...

Kind regards

Ernst

[ranaroussi]

Enjoy NIPS :)

[ErnstTmp]

Thanks! Will be long days :-) 8000 are there ... Will be interesting to see what they present at the Timeseries Workshop :-)

I tried to run with the newest package from today. If I omit preload='1D', the strategy runs fine.

If I include preload='1D' into the strategy, I get an error:

Traceback (most recent call last):
  File "strategy.py", line 216, in <module>
    strategy.run()
  File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/qtpylib/algo.py", line 292, in run
    book_handler  = self._book_handler
  File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/qtpylib/blotter.py", line 969, in stream
    bar_handler(df)
  File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/qtpylib/algo.py", line 640, in _bar_handler
    self.on_bar(self.get_instrument(bar))
  File "strategy.py", line 122, in on_bar
    unixdate = last_bar.index[0]
  File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/pandas/indexes/range.py", line 483, in __getitem__
    size=len(self)))
IndexError: index 0 is out of bounds for axis 0 with size 0

Can it be a problem that there is not enough data in the database?

I will wait 1 hour to fill the DB, and the rerun with preload='1H'

What else can I do to help you characterize the problem?

Thanks, Ernst

[ranaroussi]

Can you post the part in your strategy.py that's right before unixdate = last_bar.index[0]?

[ErnstTmp]

It is a very simple test strategy and it looks like this (If you want I can post the full file)

def on_bar(self, instrument):
    # get instrument history
    bars = instrument.get_bars(lookback=168)

    last_bar = bars.tail(1)
    print("Got Bar ",type(last_bar))
    print("Got Bar ",last_bar)

    unixdate = last_bar.index[0]
    print("Unixdate ", unixdate)

    last_bar = last_bar.iloc[0]

[ranaroussi]

I can't think of any reason why these error is dependant on using preload... I just ran both of the following codes, with and without the preload part, and it all ran just fine on both my machines.

Try using this code and let me know if this was resolved.

def on_bar(self, instrument):
    # get instrument history
    bars = instrument.get_bars(lookback=168)

    last_bar = bars[-1:]
    print("Got Bar ",type(last_bar))
    print("Got Bar ",last_bar)

    dt = last_bar.index[-1]
    unixdate = dt.timestamp()

    print("Datetime ", dt)
    print("Unixdate ", unixdate)

or (using instrument.bar to get last bar as dict)

def on_bar(self, instrument):
    # get instrument history
    bars = instrument.get_bars(lookback=168)

    last_bar = instrument.bar
    print("Got Bar ",type(last_bar))
    print("Got Bar ",last_bar)

    dt = last_bar['datetime']
    unixdate = dt.timestamp()

    print("Datetime ", dt)
    print("Unixdate ", unixdate)

[ranaroussi]

Just to be on the safe side, please make sure you're running the latest version first.

$ pip install qtpylib --upgrade --no-cache-dir

:)

[ErnstTmp]

Thanks!!! I am really sorry, I tried the pip command from your last comment, and the tried the two code versions you provided.

The first version produces the same error as before, and the second outputs:

*** ERASING THE DF 
Traceback (most recent call last):
  File "strategy.py", line 221, in <module>
    strategy.run()
  File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/qtpylib/algo.py", line 292, in run
    book_handler  = self._book_handler
  File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/qtpylib/blotter.py", line 969, in stream
    bar_handler(df)
  File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/qtpylib/algo.py", line 640, in _bar_handler
    self.on_bar(self.get_instrument(bar))
  File "strategy.py", line 121, in on_bar
    last_bar = instrument.bar
  File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/qtpylib/instrument.py", line 577, in bar
    return self.get_bar()
  File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/qtpylib/instrument.py", line 75, in get_bar
    return self.get_bars(lookback=1, as_dict=True)
  File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/qtpylib/instrument.py",    line 68, in get_bars
    bars = bars[0]
    IndexError: list index out of range

[ErnstTmp]

I changed the 1D to 1H after enough data was in the database, but I still get the same errors if I use one of the code snipets from you. What do you need to debug the problem - I can add debugging code? It looks like the bar is empty. Ernst

[ranaroussi]

This looks like the on_bar() method is being called when no actual bars exist... I'm also not sure what's writing the *** ERASING THE DF message.

Can you run this and post the results here?

def on_bar(self, instrument):
    # get instrument history
    bars = instrument.get_bars(lookback=168)

    print("--- BARS ------------")
    print(bars)
    print("----------------------")

    last_bar = bars[-1:]
    print("Got Bar ",type(last_bar))
    print("Got Bar ",last_bar)

    dt = last_bar.index[-1]
    unixdate = dt.timestamp()

    print("Datetime ", dt)
    print("Unixdate ", unixdate)

[ErnstTmp]

That is some code that is called on intialize. If you want, I can remove that entirely and see if it still happens. But the strage thing is this only happens when preload is active. Strange.

[ErnstTmp]

The full py and out files are attached. I have removed all the unimportant stuff from the py file.

  --- BARS --------
  Empty DataFrame
  Columns: [opt_theta, opt_delta, open, opt_volume, low, opt_dividend, opt_iv, high, opt_underlying,   opt_vega, close, opt_oi, volume, opt_gamma, opt_price, symbol, symbol_group, asset_class, signal]
  Index: []
  ----------------------
  Got Bar  <class 'pandas.core.frame.DataFrame'>
  Got Bar  Empty DataFrame
  Columns: [opt_theta, opt_delta, open, opt_volume, low, opt_dividend, opt_iv, high, opt_underlying, opt_vega, close, opt_oi, volume, opt_gamma, opt_price, symbol, symbol_group, asset_class, signal]
  Index: []
  Traceback (most recent call last):
   File "strategy_ran.py", line 67, in <module>
    strategy.run()
    File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/qtpylib/algo.py", line 292, in run
    book_handler  = self._book_handler
  File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/qtpylib/blotter.py", line 969, in stream
    bar_handler(df)
  File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/qtpylib/algo.py", line 640, in _bar_handler
    self.on_bar(self.get_instrument(bar))
  File "strategy_ran.py", line 51, in on_bar
    dt = last_bar.index[-1]
  File "/home/ernst/anaconda2/envs/anaconda3/lib/python3.5/site-packages/pandas/indexes/range.py", line 483, in __getitem__
    size=len(self)))
IndexError: index -1 is out of bounds for axis 0 with size 0

ran.zip

[ranaroussi]

I ran your strategy "as-is" with no problems whatsoever, so I'm really not sure what the problem is :(

I wasn't being able to reproduce this error no matter what I did on my local machine (Python 3.4), my trading server and my forward-testing server (both running Python 3.5).

If this is a new environment like you mentioned, it may be something in is configuration. Can you try installing it on a new Python 3.4/3.5 environment?

[ErnstTmp]

This is really strange. This is what I did: 1) Create a new linux user and sudo su 2) Install a new anaconda 4.2 for the fresh user 3) conda update conda 4) create a new conda environment and activate it 5) pip install --ignore-installed --upgrade pip setuptools 6) pip install qtpylib --upgrade --no-cache-dir 7) start blotter.py and then strategy_ran.py

I got the same error as before (I reproduced the whole process twice). Can it be a problem of the TWS not running on the same machine as the qtpylib?

Empty DataFrame
Columns: [close, opt_volume, opt_oi, high, opt_dividend, opt_vega, opt_delta, low, volume, open, opt_theta, opt_underlying, opt_price, opt_gamma, opt_iv, symbol, symbol_group, asset_class, signal]
Index: []
----------------------
Got Bar  <class 'pandas.core.frame.DataFrame'>
Got Bar  Empty DataFrame
Columns: [close, opt_volume, opt_oi, high, opt_dividend, opt_vega, opt_delta, low, volume, open, opt_theta, opt_underlying, opt_price, opt_gamma, opt_iv, symbol, symbol_group, asset_class, signal]
Index: []
Traceback (most recent call last):
  File "strategy_ran.py", line 66, in <module>
    strategy.run()
  File "/home/ernst5/anaconda3/lib/python3.5/site-packages/qtpylib/algo.py", line 292, in run
    book_handler  = self._book_handler
  File "/home/ernst5/anaconda3/lib/python3.5/site-packages/qtpylib/blotter.py", line 969, in stream
    bar_handler(df)
  File "/home/ernst5/anaconda3/lib/python3.5/site-packages/qtpylib/algo.py", line 640, in _bar_handler
    self.on_bar(self.get_instrument(bar))
  File "strategy_ran.py", line 50, in on_bar
    dt = last_bar.index[-1]
  File "/home/ernst5/anaconda3/lib/python3.5/site-packages/pandas/indexes/range.py", line 483, in __getitem__
    size=len(self)))

[ranaroussi]

You're not getting data because you don't have TWS/GW running and you're not in backtesting mode either - but you shouldn't get these errors neither. I'm not sure why is on_bar() being called when no bars actually exist.

Please change the on_bar() logic to this and run it twice (with and without TWS running) and let me know what was the output:

def on_bar(self, instrument):
    print("--- INSTRUMENT BARS ---")
    print( instrument.bars.tail() )
    print("-----------------------")
    print("")
    print("------ ALL BARS -------")
    print( self.bars.tail() )
    print("-----------------------")

[ErnstTmp]

Thank you again for your help!! The TWS is running on 2nd machine which is available over the network, and bars are fed into the database by the blotter. The current bars are also sent into the strategy, if I turn preload off.

Following your suggestion, I ran the strategy with the on_bar you supplied:

1) if TWS is not running on the remote machine, I do not get any output from on_bar

2) If TWS is running remotely, I get output (enclosed from 2 runs.). Only 5 bars are printed after start from self.bars, while instrument bars is empty. If I stop and start the strategy, I get other 5 bars (it is printing always the 5 last bars). Then, the strtegy ran for almost an hour, but nothing was printed until I stopped it.

-- INSTRUMENT BARS ---
Empty DataFrame
Columns: [open, opt_gamma, close, volume, opt_iv, opt_theta, opt_oi, opt_underlying, opt_vega, opt_price, opt_delta, high, low, opt_dividend, opt_volume, symbol, symbol_group, asset_class, signal]
Index: []
-----------------------
------ ALL BARS -------
                              open  opt_gamma    close  volume  opt_iv  \
datetime
2016-12-02 08:25:00+00:00  1.06695        NaN  1.06695       0     NaN
2016-12-02 08:26:00+00:00  1.06683        NaN  1.06683       0     NaN
2016-12-02 08:27:00+00:00  1.06658        NaN  1.06658       0     NaN
2016-12-02 08:28:00+00:00  1.06652        NaN  1.06652       0     NaN
2016-12-02 08:29:00+00:00  1.06658        NaN  1.06658       0     NaN
................... 

see files. run.zip

[ranaroussi]

Things are getting clearer. One last question... is your blotter also running on the remote machine?

[ErnstTmp]

No, blotter, strategy and database are running on the same machine, only TWS is remote.

[ranaroussi]

I'll try to replicate this setup later today and report back.

[ErnstTmp]

Thank you very much for putting in so much work!

I am leaving for Barcelona today, then I have no access to the development set up I have described for the next week.

BTW - do you need some info from NIPS I can relay back to you (like visiting a poster & asking some questions and reporting their answers, attending a talk and reporting back, etc?)

[ranaroussi]

Much appreciated 👍👍👍 tho nothing specific comes to mind. If any interesting links/repos are mentioned - just post them here :)

[ranaroussi]

This was a hard one to catch... Should be fixed in version 1.5.40.

To upgrade, run:

$ pip install qtpylib --upgrade --no-cache-dir

LMK if that worked for you

[ErnstTmp]

Hi Ran,

Thanks for fixing this!!! I really appreciate your work.

...

I am at NIPS. The last 2 days, I was at the European Workshop of Reinforcement Learning.

I learnt about a framework that should be better than conventional RL, it is open source and from John Langford of Microsoft Research. However, it seems not to be ready for trading.

http://hunch.net/~vw/

Some guys there have mentioned that using RL for trading is complicated since it has sparse rewards, and since a trading action does not change the environment, both of this makes it hard for RL.

I'll keep you posted.

Kind regards Ernst

[ranaroussi]

The Vowpal Wabbit project looks very interesting, tho I agree that on first glance at least it's doesn't look like the perfect fit for trading. I'll be following the project just the same - so thanks!! :)

Anyway, I'm glad this issue is resolved. Closing this issue for now. Please re-open if you discover otherwise...

[ErnstTmp]

Hi Ran, thank you very much, for fixing the bug. As promised, my summary from NIPS. Please do not hesitate to contact me if you have questions or need more info. Kind regards Ernst Nips2016TrendsSummary.pdf !

[ranaroussi]

NICE!

Looks like you had a great time 👍

Thanks!