cancel_all_orders_for_asset fails

[abhilashchowdhary]

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: (Windows Version or $ uname --all) macOS Sierra
• Python Version: $ python --version Python 2.7.13
• Python Bitness: $ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))' 64
• How did you install Zipline: (pip, conda, or other (please explain)) pip
• Python packages: $ pip freeze or $ conda list

alembic==0.9.5
alpha-vantage==1.2.1
appnope==0.1.0
backports-abc==0.5
backports.shutil-get-terminal-size==1.0.0
bcolz==0.12.1
bleach==2.0.0
Bottleneck==1.2.1
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
configparser==3.5.0
contextlib2==0.5.5
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.26.1
decorator==4.1.2
empyrical==0.3.2
entrypoints==0.2.3
enum34==1.1.6
functools32==3.2.3.post2
html5lib==0.999999999
idna==2.6
intervaltree==2.1.0
ipykernel==4.6.1
ipython==5.5.0
ipython-genutils==0.2.0
ipywidgets==7.0.0
Jinja2==2.9.6
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.1.0
jupyter-console==5.2.0
jupyter-core==4.3.0
Logbook==1.1.0
lru-dict==1.1.6
Mako==1.0.7
MarkupSafe==1.0
matplotlib==2.0.2
mistune==0.7.4
multipledispatch==0.4.9
nbconvert==5.3.1
nbformat==4.4.0
networkx==1.11
nose==1.3.7
notebook==5.1.0
numexpr==2.6.3
numpy==1.13.1
pandas==0.18.1
pandas-datareader==0.5.0
pandocfilters==1.4.2
pathlib2==2.3.0
patsy==0.4.1
pexpect==4.2.1
pickleshare==0.7.4
plotly==2.0.15
prompt-toolkit==1.0.15
ptyprocess==0.5.2
Pygments==2.2.0
pyparsing==2.2.0
python-dateutil==2.6.1
python-editor==1.0.3
pytz==2017.2
pyzmq==16.0.2
qtconsole==4.3.1
requests==2.18.4
requests-file==1.4.2
requests-ftp==0.3.1
scandir==1.5
scipy==0.19.1
simplegeneric==0.8.1
simplejson==3.11.1
singledispatch==3.4.0.3
six==1.10.0
sortedcontainers==1.5.7
SQLAlchemy==1.1.14
statsmodels==0.8.0
subprocess32==3.2.7
tables==3.4.2
terminado==0.6
testpath==0.3.1
toolz==0.8.2
tornado==4.5.2
traitlets==4.3.2
urllib3==1.22
wcwidth==0.1.7
webencodings==0.5.1
widgetsnbextension==3.0.2
yahoo-finance==1.4.0
zipline==1.1.1

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

cancle_all_orders_for_asset at https://github.com/quantopian/zipline/blob/master/zipline/finance/blotter.py#L199 fails in assert not orders because open_orders dictionary has a order which isn't open and the cancel at https://github.com/quantopian/zipline/blob/master/zipline/finance/blotter.py#L178 cannot cancel a order that is not open .

I get the following stack trace

File "/Users/abhilash/backtest/library/backtest.py", line 410, in run_backtest
    result = trading_algo.run(data, overwrite_sim_params=False)
  File "/Users/abhilash/.virtualenvs/abc/lib/python2.7/site-packages/zipline/algorithm.py", line 709, in run
    for perf in self.get_generator():
  File "/Users/abhilash/.virtualenvs/abc/lib/python2.7/site-packages/zipline/gens/tradesimulation.py", line 225, in transform
    for capital_change_packet in once_a_day(dt):
  File "/Users/abhilash/.virtualenvs/abc/lib/python2.7/site-packages/zipline/gens/tradesimulation.py", line 174, in once_a_day
    self._cleanup_expired_assets(midnight_dt, position_assets)
  File "/Users/abhilash/.virtualenvs/abc/lib/python2.7/site-packages/zipline/gens/tradesimulation.py", line 282, in _cleanup_expired_assets
    blotter.cancel_all_orders_for_asset(asset)
  File "/Users/abhilash/.virtualenvs/abc/lib/python2.7/site-packages/zipline/finance/blotter.py", line 254, in cancel_all_orders_for_asset
    assert not order
AssertionError

Also, how do I change the warn flag equal to True in the function call at https://github.com/quantopian/zipline/blob/master/zipline/finance/blotter.py#L199. Is there any way other than hardcoding it in the file ?

• What did you expect to happen?

I expected the open_orders to not contain order which is not open

• What happened instead?

The open_orders had a order which had open flag as False

Following are the variables in cancle_all_orders_for_asset when it fails

asset.auto_close_date = 2017-09-16 00:00:00+00:00

self =  Blotter(slippage_models={ < type
'zipline.assets._assets.Equity' >: VolumeShareSlippage(volume_limit=0.025,
                                                       price_impact=0.1), < type
'zipline.assets._assets.Future' >: VolatilityVolumeShare(volume_limit=0.05,
                                                         eta= < varies >)}, commission_models = { < type
'zipline.assets._assets.Equity' >: PerShare(cost_per_share=0.0075,
                                            min_trade_cost=1.0), < type
'zipline.assets._assets.Future' >: PerContract(cost_per_contract=0.85,
                                               exchange_fee= < varies >, min_trade_cost = 1.0)}, open_orders = defaultdict( < type
'list' >, {Equity(260[BAS]): [Order(
    {'status': 1, 'created': Timestamp('2016-12-23 21:00:00+0000', tz='UTC'),
     'limit_reached': False, 'stop': None, 'amount': 0, 'stop_reached': False,
     'commission': 0, 'reason': None, 'limit': None, 'sid': Equity(260[BAS]),
     'dt': Timestamp('2016-12-23 21:00:00+0000', tz='UTC'),
     'id': 'a5ed9000087c4a5d9f5b059b68059067', 'filled': 0})],
           u'default_factory': []}), orders = {
                                                  '3190a5ed7d164bcd8bf4ef7d22cf3774': Order(
                                                      {'status': 1,
                                                       'created': Timestamp(
                                                           '2016-12-22 21:00:00+0000',
                                                           tz='UTC'),
                                                       'limit_reached': False,
                                                       'stop': None,
                                                       'amount': -786,
                                                       'stop_reached': False,
                                                       'commission': 5.895,
                                                       'reason': None,
                                                       'limit': None,
                                                       'sid': Equity(260[BAS]),
                                                       'dt': Timestamp(
                                                           '2016-12-23 21:00:00+0000',
                                                           tz='UTC'),
                                                       'id': '3190a5ed7d164bcd8bf4ef7d22cf3774',
                                                       'filled': -786}),
                                                  'a5ed9000087c4a5d9f5b059b68059067': Order(
                                                      {'status': 1,
                                                       'created': Timestamp(
                                                           '2016-12-23 21:00:00+0000',
                                                           tz='UTC'),
                                                       'limit_reached': False,
                                                       'stop': None,
                                                       'amount': 0,
                                                       'stop_reached': False,
                                                       'commission': 0,
                                                       'reason': None,
                                                       'limit': None,
                                                       'sid': Equity(260[BAS]),
                                                       'dt': Timestamp(
                                                           '2016-12-23 21:00:00+0000',
                                                           tz='UTC'),
                                                       'id': 'a5ed9000087c4a5d9f5b059b68059067',
                                                       'filled': 0}),
                                                  '64aa9877b65048cc8e53e507347990d1': Order(
                                                      {'status': 1,
                                                       'created': Timestamp(
                                                           '2016-12-12 21:00:00+0000',
                                                           tz='UTC'),
                                                       'limit_reached': False,
                                                       'stop': None,
                                                       'amount': 224719,
                                                       'stop_reached': False,
                                                       'commission': 1685.3925,
                                                       'reason': None,
                                                       'limit': None,
                                                       'sid': Equity(260[BAS]),
                                                       'dt': Timestamp(
                                                           '2016-12-22 21:00:00+0000',
                                                           tz='UTC'),
                                                       'id': '64aa9877b65048cc8e53e507347990d1',
                                                       'filled': 224719})}, new_orders = [], current_dt = 2017 - 0
9 - 18
00:00:00 + 00:00)

self.open_orders = defaultdict(<type 'list'>, {Equity(260 [BAS]): [Order({'status': 1, 'created': Timestamp('2016-12-23 21:00:00+0000', tz='UTC'), 'limit_reached': False, 'stop': None, 'amount': 0, 'stop_reached': False, 'commission': 0, 'reason': None, 'limit': None, 'sid': Equity(260 [BAS]), 'dt': Timestamp('2016-12-23 21:00:00+0000', tz='UTC'), 'id': 'a5ed9000087c4a5d9f5b059b68059067', 'filled': 0})], u'default_factory': []}) orders = <type 'list'>: [Order({'status': 1, 'created': Timestamp('2016-12-23 21:00:00+0000', tz='UTC'), 'limit_reached': False, 'stop': None, 'amount': 0, 'stop_reached': False, 'commission': 0, 'reason': None, 'limit': None, 'sid': Equity(260 [BAS]), 'dt': Timestamp('2016-12-23 21:00:00+0000', tz='UTC'), 'id': 'a5ed9000087c4a5d9f5b059b68059067', 'filled': 0})]

So order with the id a5ed9000087c4a5d9f5b059b68059067 has the flag open as False but it is still in open_orders

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. 2. 3. ...

What steps have you taken to resolve this already?

...

Anything else?

...

Sincerely, $ whoami

[DavisOwen]

Have the same problem. Is the assert statement really necessary? What if we just don't catch this "error"?

[DavisOwen]

Update: removed the assert line. Code seemed to continue running just fine. I'll try to compare my output on this custom zipline install to the output of same algorithm on quantopian and see how the results compare.

[MathiasGruber]

Also experiencing this issue. Any updates?