alpacahq / alpaca-backtrader-api

Alpaca Trading API integrated with backtrader
https://pypi.org/project/alpaca-backtrader-api/
Apache License 2.0
622 stars 146 forks source link

Position/Notifications not updated for bracket orders #69

Closed nelfata closed 4 years ago

nelfata commented 4 years ago

After a buy order has been placed with the LIVE data feed, and was properly accepted. Monitoring the position from next() does not match what is shown on Alpaca's site. That can be easily tested by performing a buy order through the alpaca-trade-api, then closing the position on the Alpaca website. The position does not get updated in the next() function.

I am also experiencing missing notifications for bracket orders (buy limit, sell stop, sell limit). The first buy order notifications is received, but once the stop or sell limits are eventually triggered, no notifications are sent to notify_order().

This is working fine for non-live trading.

nelfata commented 4 years ago

Here is the strategy I have used:

class Template(bt.Strategy):
    """A simple strategy template"""
    params = {'slow'        : 20,
              'usebracket'  : True,
              'rawbracket'  : False,
              'pentry'      : 0,   # % below previous close
              'plimit'      : 5,  # % limit price (make profit)
              'pstop'       : 2,   # % stop price (stop loss)
              'valid'       : 1,   #
              }

    def __init__(self):
        self.slowma = dict()
        self._addobserver(True, bt.observers.BuySell)

        self.buycnt  = 0
        self.sellcnt = 0
        self.total   = 0

        self.tLastOrder = time.time()

        # create array of dictionary
        self.o = (dict())  # orders per data (main, stop, limit, manual-close)
        # fill the array index with symbols each with its own dictionary
        for sym in self.getdatanames():
            self.o[sym] = dict()
        for sym in self.getdatanames():
            # The moving averages
            self.slowma[sym] = bt.indicators.SimpleMovingAverage( self.getdatabyname(sym),      # The symbol for the moving average
                                                                period=self.params.slow,    # Slow moving average
                                                                plotname="SlowMA: " + sym)

    def log(self, txt, dt=None):
        dt = dt or self.datas[0].datetime.datetime(0)
        print(dt.strftime('%m-%d %H:%M:%S'), txt)

    def notify_cashvalue(self, cash, value):
        return
        self.log('Cash %s Value %s' % (cash, value))

    def notify_trade(self, trade):
        sym = trade.getdataname()
        #self.log(f"{sym}: SUBMITTING trade: {trade.size:.2f} {trade.price:.2f}")
        if trade.isclosed:
            #self.log(f"{sym}: CLOSED gross {trade.pnl:.2f} net {trade.pnlcomm:.2f}")
            self.log(f"{sym}: PROFIT {trade.pnlcomm:.2f}")

    def notify_order(self, order):
        sym    = order.data._name
        ref    = order.ref
        date   = self.datetime.date()
        status = order.getstatusname()

        # get the order from the
        whichord = ['main', 'stop', 'limit', 'close']
        dorders  = self.o[sym][order.data]
        idx      = dorders.index(order)

        self.log(f'   {sym}: ref {ref} {status}: {whichord[idx]}')

        # show some possible errors
        if order.status in [order.Expired, order.Margin, order.Rejected]:
            self.log(f'   {sym}: ref {ref} {status}: {whichord[idx]}')

        # NOTE:
        # occasionally in live trading we receive Partial instead of Completed even though the order has been filled

        if order.status in [order.Completed, order.Partial, order.Canceled, order.Expired, order.Margin, order.Rejected]:
            # order canceled, remove it
            #self.log(f'   {sym}: ref {ref} {status}: {whichord[idx]}')
            try:
                dorders[idx] = None
                self.log(f'   {sym}: ref {ref} removed: {whichord[idx]}')
                if all(x is None for x in dorders):
                    dorders[:] = []  # empty list - New orders allowed
                    self.log(f'   {sym}: order list empty')
            except:
                print('   ERR *** order.data empty')

    def next(self):
        """Define what will be done in a single step, including creating and closing trades"""
        for sym in self.getdatanames():    # Looping through all symbols
            d     = self.getdatabyname(sym)
            pos   = self.getpositionbyname(sym) # pos.size, pos.price
            dt    = self.datetime.datetime(0)
            qty   = self.getsizing(d, isbuy=True)
            price = d.close[0]

            self.log(f'{sym}: CHECK  {price:.2f} {pos.size:.2f} {pos.price:.2f} {time.time()}')

            # no position, no orders
            if not pos.size and not self.o[sym].get(d, None):
                # Consider the possibility of entrance
                # Notice the indexing; [0] always means the present bar, and [-1] the bar immediately preceding
                # Thus, the condition below translates to: "If today the regime is bullish (greater than
                # 0) and yesterday the regime was not bullish"

                if True: #self.slowma[sym][0] > self.slowma[sym][-1]:    # A buy signal

                    # upon startup, not all data is valid, so we wait a few seconds...
                    # also we need to allow time between placing orders...this is still under test
                    if time.time() - self.tLastOrder < 10.0:
                       return
                    self.tLastOrder = time.time()

                    self.log(f'{sym}: CHECK  {price:.2f} {pos.size:.2f} {pos.price:.2f} {time.time()}')

                    if self.params.usebracket:
                        price = price * (1.0 - self.params.pentry / 100.0)
                        pstp  = price * (1.0 - self.params.pstop  / 100.0)
                        plmt  = price * (1.0 + self.params.plimit / 100.0)
                        valid = datetime.timedelta(self.params.valid)

                        #pstp  = 118.55
                        #plmt  = 123.64

                        self.log(f'{sym}: BUY BRKT @{pstp:.2f} {price:.2f} {plmt:.2f} {valid} qty: {qty:.2f}')

                        if self.p.rawbracket:
                            o1 = self.buy (data=d, exectype=bt.Order.Limit, price=price, valid=valid, transmit=False)
                            o2 = self.sell(data=d, exectype=bt.Order.Stop,  price=pstp, size=o1.size,transmit=False, parent=o1)
                            o3 = self.sell(data=d, exectype=bt.Order.Limit, price=plmt, size=o1.size,transmit=True,  parent=o1)
                            self.o[sym][d] = [o1, o2, o3]
                        else:
                            self.o[sym][d] = self.buy_bracket(  data       = d,
                                                                price      = price,
                                                                stopprice  = pstp,
                                                                limitprice = plmt,
                                                                exectype   = bt.Order.Market,
                                                                #oargs      = dict(valid=valid)
                                                                    )
                        #self.log('{}: BUY REF [Main {} Stp {} Lmt {}]'.format(sym, *(x.ref for x in self.o[sym][d])))
                    else:
                        self.o[sym][d] = [self.buy(data=d)]
                        self.log(f'{sym}: BUY @{price:.2f} qty: {qty:.2f}')
                        #self.log('{}: Buy {}'.format(sym, self.o[sym][d][0].ref))

            else:    # We have an open position
                return
                if self.slowma[sym][0] < self.slowma[sym][-1]:    # A sell signal
                    self.log(f'{sym}: SELL *qty {self.getsizing(d, isbuy=False):.2f}')
                    o = self.close(data=d)
                    try:
                        self.o[sym][d].append(o)  # manual order to list of orders
                    except:
                        print('Not part of order')
                        #return
                    try:
                        #for xo in self.o[sym][d]:
                        #    if xo is not None:
                        #        print('REF ',xo.ref)
                        #self.log('{}: CLOSE [Main {} Stp {} Lmt {}]'.format(sym, *(x.ref for x in self.o[sym][d])))
                        self.log('{}: Manual Close {}'.format(sym, o.ref))
                    except:
                        self.log('{sym}: No ref')
                    if self.p.usebracket:
                        try:
                            o = self.cancel(self.o[sym][d][1])  # cancel stop side
                            self.log('{}: Cancel {}'.format(sym, self.o[sym][d][1].ref))
                        except:
                            #print(self.o[sym][d])
                            #print('bad order sequence')
                            pass
shlomiku commented 4 years ago

first try installing the most updated code for both packages like so: pip install -U git+https://github.com/alpacahq/alpaca-trade-api-python pip install -U git+https://github.com/alpacahq/alpaca-backtrader-api and let's see if this issue still occurs.

nelfata commented 4 years ago

I tried the latest updates that you mentioned on the live market today, but with the same behavior. Cancelling pending orders (stop on loss) does not get any notifications and the position is not updated. This is based on the code that is based above. Please keep in mind that the code is written to handle multiple symbols (only one is in use). I may be using the wrong class members, but the code is based on: https://www.backtrader.com/blog/posts/2017-04-09-multi-example/multi-example/

nelfata commented 4 years ago

Ctrl-C out of the program shows the following (if that might help):

await handler(self, channel, ent) 06-15 00:00:00 TVIX: CHECK 168.92 0.00 0.00 1592241781.8473768 06-15 00:00:00 TVIX: BUY BRKT @165.54 168.92 177.37 1 day, 0:00:00 qty: 526.67 06-15 13:23:00 TVIX: ref 1 Submitted: main 06-15 13:23:00 TVIX: ref 2 Submitted: stop 06-15 13:23:00 TVIX: ref 3 Submitted: limit 06-15 13:23:01 TVIX: ref 1 Accepted: main 06-15 13:23:01 TVIX: ref 2 Accepted: stop 06-15 13:23:01 TVIX: ref 3 Accepted: limit 06-15 13:23:01 TVIX: ref 1 Accepted: main 06-15 13:23:02 TVIX: ref 2 Accepted: stop 06-15 13:23:02 TVIX: ref 3 Accepted: limit 06-15 13:23:02 TVIX: ref 1 Partial: main 06-15 13:23:02 TVIX: ref 1 removed: main Traceback (most recent call last): File "C:\Users\NEF\Desktop\NextCloud\Clones\Projects\Trading\Backtrader\btTest1.py", line 100, in results = cerebro.run() File "C:\Users\NEF\AppData\Local\Programs\Python\Python37\lib\site-packages\backtrader\cerebro.py", line 1127, in run runstrat = self.runstrategies(iterstrat) File "C:\Users\NEF\AppData\Local\Programs\Python\Python37\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies self._runnext(runstrats) File "C:\Users\NEF\AppData\Local\Programs\Python\Python37\lib\site-packages\backtrader\cerebro.py", line 1542, in _runnext drets.append(d.next(ticks=False)) File "C:\Users\NEF\AppData\Local\Programs\Python\Python37\lib\site-packages\backtrader\feed.py", line 407, in next ret = self.load() File "C:\Users\NEF\AppData\Local\Programs\Python\Python37\lib\site-packages\backtrader\feed.py", line 479, in load _loadret = self._load() File "C:\Users\NEF\AppData\Local\Programs\Python\Python37\lib\site-packages\alpaca_backtrader_api\alpacadata.py", line 259, in _load self.qlive.get(timeout=self._qcheck)) File "C:\Users\NEF\AppData\Local\Programs\Python\Python37\lib\queue.py", line 179, in get self.not_empty.wait(remaining) File "C:\Users\NEF\AppData\Local\Programs\Python\Python37\lib\threading.py", line 300, in wait gotit = waiter.acquire(True, timeout) KeyboardInterrupt

shlomiku commented 4 years ago

That can be easily tested by performing a buy order through the alpaca-trade-api, then closing the position on the Alpaca website.

what you have to understand is that this repo is an integration between 2 different entities. each entity manages its own resources (account, porfolio, positions, ..) so when you change it on the web, you basically creating a difficult situation for the alpaca-backtrader instance, because the data is really stored on the alpaca servers. not on your local running instance. one solution (which I actually tried before) is to always make sure the positions are synced but, that creates an overload api requests to the servers.

now, there's another way for you get the exact and synced data, and it's by doing the api call from next() so if you do this: self.broker.update_positions() you will get the exact position values even if you change it in the website

nelfata commented 4 years ago

You have a point here, but I think the data should never be stale when performing live trading. The fact that backtesting works and live trading does not, shows that some fundamental design needs to be reviewed, I am assuming here that this is not a library for backtesting only, but also for live trading. It is ok if we need to perform extra steps for live trading, but this is not documented. The position is not the only problem, the notification updates for all the transactions are also not reported properly during live trading. This is just my opinion as I am also interested in live trading and this library is great.

shlomiku commented 4 years ago

image

I just filled an order on the web and got the transaction inside the app the entire processing of the transaction relies on having an order object attached to it. and we don't. because we didn't generate the order inside the app so we cannot process it. so we cannot notify the user about the transaction

shlomiku commented 4 years ago

I have created a patch that updates the positions even if we didn't generate the order inside the app. try it, and let me know if that helps you it's in this PR: https://github.com/alpacahq/alpaca-backtrader-api/pull/72 and you could install it like this: pip install -U git+https://github.com/alpacahq/alpaca-backtrader-api@update_positions_outside_of_scope

nelfata commented 4 years ago

Thanks for the follow up. I tried your changes, but nothing changed. I made a bracket order as shown in the code above. The main order was filled the other orders were still open. Cancelling the open orders did not update and closing the position did not update. On the third try, the main order was filled but for some reason it did not update in the app.

ghost commented 4 years ago

@shlomikushchi Hey, I've been facing the same issue any help on how to solve it?

arahmed24 commented 4 years ago

@shlomikushchi I was trying to review/understand "def _t_order_create(self):" and it seems like the store only stores the parent mapping and does not store the bracket child orders and thats why the status does not update the order properly.

Here's the line which maps the parent only.

self._ordersrev[oid] = oref # maps ids to backtrader order

Any help would be greatly appreciated or lets us know how to map children ids and we update the code and test/valid.

Thanks in advance.

cc: @ichippa96

shlomiku commented 4 years ago

Hi guys, thanks for the input. I'm working on this issue still. I will update you once I have anything new

arahmed24 commented 4 years ago

@shlomikushchi Thanks for the update.

We will try to update the children ids in the list and see how that works out for us. We will keep you posted as well.

Thanks

shlomiku commented 4 years ago

so the issue is that when we get the notification, we don't have a backtrader order object. this is because when it is created in the API it is a new order, not part of the backtrader bracket order. so it is difficult to notify through the regular pipes (still trying) but for now, I can notify you through notify_store. will this help you guys with the issue?

ghost commented 4 years ago

@shlomikushchi we managed to fix it by iterating through the legs of the parent order and assigning the child/legs order the parents ref as well.

def _t_order_create(self):
        while True:
            try:
                # if self.q_ordercreate.empty():
                #     continue
                msg = self.q_ordercreate.get()
                if msg is None:
                    break
                print(f'{datetime.now()} msg in _t_order_create: {msg}')
                oref, okwargs = msg
                try:
                    o = self.oapi.submit_order(**okwargs)
                except Exception as e:
                    self.put_notification(e)
                    self.broker._reject(oref)
                    return

                try:
                    oid = o.id
                except Exception:
                    if 'code' in o._raw:
                        self.put_notification(o.message)
                    else:
                        self.put_notification(
                            "General error from the Alpaca server")
                    self.broker._reject(oref)
                    return
                self._orders[oref] = oid
                self.broker._submit(oref)

                if okwargs['type'] == 'market':
                    self.broker._accept(oref)  # taken immediately

                oids=list()
                oids.append(oid)
                if o.legs is not None:
                    for leg in o.legs:
                        oids.append(leg.id)

                self._orders[oref] = oids[0]
                self.broker._submit(oref)

                if okwargs['type'] == 'market':
                    self.broker._accept(oref)  # taken immediately

                for oid in oids:
                    self._ordersrev[oid] = oref  # maps ids to backtrader order

                    # An transaction may have happened and was stored
                    tpending = self._transpend[oid]
                    tpending.append(None)  # eom marker
                    while True:
                        trans = tpending.popleft()
                        if trans is None:
                            break
                        self._process_transaction(oid, trans)
            except Exception as e:
                print(str(e))
shlomiku commented 4 years ago

thanks, I will try it tomorrow when the market opens

arahmed24 commented 4 years ago

@shlomikushchi Thanks!

We made progress and it works when the take/limit side is executed versus the stop side. When the stop is executed we get an issue processing replaced status of take/limit side. We tried to execute _cancel for the take/limit side but it's giving us an error. We are thinking of trying to either just do nothing (just return) or write a new cancel function to handle replaced order.

Thanks

shlomiku commented 4 years ago

Hi Guys, I created this PR that addresses this issue: https://github.com/alpacahq/alpaca-backtrader-api/pull/77 thank you @arahmed24 @ichippa96 and Ibrahim for helping out with this issue. it will be merged after I am sure it is working correctly.

you could install it like this: pip install -U git+https://github.com/alpacahq/alpaca-backtrader-api@update_bracket_orders