search

nautechsystems / nautilus_trader

A high-performance algorithmic trading platform and event-driven backtester
https://nautilustrader.io
GNU Lesser General Public License v3.0
1.73k stars 405 forks source link

Backtest Equity with EOD bar data seems to be broken #1476

Open Flipper1509 opened 5 months ago

Flipper1509 commented 5 months ago

Bug Report

Hello,

I implemented the example strategy from the readme.md, the "minimal EMA Cross strategy example which just uses bar data". Therefore I created an Equity:


    symbol="AAPL"
    venue="SIM"

    myStock = Equity(
            instrument_id=InstrumentId(symbol=Symbol(symbol), venue=Venue(venue)),
            raw_symbol=Symbol(symbol),
            currency=USD,
            price_precision=2,
            price_increment=Price.from_str("0.01"),
            lot_size=Quantity.from_int(100),
            margin_init=Decimal("0.3"),
            margin_maint=Decimal("0.3"),
            ts_event=0,
            ts_init=0,

        )

I feed EOD datato that strategy:

mydata = dw.fetch_daily_data_from_db('AAPL', from_date='2005-01-01', until_date='2023-01-01')

    # Setup wrangler
    myBar_wrangler = BarDataWrangler(
        bar_type=BarType.from_str("AAPL.SIM-1-DAY-LAST-EXTERNAL"),
        instrument=myStock,
    )

    mybars = myBar_wrangler.process(data=mydata)

    # Add data
    engine.add_data(mybars)`

orders are generated and submitted. A market order (GTD, quantity=1) get filled immediately. That means within the same point in time:

2005-09-08T00:00:00.000000000Z [INFO] BACKTESTER-001.EMACross: <--[EVT] OrderInitialized(instrument_id=AAPL.SIM, client_order_id=O-20050908-0000-001-000-154, side=BUY, type=MARKET, quantity=1, time_in_force=GTC, post_only=False, reduce_only=False, quote_quantity=False, options={}, emulation_trigger=NO_TRIGGER, trigger_instrument_id=None, contingency_type=NO_CONTINGENCY, order_list_id=None, linked_order_ids=None, parent_order_id=None, exec_algorithm_id=None, exec_algorithm_params=None, exec_spawn_id=None, tags=from flat to long).

2005-09-08T00:00:00.000000000Z [DEBUG] BACKTESTER-001.Cache: Added MarketOrder(BUY 1 AAPL.SIM MARKET GTC, status=INITIALIZED, client_order_id=O-20050908-0000-001-000-154, venue_order_id=None, position_id=None, tags=from flat to long).

2005-09-08T00:00:00.000000000Z [INFO] BACKTESTER-001.EMACross: <--[EVT] OrderSubmitted(instrument_id=AAPL.SIM, client_order_id=O-20050908-0000-001-000-154, account_id=SIM-001, ts_event=1126137600000000000).

2005-09-08T00:00:00.000000000Z [INFO] BACKTESTER-001.Portfolio: AAPL.SIM margin_init=0.00 USD

2005-09-08T00:00:00.000000000Z [INFO] BACKTESTER-001.Portfolio: Updated AccountState(account_id=SIM-001, account_type=MARGIN, base_currency=USD, is_reported=False, balances=[AccountBalance(total=10_000.00 USD, locked=0.00 USD, free=10_000.00 USD)], margins=[], event_id=dec34ee7-3f06-4167-924e-5769a4f52459).

2005-09-08T00:00:00.000000000Z [DEBUG] BACKTESTER-001.Portfolio: Updated OrderFilled(instrument_id=AAPL.SIM, client_order_id=O-20050908-0000-001-000-154, venue_order_id=SIM-1-001, account_id=SIM-001, trade_id=SIM-1-174, position_id=AAPL.SIM-EMACross-000, order_side=BUY, order_type=MARKET, last_qty=1, last_px=49.78 USD, commission=0.00 USD, liquidity_side=TAKER, ts_event=1126137600000000000).

2005-09-08T00:00:00.000000000Z [INFO] BACKTESTER-001.EMACross: <--[EVT] OrderFilled(instrument_id=AAPL.SIM, client_order_id=O-20050908-0000-001-000-154, venue_order_id=SIM-1-001, account_id=SIM-001, trade_id=SIM-1-174, position_id=AAPL.SIM-EMACross-000, order_side=BUY, order_type=MARKET, last_qty=1, last_px=49.78 USD, commission=0.00 USD, liquidity_side=TAKER, ts_event=1126137600000000000).

2005-09-08T00:00:00.000000000Z [DEBUG] BACKTESTER-001.Cache: Indexed PositionId('AAPL.SIM-EMACross-000'), client_order_id=O-20050908-0000-001-000-154, strategy_id=EMACross-000).

2005-09-08T00:00:00.000000000Z [DEBUG] BACKTESTER-001.Cache: Added Position(id=AAPL.SIM-EMACross-000, strategy_id=EMACross-000).

2005-09-08T00:00:00.000000000Z [INFO] BACKTESTER-001.Portfolio: AAPL.SIM net_position=1

2005-09-08T00:00:00.000000000Z [DEBUG] BACKTESTER-001.Portfolio: Cannot calculate unrealized PnL: no prices for AAPL.SIM.

2005-09-08T00:00:00.000000000Z [INFO] BACKTESTER-001.Portfolio: AAPL.SIM margin_maint=1.49 USD

2005-09-08T00:00:00.000000000Z [INFO] BACKTESTER-001.Portfolio: Updated AccountState(account_id=SIM-001, account_type=MARGIN, base_currency=USD, is_reported=False, balances=[AccountBalance(total=10_000.00 USD, locked=1.49 USD, free=9_998.51 USD)], margins=[MarginBalance(initial=0.00 USD, maintenance=1.49 USD, instrument_id=AAPL.SIM)], event_id=0e5c2b67-a3de-49c8-98bc-ac0e39ff1e90).

2005-09-08T00:00:00.000000000Z [DEBUG] BACKTESTER-001.Portfolio: Updated PositionOpened(instrument_id=AAPL.SIM, position_id=AAPL.SIM-EMACross-000, account_id=SIM-001, opening_order_id=O-20050908-0000-001-000-154, closing_order_id=None, entry=BUY, side=LONG, signed_qty=1.0, quantity=1, peak_qty=1, currency=USD, avg_px_open=49.78, avg_px_close=0.0, realized_return=0.00000, realized_pnl=0.00 USD, unrealized_pnl=0.00 USD, ts_opened=1126137600000000000, ts_last=1126137600000000000, ts_closed=0, duration_ns=0).

A modified market order(time_in_force=AT_THE_OPEN) should do the trick, as far as I understand, because it should be only in force at the trading session open


    def buy_next_opening(self, tags=None) -> None:
          order: MarketOrder = self.order_factory.market(
            instrument_id=self.instrument_id,
            order_side=OrderSide.BUY,
            quantity=self.instrument.make_qty(self.trade_size),
            time_in_force=TimeInForce.AT_THE_OPEN,
            tags=tags,
        )
        self.submit_order(order)`

Expected Behavior

The market order should get filled as soon as the next bar is processed. It should be filled at the opening price from the next bar.

Actual Behavior

The program aborts:


Traceback (most recent call last):
  File "nautilus_trader/backtest/matching_engine.pyx", line 927, in nautilus_trader.backtest.matching_engine.OrderMatchingEngine._process_auction_market_order
TypeError: Argument 'price' has incorrect type (expected nautilus_trader.model.objects.Price, got NoneType)

It seems, that the matching engine tries to access the max_price or the min_price. But those can´t be set :

In nautilus_trader/model/instruments/equity.pyx (line 93) the baseclass gets initialized. But in lines 110 und 111 there is hardcoded max_price=None and min_price=None. No chance to set plausible values there.

Before trying to make those properties setable, I hardcoded a min and a max price in equity.pyx like this:


    max_price=Price.from_int_c(100000),
    min_price=price_increment,

and recompiled the project. The program won't abort anymore, but with those values none of the AT_THE_OPEN orders get filled at all. GTD Orders behave the same way as before.

So there must be more to get EOD trading handled in the matching_engine the right way...

Unfortunately I'm not able to understand the framework deep enough to be more helpfull than this report, sorry.

Steps to Reproduce the Problem

  1. setup a strategy with equity
  2. build market order with time in force AT_THE_OPEN
  3. submit the order

Specifications

cjdsellers commented 5 months ago

Hi @Flipper1509

Thanks for the detailed report and attempt at fixing here.

This time in force hasn't really been very thoroughly tested, and seems there's definitely a bug here. I'll look into it.

fredmonroe commented 1 month ago

Hi @cjdsellers , if you're interested - i'd be open to working on this because i want it to work for market on close i've already looked at it a bit.

cjdsellers commented 1 month ago

Absolutely, you'll probably find plenty of commented out code along the paths. Keeping the test coverage up would be ideal too.

Let me know if you need any help.

fredmonroe commented 1 month ago

Absolutely, you'll probably find plenty of commented out code along the paths. Keeping the test coverage up would be ideal too.

Let me know if you need any help.

@cjdsellers Sorry for the slow progress, it took me a minute to get the environment setup but i have that now and a small python file reproducing the scenario above

As I understand it, there are two issues at play here:

1) Equity instruments don't have a price_min and price_max - they are set to None. The simulation exchanges use them to turn the idea of a market order into that of a limit order with a maximum price in the case of a buy or minimum price in the case of a sell. I wanted to ask your guidance on how you think nautilus philosophically should work, for example:

2) The second issue - implement process_auction_book(self, OrderBook book) which is currently unfinished afaict. It looks to me like if the min and mix price issue is fixed we would run into problems here. Currently, the auction market order would be converted to a book order with a super high (or super low for selling) price and added to either the opening or closing book as appropriate. Then if the market state moves to paused for the open or close - we call process_auction_book to do the match. So we'd need to implement process_auction_book - some issues I notice with that are that we'd have to decide how to simulate that match. If we have full inside book or market data we could match against that, other wise we would need to fall back to bar data which may have bid/ask or may only have trades. The bar data may only be daily as well i guess. So this seems non trivial to handle all these cases - part of me wonders the auction choice made should be configured when the market is setup to make it explicit. For me personally, i'd want my orders matched at the auction price the market used if i was backtesting - so i guess that would be whatever the official close was for the day. But that probably shows an equities bias on my part I'm not sure.

Anyway, thanks for reading all that, I just wanted to check in and see if you had any thoughts or direction before i start writing code or if you see mistakes in my understanding of whats going on above.

cjdsellers commented 1 month ago

Hey @fredmonroe

A little more back story here is this was working at one stage, and was disabled to make implementing the Rust order book easier.

  1. I think we should use the max/min for the instrument if available, otherwise fall back to PRICE_MAX / PRICE_MIN.

  2. For the configuration, looks like we have a commented out auction_match_algo parameter - which could address the auction choice. We have a module auction.py which looks entirely commented out too.

    • When a MarketStatus is passed to process_status then this potentially triggers the auction (it just needs to exist in the data stream for the backtest).
    • The auction match algo will produce traded bids and asks.
    • We should use the most granular data possible and progressively "fall back" to less granular data which would finally be that single closing price. This is also a fairly typical flow with Nautilus used in some other areas related to risk and accounting too. Then we'd have to decide if there simply isn't enough data to have an auction what to do.

@limx0 may have some other thoughts on this.

limx0 commented 1 month ago

Just adding my 2c also:

For me personally, i'd want my orders matched at the auction price the market used if i was backtesting - so i guess that would be whatever the official close was for the day. But that probably shows an equities bias on my part I'm not sure.

Yep I think this is the direction we want to head.

EOD bar data is definitely not where nautilus shines (one of the main selling points is backtest-live parity, its not clear to me what a live version of a backtest like this looks like), so whatever you see as the simplest solution is that gets this functional would be the preference.

PRs are very much appreciated so happy to assist where we can.