Open sdmovie opened 4 years ago
Afaik, there were no prior efforts of getting real, live trading in. I guess _Broker
is certainly to be overridden. Just please do base your findings upon #47 as I do eventually intend to get that in.
after furthur study, I think practical approch is to mirror backtesting.py to a realtrading.py file (not disturbing original backtesting.py), still share Strategy class from backtesting.py. Backtest class, _broker, position need rewrite in new file, order/trade not sure as of now, probably keep as mirror of trade serverside data. No obvious barrier seen now for the real trade implementation(very broker API specific) :-) The way of passing dynamic data into _broker,strategy using a _Data class is really smart , will copy the method in the realtrading.
I envision live trading by way of a separate class, in Python type notation, say:
from typing import Type, Union, Literal, NoReturn
LiveTrade(strategy: Type[Strategy],
period: str,
broker: Union[Literal["ib", "oanda", "ccxt"], Broker],
**kwargs) -> NoReturn
The idea is we can reuse the same Strategy
(hence we must adapt the _Broker
), we must provide the trading cyclic period which to call Strategy.next()
on (should likely match the period in bars of backtested-on data), and the chosen broker backend (here a string, mapping to each standalone class internally). Extra kwargs (a list of instruments, margins, and such) would be passed to the broker.
The common broker API is sure in the domain of its implementer. :smile:
@kernc This needs some support, there has to be an abstract base class for live trading so that people can implement the Brokers of their choice as they like and ccxt library( for Crypto ) support as an example.
cc @ttfreeman Tagging you in case you are interested.
@arunavo4 Yes, definitely interested, though need to research on the best way to tackle this prior to diving in. I like @kernc 's suggestion to have a separate class (LiveTrade) but we might also need another class (LiveData) that would bridge the live data from broker to LiveTrade class.
feed=LiveData(....init params)
feed.connect(localhost, 567, ...)
LiveTrade( feed.data,
strategy: Type[Strategy],
period: str,
broker: Union[Literal["ib", "oanda", "ccxt"], Broker],
**kwargs) -> NoReturn
Data acquisition can surely be handled by the chosen broker (i.e. Broker
)? Broker implementations should take their configuration parameters from **kwargs
.
practically my current process to live trading is rewrite strategy ( slight modification - thanks to the backtesting's clear structure and pandas compatible data carrier) , to fit in live broker/data source. Building a BT compatible live trading framework is not a small project - considering BT is single equity, single datafeed( though resample applicable) , while trading is diversified. Now I use backtesting.py to verify my alorithm( or subset of algo) which works pefectly.Then write trading in a flexible way. ps. I give up portion buy/sell in BT, in real world(trading), broker has no percentage trade API, strategy is responsible of calculating target buy/sell volume, also 'cause calculation for buy-open/sell-open/buy-close/sell-close volume and margin is too complicate for me.
I'd recommend keeping data feeds and brokers separate responsibilities. That way theoretically one could subscribe to any data source then execute trades with the broker of their choice without being tightly coupled to anything.
Definitely agree that having a path for people to BYOB (broker) is the way to go. Then strategy.buy() or strategy.sell() is executed on their broker of choice, even if it is a backtest broker.
Has anyone started work on this (that's shareable)?
@crazy25000 Have started working on it. but it's very integrated into my project. The gist is that we can easily (maybe not too easy) Make a class Similar to Backtest. Which can have Corresponding Broker Classes like CCXT/ Oanda/ Alpaca.
Waiting
For those who are interested I found a workaround to generate trading signals with backetsting.py. I use a custom strategy class that extends the standard one:
This does not require any modification to the backtesting.py source code.
@crazy25000 Have started working on it. but it's very integrated into my project. The gist is that we can easily (maybe not too easy) Make a class Similar to Backtest. Which can have Corresponding Broker Classes like CCXT/ Oanda/ Alpaca.
@arunavo4 could you share how your LiveTrade Class looks like?
Any updates on this? I would love to see this feature added.
Hi everyone, I am probably late to the party but I wanted to share that I am implementing similar ideas. I do use ccxt as broker for live trading and I managed to implement a Strategy object in my framework which extends the backtesting.py Strategy. My Strategy has a run_live() and a run_backtest() the latter calls BackTesting with self. The tricky part and not so elegant perhaps is that both frameworks have models such as Order and Position which aren’t really compatible. I refrained from going down the way of implementing a Broker extension of _Broker because it has clearly a backtesting design and purpose at the moment. It would be great if Broker would be more “open” to subclassing and alternative implementations. I’ll keep you posted on the progress if someone is interested.
Hi everyone, I am probably late to the party but I wanted to share that I am implementing similar ideas. I do use ccxt as broker for live trading and I managed to implement a Strategy object in my framework which extends the backtesting.py Strategy. My Strategy has a run_live() and a run_backtest() the latter calls BackTesting with self. The tricky part and not so elegant perhaps is that both frameworks have models such as Order and Position which aren’t really compatible. I refrained from going down the way of implementing a Broker extension of _Broker because it has clearly a backtesting design and purpose at the moment. It would be great if Broker would be more “open” to subclassing and alternative implementations. I’ll keep you posted on the progress if someone is interested.
I'm about to do the same thing, can you share a gist of how you went about takcling it? Just some partial code would be great.
I'm about to do the same thing, can you share a gist of how you went about takcling it? Just some partial code would be great.
Hey @hakunin perhaps we can join forces on this, provided @kernc will provide support for merging later on. Here are some of the issues which would require a Backtesting.by redesign, as of now I am solving them with real-time override (not nice):
Strategy.I()
to support re-evaluation of live candlestick data, which are needed for live trading. At the moment I have implemented a method that swaps the _Data()
object in self._data
when there are new candlesticks available. Not really clean but kind of does the trick. A much better design would be for the logic to support paper-trading and live-trading without having to swap data under the carpet;Strategy.next()
might be very simple and flexible, it is also leaving quite a lot on the plate of the user in terms of checking balances, order types... In my original framework I have started with the idea of just having a StrategyStep
as an object, and having a Strategy.add_step(StrategyStep)
to create whatever sequence of steps one may desire. At the end of an execution cycle, the OrderProcessor
will take care of checking open orders, new orders... I have simplified this to facilitate unexperienced users, at the moment only several of my close friends, to create their own strategies by adding the following helpers:
should_buy()
: where the user can implement the logic necessary when should a position be opened;should_sell()
: same for selling.should_exit()
: when it is time to exit, with profit or loss.do_buy()
, do_sell()
, do_exit()
will implement the customized logic when the condition matches. Each of the "should" returns a boolean, each of the "do" returns a List[Order]
. While this flow can be incapsulated in the .next()
it introduces difficulties with the downloading of new data, and the replacing of the "current" indicators (see point 1).BackTesting
, PaperTrading
, LiveTrading
. As the Runner instantiates the strategies and calls the .run()
with the execution mode to get started, it is quite tricky to integrate the BackTesting mode with Backtesting.py because the Backtest
object that wants to instantiate a Strategy. It would be great if it were possible to bass to Backtest
not only a class
but also an instance
so that there wouldn't be different instances of the same strategy class at the same time. Because of a strategy dealing with multiple markets (see point 2) it would be more effective to encapsulate the execution of the backtesting within the strategy, rather than having to run it outside and then aggregate the results. At the moment I am running the backtesting for each of the markets, and then aggregating the data at the end, very slow and not effective.Order
: there are quite a lot of more information needed when trading live, than what is now visibleBroker
: I have created an Exchange
class that wraps the CCXT library to connect to different exchanges, I would love to have it extend Broker
if it were more implementation independent (i.e.: depending on a lot of internal classes _Data, _Array...)Position
: same as orderAccount
: here things can really get complicated with different currencies conversion and future vs spot trading...While I am far from being done, and I have to confess I have considered multiple times to give up with the integration, I am still trying and hoping that things will evolve towards a more open framework approach, which probably won't bother the final users, but would definitely allow the Backtesting.py to become part of many trading solutions out there 😉 I will keep you posted on the progress and as soon as I will have an end-to-end working sample I will share it with you.
For those who are interested I found a workaround to generate trading signals with backetsting.py. I use a custom strategy class that extends the standard one:
- The strategy class has a bool var to determine whether I want to run a backtest or check the conditions for the latest data.
- In the init call we have the access to the entire data object, so if my bool var is set for generating signals I evaluate the entry conditions right there.
- I have the methods for checking long and short entry conditions in my strategy class where I use indices for prices and indicators relative to the end of the data array (ie. [-1], [-2] etc.), so the methods work the same in backtesting and signal generation mode.
- After evaluating the entry conditions I immediately check the order list and save the orders in an array to generate trading alerts elsewhere in the code.
This does not require any modification to the backtesting.py source code.
Live trading is a must have feature I think and I am really interested in which files or where to add those modifications. I have multiple indicators and added the bool in the custom strategy class. I've made similar modifications but I got an error and trying to solve this: `Indicators must return (optionally a tuple of) numpy.arrays of same length as 'data' (data shape: (11830,); indicator "wt_indicator(df)"shape: (7, 11831), returned value: [[ 0. 0. 0. ... 0. 0.
I have write a draft version based on comments above,
https://github.com/OppOops/backtesting.py/commit/1879d9d16dd1029c0f8a829249191c298101b84d
the codes needs original Backtest
class to run.
from backtesting.live import LiveTrade, LiveMarketOhlcv
bt = Backtest(df_partial, SmaCross, commission=.002,
exclusive_orders=True)
live_trade = LiveTrade(bt)
For workaround here, I clone a mirror strategy and a mirror data instance.
Also, modify the _Data
class to the _DataCachePatch
and add antorher new class _ObjectBindingDict
.
Below are some script codes modified from the example in Readme.md: (full version at: https://gist.github.com/OppOops/282a2fb3d07019618414d2a51e9084a1)
def run_with_live_new_ohlcv(live_trade: LiveTrade):
# select data from external (iteration not run currently)
...
live_trade.on_bar(LiveMarketOhlcv(...))
live_trade.run_next()
bt = Backtest(df_partial, SmaCross, commission=.002,
exclusive_orders=True)
live_trade = LiveTrade(bt)
live_trade.init()
live_trade.run(to_end=True)
...
run_with_live_new_ohlcv(live_trade)
You can controll LiveTrade
class with backtest core to run or stop at any Ohlcv bar.
Also with on_bar
to insert new ohlcv data to it.
Use some other features like close_last_positions
, process_orders
to generate or submit your orders
I haven't test detailed in my version, but just comaparsion of GOOG
data result with sample.
Currently the value of broker._equity[-1]
will be different with original version.
(I have test entry price and trade size are same mostly, but need further checking.)
I have developed a plugin for live trading, but it's made up of homemade code and contains a lot of customizations. Therefore, I cannot release it yet. However, the basic logic is similar to that of backtrader's store.
To begin, you will need to create a new class called "LiveTrading" that will replace "Backtest." This class will trigger the "strategy.next()" and "strategy.init()" functions every time new data is received. (I'll explain why the "strategy.init()" function is necessary later on.)
Next, you will need to create a "DataFeed" class to fetch the data via ccxt/api/websocket. You can set it up however you like.
After that, create a new broker based on the broker's sdk/api. Since I trade in cryptocurrency, I created one based on ccxt. I also created a paper trading broker for testing purposes.
In "backtesting.py," all indicators are calculated when "Backtest" is initialized within the "strategy.init()" function. Therefore, it is necessary to recalculate the indicator each time new data is received. However, the current logic of "backtesting.py" will dump the indicator function, making it impossible to recalculate. A lazy solution would be to re-call the "strategy.init()" function. However, if you have other states inside "strategy.init()" (which is very common), you will lose the state each time you call "strategy.init()." To address this, I created another abstract method for the "Strategy" class that is dedicated to state initialization, and ensured that it is called only once.
With the "DataFeed"/ "LiveTrading" (replacing "Backtest")/ "Broker," you can easily build your own robot with just a few modifications to your strategy code (making it less likely to introduce new bugs after modification).
My approach into using the backtesting.py together with live trading is a bit different: Created a Multi-bot that trades multiple pairs with Binance.
bt = Backtest(_df,
bot.strategy,
cash=self.cash,
commission=self.commission,
exclusive_orders=True,
trade_on_close=True,
size_second=self.size_second
)
The last param size_second is of my own adding to know if we want to use fixed amount or legacy behavior. Somewhere in the backtesting.py I have placed (among other half a dozen places to prevent asserts):
# update the size, calculating it from last_price if size_second
size_orig = size
size = size / self.last_price if self._size_second else float(size)
To get my signal next couple of things is done:
# Snippet from the Strategy (SuperTrend):
def next_signal(self):
signal = 'None'
if crossover(self.data.Close, self.st):
# self.buy(size=self.position_size) # Was 0.99
signal = 'buy'
elif crossover(self.st, self.data.Close):
# self.position.close()
signal = 'sell'
return signal
strategy=Backtest(df, bot.strategy, cash=self.cash, commission=self.commission, exclusive_orders=True)
strategy.run(**settings) signal = strategy._results._strategy.next_signal()
For the **compute_stats** it gets a bit more complicated.
We need equity, list of trades and ohlcv data.
So at every candle/trade these are saved externaly so that the following can be executed to get stats:
```python
...
# My own local copy and the imports, had to change __init__.py, added _Broker to allow its import
from common.backtesting import Trade, _Broker
from common.backtesting._stats import compute_stats
# Note: self._trades is a list()
if trade_status == 'Bought':
# Keep track of trading for compute stats
broker = _Broker(data=self._data, cash=self.cash, commission=2 * self.trade_commission, margin=1,
trade_on_close=True, hedging=False, exclusive_orders=True, size_second=True,
index=self._data.index)
self._trade = Trade(broker=broker, size=executed_qty, entry_price=avg_price, entry_bar=self.candle_cnt - 1)
elif trade_status == 'Sold':
with suppress(AttributeError):
# Updating the _trade exit_price with avg_price from Market Order from Binance
self._trade._replace(exit_price=avg_price, exit_bar=self.candle_cnt-1)
self._trades += self._trade,
# And Finally
equity = pd.Series(self._data.equity).bfill().fillna(self.cash).values
stats = compute_stats(
trades=self._trades, # broker.closed_trades,
equity=equity,
ohlc_data=self._data,
risk_free_rate=0.0,
strategy_instance=None # strategy,
)
# Then cleaning the protected element
stats_clean = {k: v for k, v in stats.items() if not str(k).startswith('_')}
Quite cool, the backtesting.py.
who can integration live trade to Strategy
Expected Behavior
I am trying to bind the realtrade into this frame work. Of course replacements of Backtest , _Broker , and datafeeding need rewrite. I got them (data API, broker API) on hand and had some ideas to plant in. But I want to hear from @kernc you on your advices/thought first before start, esp. if there are already a reference ~
Actual Behavior
Steps to Reproduce
1. 2. 3.
Additional info