Specifics of implementation

[vfilimonov]

Hello Philippe,

first of all, thank you very much for releasing your library and congratulations on it - I like your approach to backtesting very much.

Let me ask you here a couple of questions about specifics of implementation:

1. Perhaps I'm not understanding it correctly, but it looks like you have one day look-ahead bias for momentum strategies: bt.algos.SelectMomentum() is based on bt.algos.StatTotalReturn(), which calculates the return from target.now - self.lookback to target.now. Then the rebalancing is performed by the same price of target.now. Typically one would work on daily closing prices, and it's impossible to enter the market on the same day - one should use next day's close instead. Am I right?
2. Another issue related to look-ahead bias. Suppose we're closing position on the day where we don't have the data any more (e.g. the ticker is already dead / delisted / company went bankrupt). We still need to be able to do that, assuming that we've lost everything (current price is zero). I suggest that this part of bt.core.SecurityBase.allocate() that raises an exception could be removed and NaN should be better treated as zero. Does it make sense?
3. What was the reason for introducing price-independent commissions (i.e. that only depend on quantity)? Usually one adds a slippage as a fraction of target price to the total transaction costs. If I was to introduce that kind of "implementation fees" in bps, would it be enough to change bt.core.SecurityBase.outlay() or are there any other places to modify?
4. I'm also interested in recording each of transactions (which are so far implicit in the engine). Would you rather suggest to modify the backtesting engine itself and build-in the logging system directly to the core, or is it better to implement an "Algo" that is to be called after before bt.algos.Rebalance() and would record all implied transactions?

Best regards Vladimir

[pmorissette]

Hey Vladimir,

Thanks for the kind words - glad you like my approach!

I'll try to answer your questions as best I can - let me know if you have any more questions or if something is not clear.

Q1: This is indeed the case. However, given the type of strategies that I test, I don't think it is really an issue. For example, say you have a monthly rebalancing strategy. On the rebalancing day, you will run the algo a few minutes before the close and execute the necessary trades. Although the price used to calculate the momentum and the trade price might be different, I don't think there will be major discrepancies and nothing would suggest a systematic bias in my favor. Even for daily trades I can't see this being a big issue and as such, I prefer using the current day's price vs. the next day's close.

Furthermore, I don't think you will have a look-ahead bias because the close price is your entry price. Although it is technically not the price that you can trade at, it is quite near and differences should cancel out over time. Look-ahead bias is more prevalent in some other frameworks where returns are used instead of prices. In this case you cannot use the same day's return to determine algo logic (buy/sell) and calculate the performance. You must take the next day's return as your return. I hope this makes sense...

That being said, if you were to use this framework on an intra-day basis, then this would probably become more of an issue and in that case it would indeed be better to be more conservative and take the next bar's close. Maybe we should add some kind of option to Backtest that would modulate this behavior?

Q2: Good catch. I had not yet come across a backtest with this need since I mostly focused on ETFs and futures, but this would indeed be an issue with a dead / delisted / bankrupt company. I think this should be relatively easy to implement since we can check if the position is non-zero and act appropriately. I think we still would need to throw some kind of exception if someone tries to initiate a position on a NaN priced security though.

However, now that I think about it, you could have some tricky bugs here. For example, say you rebalanced at time t and for some reason (exchange closed or something like this) some securities had NaN values on this date only (while others did not). You cannot simply replace NaN for 0 here and assume the company is no longer operational...

I need more time to think this one through - maybe a distinction between NaN and 0 - 0 meaning bankrupt and NaN meaning missing data. This way if we have a non-zero position and a 0 price, we know that the company is bankrupt/removed and we can deal with it. If it is simply missing data, then it might be up to the user to fill the NaNs with appropriate values...

Q3: Very good point once again. I think this should indeed be part of the calculation (and up to the user to use this piece of data or not). Therefore, bt.core.StrategyBase.commission_fn should accept both quantity and price (we can add price as second argument to avoid breaking existing code). It will then be up to the user to determine how and what to use. I assume slippage could be included in the commission function as well as I don't think a separate function is necessary here.

Q4: Depends what you need here. As you mentioned, the easiest way would simply be to add some kind of logging algo and record the change in each security's position (if different from 0). That way you could pop this algo in when needed and it wouldn't affect the core implementation's performance when not in use. In my experience, I rarely look at detailed transaction logs - I prefer looking at security weights, for example, to make sure everything is running along smoothly.

Hope this helps!

Cheers, Phil

[vfilimonov]

Hi Phil,

thank you very much for your detailed answer!

Q1: I think that it would be easier to have an option for bt.algos.SelectMomentum(). Indeed depending on the statistics, bt.algos.SelectN() could be also subjected to this issue, but it should be then user's responsibility to calculate statistics in a responsible way. I partially agree with you that in most cases 60-30 minutes before closing might give a good approximation for the closing price, but before I do tests to check this, I would not be 100% sure. :)

Q2. I agree, it's better to allow to close position at zero price, but raise an exception at opening such position.

Q3. Agree - better to keep it within one function.

Q4. For me transaction logs alone are useful to double-check P&L calculation, and if complemented with signal values, I found them useful to dissect the strategy. Possibly the most use of them one can find when analyzing a portfolio of strategies on the same asset space. Then change in weight does not fully reflect the decision process, as it could be changed simultaneously by two strategies running in parallel.

I haven't chosen the framework for my backtesting yet, but most likely I'll build on yours. In this case I'll submit PRs for Q1-Q3 :) BTW, are you still working on this project/library, or not any more?

Best Vladimir

[pmorissette]

Hey Vladimir,

My pleasure. Glad to see you are interested in the framework.

Q1: Yeah upon further thought, I do agree that this should be settled at the algo level and not the framework level. Adding an option to existing algos such as `bt.algos.SelectMomentum()`` sounds like a sensible approach. I still believe that at the daily level and longer, calculating the momentum return and trading at the same price should not be a big deal as this is more or less what you would do in practice - calculate the metric, sort the stocks, pick the top n stocks and execute the orders x minutes before the close (or at any time really - the point being that you will calculate and execute with roughly the same price).

Q4: Good point on portfolios of strategies - it can get pretty confusing! It is definitely a good debugging tool and would be a great addition to the default set of algos.

I am currently working on a new project at work that does not rely on bt (although it might in the future). As such, I cannot devote as much time as when I first published the code, but I do plan on maintaining the code and contributing new code in my spare time. Hopefully, you and others will be able to help me out :) I think this framework has a lot of potential and I would love to see it grow!

Cheers, Phil