difficulties recreating alphalens simulated performance in zipline

[tdrobbin]

Hello, I am testing a factor and having some difficulty recreating in zipline the simulated performance computed by alphalens. My factor is 50% long / 50% short and constructed such that when I pass it to performance.utils.create_pyfolio_input the positions returned are the same as the factor passed in.

This might be more of a zipline issue but I would very much like to be able to take a factor I'm testing with alphalens and have some bioler plate in zipline to test, add tcosts, optimizations, etc. and be sure that it's trading the same factor as in alphalens. I've looked through the documentation, , examples, and past issues for alphalens and zipline but am still having difficulties.

There might be a timezone issue going on, but I'm not quite able to figure it out. Below are the versions of the libraries I'm using as well as some code I'm running to recreate the issue. I'm constructing a simple test factor for a 4 stock universe over 3 months and comparing the performance computed in alphalens vs zipline. I've also attached a pdf of the notebook (below is the exported markdown version of the attached notebook) Thanks!

alphalens_vs_zipline.pdf

alphalens=0.3.3 zipline=1.3.0

import zipline, alphalens, pyfolio
import pandas as pd
import numpy as np
import pandas_datareader.data as web

from zipline.api import order, record, symbol, order_target_percent, symbol
from zipline.finance.execution import MarketOrder
from zipline import run_algorithm

assets = ['FB', 'AMZN', 'NFLX', 'GOOG']
start = '2018-01-01'
end = '2018-03-30'

df_cache = web.DataReader(assets, data_source='iex', start=start, end=end)

df = df_cache.copy()
df.index = df.index.map(lambda x: pd.Timestamp(x))
# df.index = df.index.map(lambda x: pd.Timestamp(x, tz='UTC'))

pn = df.stack(0).to_panel()

/Users/tdrobbin/anaconda/envs/zipline/lib/python3.6/site-packages/ipykernel/__main__.py:5: DeprecationWarning: 
Panel is deprecated and will be removed in a future version.
The recommended way to represent these types of 3-dimensional data are with a MultiIndex on a DataFrame, via the Panel.to_frame() method
Alternatively, you can use the xarray package http://xarray.pydata.org/en/stable/.
Pandas provides a `.to_xarray()` method to help automate this conversion.

pn

<class 'pandas.core.panel.Panel'>
Dimensions: 4 (items) x 61 (major_axis) x 5 (minor_axis)
Items axis: AMZN to NFLX
Major_axis axis: 2018-01-02 00:00:00 to 2018-03-29 00:00:00
Minor_axis axis: close to volume

factor = pn.loc[:, :, 'close'].pct_change(1)

factor = factor.rank(axis=1)
factor = factor.apply(lambda x: (x - x.mean()) / x.std(), axis=1)
factor = factor.apply(lambda x: (x / (x.abs().sum() * .5)) * .5, axis=1)

factor.head()

Symbols	AMZN	FB	GOOG	NFLX
date
2018-01-02	NaN	NaN	NaN	NaN
2018-01-03	-0.375	0.125	-0.125	0.375
2018-01-04	-0.375	-0.125	0.375	0.125
2018-01-05	0.375	-0.375	0.125	-0.125
2018-01-08	0.375	-0.125	-0.375	0.125

factor = factor.stack()
factor.index = factor.index.set_names(['date', 'asset'])

pricing = pn.loc[:, :, 'close']
factor_data = alphalens.utils.get_clean_factor_and_forward_returns(factor, pricing, quantiles=2)

Dropped 16.7% entries from factor data: 16.7% in forward returns computation and 0.0% in binning phase (set max_loss=0 to see potentially suppressed Exceptions).
max_loss is 35.0%, not exceeded: OK!

factor['20180103']

asset
AMZN   -0.375
FB      0.125
GOOG   -0.125
NFLX    0.375
dtype: float64

pf_returns, pf_positions, pf_benchmark = alphalens.performance.create_pyfolio_input(factor_data, period='1D')

/Users/tdrobbin/anaconda/envs/zipline/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py:840: PerformanceWarning: Non-vectorized DateOffset being applied to Series or DatetimeIndex
  "or DatetimeIndex", PerformanceWarning)

pf_positions.head()

asset	AMZN	FB	GOOG	NFLX	cash
date
2018-01-03	-0.375	0.125	-0.125	0.375	1.0
2018-01-04	-0.375	-0.125	0.375	0.125	1.0
2018-01-05	0.375	-0.375	0.125	-0.125	1.0
2018-01-06	0.000	0.000	0.000	0.000	1.0
2018-01-07	0.000	0.000	0.000	0.000	1.0

def initialize(context):
    context.factor = factor

def handle_data(context, data):
    today = context.get_datetime()
    yesterday = today - pd.tseries.offsets.BDay()

    yesterday = pd.core.tools.datetimes.format(yesterday)

    if yesterday not in context.factor.index:
        return

    todays_target_wgts = context.factor[yesterday]

    for asset, wgt in todays_target_wgts.items():
        order_target_percent(symbol(asset), wgt, style=MarketOrder())

df_zp = run_algorithm(
    initialize=initialize, 
    handle_data=handle_data, 
    start=pd.Timestamp(start, tz='UTC'), 
#     start=pd.to_datetime(start).tz_localize('UTC'),
    end=pd.Timestamp(end, tz='UTC'),
#     end=pd.Timestamp(end),
    data=pn, 
    capital_base=10000)

/Users/tdrobbin/anaconda/envs/zipline/lib/python3.6/site-packages/empyrical/stats.py:704: RuntimeWarning: invalid value encountered in true_divide
  out=out,
/Users/tdrobbin/anaconda/envs/zipline/lib/python3.6/site-packages/empyrical/stats.py:790: RuntimeWarning: invalid value encountered in true_divide
  np.divide(average_annual_return, annualized_downside_risk, out=out)

df_zp.returns.head(10)

2018-01-02 21:00:00+00:00    0.000000
2018-01-03 21:00:00+00:00    0.000000
2018-01-04 21:00:00+00:00    0.000000
2018-01-05 21:00:00+00:00   -0.000482
2018-01-08 21:00:00+00:00   -0.000869
2018-01-09 21:00:00+00:00   -0.004257
2018-01-10 21:00:00+00:00   -0.000701
2018-01-11 21:00:00+00:00    0.003980
2018-01-12 21:00:00+00:00    0.013318
2018-01-16 21:00:00+00:00   -0.007198
Name: returns, dtype: float64

pf_returns.head(10)

date
2018-01-03    0.000000
2018-01-04    0.001725
2018-01-05   -0.001655
2018-01-06    0.000000
2018-01-07    0.000000
2018-01-08    0.001712
2018-01-09    0.004144
2018-01-10   -0.000858
2018-01-11   -0.008795
2018-01-12    0.027433
Freq: D, dtype: float64

comparison = pd.DataFrame(dict(
    zipline_computed=df_zp.returns.rename(index=pd.core.tools.datetimes.format),
    alphalens_computed=pf_returns.rename(index=pd.core.tools.datetimes.format)
)).replace(0, np.nan)

comparison.head(10)

	alphalens_computed	zipline_computed
20180102	NaN	NaN
20180103	NaN	NaN
20180104	0.001725	NaN
20180105	-0.001655	-0.000482
20180106	NaN	NaN
20180107	NaN	NaN
20180108	0.001712	-0.000869
20180109	0.004144	-0.004257
20180110	-0.000858	-0.000701
20180111	-0.008795	0.003980

# Ideally the two returns series would have correlation of 1
comparison.dropna().corr()

	alphalens_computed	zipline_computed
alphalens_computed	1.000000	-0.063873
zipline_computed	-0.063873	1.000000

[luca-s]

Hi @tdrobbin, I can give you some advice on how to make sure to recreate the same performance in zipline, but I unfortunately cannot go through your code and make sure there are no bugs, that's up to you.

First thing is to understand how returns are computed in Alphalens. If you look at the documentation of alphalens.utils.get_clean_factor_and_forward_returns you can see how the prices information is used to compute entry and exit price for securities, those prices are used to compute the actual returns.

The pricing data must contain at least an entry for each timestamp/asset combination in the factor. This entry should reflect the buy price for the assets and usually it is the next available price after the factor is computed but it can also be a later price if the factor is meant to be traded later. The pricing data is also needed for the assets sell price, the asset price after period timestamps will be considered the sell price for that asset when computing period forward returns.

In you code you use the percentage change of close price as factor value. This means your factor value is available at market close and the earliest time you can trade the factor is the next day at market open. So every day, at market open, the portfolio will be rebalanced, entering new positions and exiting old ones. This means the prices DataFrame passed to Alphalens should contain the open price of the securities: for each timestamp of factor index the price DataFrame must contain the open price of the following day.

If you want to replicate the same on zipline you need to write an algorithm that does the full portfolio rebalance only once per day, at market open. Also, during the testing phase, you need to set commission and slippage to 0. The slippage has to be set to FixedSlippage, because that's the only slippage that doesn't limit the amount of securities you can trade in a minute bar.

hope these help

[tdrobbin]

I see, that explanation was very helpful. I believe my issue was with not trading all the securities at once with the correct slippage setting. Thank you very for your help!