stefan-jansen / zipline-reloaded

Zipline, a Pythonic Algorithmic Trading Library
https://zipline.ml4trading.io
Apache License 2.0
1.02k stars 198 forks source link

ValueError: Parameter `start` received with timezone defined as 'UTC' although a Date must be timezone naive. #224

Open alex9434 opened 9 months ago

alex9434 commented 9 months ago

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

* Operating System: Linux inspiron 6.5.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.3-1 (2023-09-13) x86_64 GNU/Linux * Python Version: Python 3.10.12 * Python Bitness: 64 * How did you install Zipline: conda * Python packages: # packages in environment at /opt/conda3/envs/zipline: ``` # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge alembic 1.12.0 pyhd8ed1ab_0 conda-forge alsa-lib 1.2.10 hd590300_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge asttokens 2.4.0 pyhd8ed1ab_0 conda-forge attr 2.5.1 h166bdaf_1 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.5 pyhd8ed1ab_0 conda-forge bcolz-zipline 1.2.6 py310h1f7b6fc_1 conda-forge beautifulsoup4 4.12.2 pyha770c72_0 conda-forge blosc 1.21.5 h0f2a231_0 conda-forge bottleneck 1.3.7 py310h1f7b6fc_1 conda-forge brotli 1.1.0 hd590300_1 conda-forge brotli-bin 1.1.0 hd590300_1 conda-forge brotli-python 1.1.0 py310hc6cd4ac_1 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.19.1 hd590300_0 conda-forge c-blosc2 2.10.5 hb4ffafa_0 conda-forge ca-certificates 2023.7.22 hbcca054_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cairo 1.16.0 h0c91306_1017 conda-forge certifi 2023.7.22 pyhd8ed1ab_0 conda-forge cffi 1.16.0 py310h2fee648_0 conda-forge charset-normalizer 3.3.0 pyhd8ed1ab_0 conda-forge click 8.1.7 unix_pyh707e725_0 conda-forge comm 0.1.4 pyhd8ed1ab_0 conda-forge contourpy 1.1.1 py310hd41b1e2_1 conda-forge cryptography 41.0.4 py310h75e40e8_0 conda-forge cycler 0.12.0 pyhd8ed1ab_0 conda-forge dbus 1.13.6 h5008d03_3 conda-forge debugpy 1.8.0 py310hc6cd4ac_1 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge empyrical-reloaded 0.5.9 pyhd8ed1ab_0 conda-forge exceptiongroup 1.1.3 pyhd8ed1ab_0 conda-forge exchange-calendars 4.2.8 pyhd8ed1ab_1 conda-forge executing 1.2.0 pyhd8ed1ab_0 conda-forge expat 2.5.0 hcb278e6_1 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.2 h14ed4e7_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.43.0 py310h2372a71_0 conda-forge freetype 2.12.1 h267a509_2 conda-forge frozendict 2.3.8 py310h2372a71_1 conda-forge gettext 0.21.1 h27087fc_0 conda-forge glib 2.78.0 hfc55251_0 conda-forge glib-tools 2.78.0 hfc55251_0 conda-forge graphite2 1.3.13 h58526e2_1001 conda-forge greenlet 2.0.2 py310hc6cd4ac_1 conda-forge gst-plugins-base 1.22.6 h8e1006c_2 conda-forge gstreamer 1.22.6 h98fc4e7_2 conda-forge h5py 3.9.0 nompi_py310ha2ad45a_103 conda-forge harfbuzz 8.2.1 h3d44ed6_0 conda-forge hdf5 1.14.2 nompi_h4f84152_100 conda-forge html5lib 1.1 pyh9f0ad1d_0 conda-forge icu 73.2 h59595ed_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge importlib-metadata 6.8.0 pyha770c72_0 conda-forge importlib_metadata 6.8.0 hd8ed1ab_0 conda-forge importlib_resources 6.1.0 pyhd8ed1ab_0 conda-forge intervaltree 3.1.0 pyhd8ed1ab_1 conda-forge ipykernel 6.25.2 pyh2140261_0 conda-forge ipython 8.16.1 pyh0d859eb_0 conda-forge iso3166 2.1.1 pyhd8ed1ab_0 conda-forge iso4217 1.9.20220401 pyhd8ed1ab_0 conda-forge jedi 0.19.1 pyhd8ed1ab_0 conda-forge jupyter_client 8.3.1 pyhd8ed1ab_0 conda-forge jupyter_core 5.3.2 py310hff52083_0 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.5 py310hd41b1e2_1 conda-forge korean_lunar_calendar 0.3.1 pyhd8ed1ab_0 conda-forge krb5 1.21.2 h659d440_0 conda-forge lame 3.100 h166bdaf_1003 conda-forge lcms2 2.15 h7f713cb_2 conda-forge ld_impl_linux-64 2.40 h41732ed_0 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libaec 1.1.2 h59595ed_1 conda-forge libblas 3.9.0 18_linux64_openblas conda-forge libbrotlicommon 1.1.0 hd590300_1 conda-forge libbrotlidec 1.1.0 hd590300_1 conda-forge libbrotlienc 1.1.0 hd590300_1 conda-forge libcap 2.69 h0f662aa_0 conda-forge libcblas 3.9.0 18_linux64_openblas conda-forge libclang 15.0.7 default_h7634d5b_3 conda-forge libclang13 15.0.7 default_h9986a30_3 conda-forge libcups 2.3.3 h4637d8d_4 conda-forge libcurl 8.3.0 hca28451_0 conda-forge libdeflate 1.19 hd590300_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.12 hf998b51_1 conda-forge libexpat 2.5.0 hcb278e6_1 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libflac 1.4.3 h59595ed_0 conda-forge libgcc-ng 13.2.0 h807b86a_2 conda-forge libgcrypt 1.10.1 h166bdaf_0 conda-forge libgfortran-ng 13.2.0 h69a702a_2 conda-forge libgfortran5 13.2.0 ha4646dd_2 conda-forge libglib 2.78.0 hebfc3b9_0 conda-forge libgomp 13.2.0 h807b86a_2 conda-forge libgpg-error 1.47 h71f35ed_0 conda-forge libiconv 1.17 h166bdaf_0 conda-forge libjpeg-turbo 2.1.5.1 hd590300_1 conda-forge liblapack 3.9.0 18_linux64_openblas conda-forge libllvm15 15.0.7 h5cf9203_3 conda-forge libnghttp2 1.52.0 h61bc06f_0 conda-forge libnsl 2.0.0 hd590300_1 conda-forge libogg 1.3.4 h7f98852_1 conda-forge libopenblas 0.3.24 pthreads_h413a1c8_0 conda-forge libopus 1.3.1 h7f98852_1 conda-forge libpng 1.6.39 h753d276_0 conda-forge libpq 15.4 hfc447b1_2 conda-forge libsndfile 1.2.2 hc60ed4a_1 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libsqlite 3.43.0 h2797004_0 conda-forge libssh2 1.11.0 h0841786_0 conda-forge libstdcxx-ng 13.2.0 h7e041cc_2 conda-forge libsystemd0 254 h3516f8a_0 conda-forge libta-lib 0.4.0 hd590300_2 conda-forge libtiff 4.6.0 h29866fb_1 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libvorbis 1.3.7 h9c3ff4c_0 conda-forge libwebp-base 1.3.2 hd590300_0 conda-forge libxcb 1.15 h0b41bf4_0 conda-forge libxkbcommon 1.5.0 h5d7e998_3 conda-forge libxml2 2.11.5 h232c23b_1 conda-forge libxslt 1.1.37 h0054252_1 conda-forge libzlib 1.2.13 hd590300_5 conda-forge lru-dict 1.2.0 py310h2372a71_1 conda-forge lxml 4.9.3 py310h9b7343a_1 conda-forge lz4-c 1.9.4 hcb278e6_0 conda-forge lzo 2.10 h516909a_1000 conda-forge mako 1.2.4 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 py310h2372a71_1 conda-forge matplotlib 3.8.0 py310hff52083_1 conda-forge matplotlib-base 3.8.0 py310h62c0568_1 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mpg123 1.32.3 h59595ed_0 conda-forge multipledispatch 0.6.0 py_0 conda-forge multitasking 0.0.9 pyhd8ed1ab_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.0.33 hf1915f5_4 conda-forge mysql-libs 8.0.33 hca2cd23_4 conda-forge ncurses 6.4 hcb278e6_0 conda-forge nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge networkx 3.1 pyhd8ed1ab_0 conda-forge nomkl 1.0 h5ca1d4c_0 conda-forge nspr 4.35 h27087fc_0 conda-forge nss 3.94 h1d7d5a4_0 conda-forge numexpr 2.8.7 py310hcc13569_102 conda-forge numpy 1.26.0 py310hb13e2d6_0 conda-forge openjpeg 2.5.0 h488ebb8_3 conda-forge openssl 3.1.3 hd590300_0 conda-forge packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 2.1.1 py310hcc13569_1 conda-forge pandas-datareader 0.10.0 pyh6c4a22f_0 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge patsy 0.5.3 pyhd8ed1ab_0 conda-forge pcre2 10.40 hc3806b6_0 conda-forge peewee 3.16.3 py310hbc90443_1 conda-forge pexpect 4.8.0 pyh1a96a4e_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 10.0.1 py310h29da1c1_1 conda-forge pip 23.2.1 pyhd8ed1ab_0 conda-forge pixman 0.42.2 h59595ed_0 conda-forge platformdirs 3.11.0 pyhd8ed1ab_0 conda-forge ply 3.11 py_1 conda-forge prompt-toolkit 3.0.39 pyha770c72_0 conda-forge prompt_toolkit 3.0.39 hd8ed1ab_0 conda-forge psutil 5.9.5 py310h2372a71_1 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pulseaudio-client 16.1 hb77b528_5 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge py-cpuinfo 9.0.0 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.16.1 pyhd8ed1ab_0 conda-forge pyluach 2.2.0 pyhd8ed1ab_0 conda-forge pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge pyqt 5.15.9 py310h04931ad_5 conda-forge pyqt5-sip 12.12.2 py310hc6cd4ac_5 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge pytables 3.8.0 py310h374b01c_4 conda-forge python 3.10.12 hd12c33a_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-interface 1.6.0 py_0 conda-forge python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge python_abi 3.10 4_cp310 conda-forge pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge pyzmq 25.1.1 py310h5bbb5d0_1 conda-forge qt-main 5.15.8 hc47bfe8_16 conda-forge readline 8.2 h8228510_1 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge scipy 1.11.3 py310hb13e2d6_1 conda-forge setuptools 68.2.2 pyhd8ed1ab_0 conda-forge sip 6.7.11 py310hc6cd4ac_1 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.10 h9fff704_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.5 pyhd8ed1ab_1 conda-forge sqlalchemy 2.0.21 py310h2372a71_0 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge statsmodels 0.14.0 py310h1f7b6fc_2 conda-forge ta-lib 0.4.28 py310h1f7b6fc_0 conda-forge tk 8.6.13 h2797004_0 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge toolz 0.12.0 pyhd8ed1ab_0 conda-forge tornado 6.3.3 py310h2372a71_1 conda-forge traitlets 5.11.2 pyhd8ed1ab_0 conda-forge typing-extensions 4.8.0 hd8ed1ab_0 conda-forge typing_extensions 4.8.0 pyha770c72_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge unicodedata2 15.1.0 py310h2372a71_0 conda-forge urllib3 2.0.6 pyhd8ed1ab_0 conda-forge wcwidth 0.2.8 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 pyhd8ed1ab_2 conda-forge wheel 0.41.2 pyhd8ed1ab_0 conda-forge xcb-util 0.4.0 hd590300_1 conda-forge xcb-util-image 0.4.0 h8ee46fc_1 conda-forge xcb-util-keysyms 0.4.0 h8ee46fc_1 conda-forge xcb-util-renderutil 0.3.9 hd590300_1 conda-forge xcb-util-wm 0.4.1 h8ee46fc_1 conda-forge xkeyboard-config 2.39 hd590300_0 conda-forge xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.1.1 hd590300_0 conda-forge xorg-libsm 1.2.4 h7391055_0 conda-forge xorg-libx11 1.8.6 h8ee46fc_0 conda-forge xorg-libxau 1.0.11 hd590300_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h0b41bf4_2 conda-forge xorg-libxrender 0.9.11 hd590300_0 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge xorg-xf86vidmodeproto 2.3.1 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.6 h166bdaf_0 conda-forge yfinance 0.2.31 pyhd8ed1ab_0 conda-forge zeromq 4.3.4 h9c3ff4c_1 conda-forge zipline-reloaded 3.0.3 py310h278f3c1_0 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 hd590300_5 conda-forge zlib-ng 2.0.7 h0b41bf4_0 conda-forge zstd 1.5.5 hfc55251_0 conda-forge ```

Description of Issue

I am trying to run a simple backtest based on the book "Trading Evolved" by Andreas Clenow. It is a slighty modified version of the BuyApple program from the tutorial:

# This ensures that our graphs will be shown properly in the notebook.
%matplotlib inline

# Import Zipline functions that we need
from zipline import run_algorithm
from zipline.api import order_target_percent, symbol

# Import date and time zone libraries
from datetime import datetime
import pytz

import pandas as pd

# Import visualization
import matplotlib.pyplot as plt

def initialize(context):
    # Which stock to trade
    context.stock = symbol('AAPL')

    # Moving average window
    context.index_average_window = 100

def handle_data(context, data):
    # Request history for the stock
    equities_hist = data.history(context.stock, 'close',
                                 context.index_average_window, '1d')

    # Check if price is above moving average
    if equities_hist[-1] > equities_hist.mean():
        stock_weight = 1.0
    else:
        stock_weight = 0.0

    # Place order
    order_target_percent(context.stock, stock_weight)

def analyze(context, perf):
    fig = plt.figure(figsize=(12, 8))

    # First chart
    ax = fig.add_subplot(311)
    ax.set_title('Strategy Results')
    ax.semilogy(perf['portfolio_value'], linestyle='-',
                label='Equity Curve', linewidth=3.0)
    ax.legend()
    ax.grid(False)

    # Second chart
    ax = fig.add_subplot(312)
    ax.plot(perf['gross_leverage'],
            label='Exposure', linestyle='-', linewidth=1.0)
    ax.legend()
    ax.grid(True)

    # Third chart
    ax = fig.add_subplot(313)
    ax.plot(perf['returns'], label='Returns', linestyle='-.', linewidth=1.0)
    ax.legend()
    ax.grid(True)

# Set start and end date
start_date = datetime(1996, 1, 1, tzinfo=pytz.UTC)
end_date = datetime(2018, 12, 31, tzinfo=pytz.UTC) 

# Fire off the backtest
results = run_algorithm(
    start=start_date,
    end=end_date,
    initialize=initialize,
    analyze=analyze,
    handle_data=handle_data,
    capital_base=10000,
    data_frequency = 'daily', bundle='quandl'
) 

File /opt/conda3/envs/zipline/lib/python3.10/site-packages/zipline/utils/run_algo.py:397, in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, bundle, bundle_timestamp, trading_calendar, metrics_set, benchmark_returns, default_extension, extensions, strict_extensions, environ, custom_loader, blotter) 393 load_extensions(default_extension, extensions, strict_extensions, environ) 395 benchmark_spec = BenchmarkSpec.from_returns(benchmark_returns) --> 397 return _run( 398 handle_data=handle_data, 399 initialize=initialize, 400 before_trading_start=before_trading_start, 401 analyze=analyze, 402 algofile=None, 403 algotext=None, 404 defines=(), 405 data_frequency=data_frequency, 406 capital_base=capital_base, 407 bundle=bundle, 408 bundle_timestamp=bundle_timestamp, 409 start=start, 410 end=end, 411 output=os.devnull, 412 trading_calendar=trading_calendar, 413 print_algo=False, 414 metrics_set=metrics_set, 415 local_namespace=False, 416 environ=environ, 417 blotter=blotter, 418 custom_loader=custom_loader, 419 benchmark_spec=benchmark_spec, 420 )

File /opt/conda3/envs/zipline/lib/python3.10/site-packages/zipline/utils/run_algo.py:103, in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter, custom_loader, benchmark_spec) 100 trading_calendar = get_calendar("XNYS") 102 # date parameter validation --> 103 if trading_calendar.sessions_distance(start, end) < 1: 104 raise _RunAlgoError( 105 "There are no trading days between %s and %s" 106 % ( (...) 109 ), 110 ) 112 benchmark_sid, benchmark_returns = benchmark_spec.resolve( 113 asset_finder=bundle_data.asset_finder, 114 start_date=start, 115 end_date=end, 116 )

File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/exchange_calendar.py:2275, in ExchangeCalendar.sessions_distance(self, start, end, _parse) 2258 def sessions_distance(self, start: Date, end: Date, _parse: bool = True) -> int: 2259 """Return the number of sessions in a range. 2260 2261 Parameters (...) 2273 end then return will be negated. 2274 """ -> 2275 start, end = self._parse_start_end_dates(start, end, _parse) 2276 negate = end < start 2277 if negate:

File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/exchange_calendar.py:2172, in ExchangeCalendar._parse_start_end_dates(self, start, end, _parse) 2170 if not _parse: 2171 return start, end -> 2172 return parse_date(start, "start", self), parse_date(end, "end", self)

File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/calendar_helpers.py:378, in parse_date(date, param_name, calendar, raise_oob) 375 ts = parse_timestamp(date, param_name, raise_oob=False, side="left", utc=False) 377 if ts.tz is not None: --> 378 raise ValueError( 379 f"Parameter {param_name} received with timezone defined as '{ts.tz.zone}'" 380 f" although a Date must be timezone naive." 381 ) 383 if not ts == ts.normalize(): 384 raise ValueError( 385 f"Parameter {param_name} parsed as '{ts}' although a Date must have" 386 f" a time component of 00:00." 387 )

ValueError: Parameter start received with timezone defined as 'UTC' although a Date must be timezone naive.



Here is how you can reproduce this issue on your machine:

## Reproduction Steps

1. Ingest quandl bundle
2. Run above program in Jupyter Lab or Notebook/the "BuyApple" from the tutorial has the same error message
3. Error message
...

## What steps have you taken to resolve this already?

Tried the solution outlined in 
https://stackoverflow.com/questions/61513097/how-to-solve-assertionerror-when-running-backtest-on-zipline
without success

# Anything else?
Could you kindly clarify the best way to provide start and end times for backtests? Given that I want to backtest with European and US markets, time zones are relevant to me.
...

Sincerely,
Alex
phelps-sg commented 9 months ago

The start and end dates use the time-zone of the trading_calendar that you pass in to run_algorithm(). In your example you don't specify the trading_calendar parameter so it defaults to the XNYS calendar which is New York time, but you can pass in a different trading_calendar e.g. XMAD. To use 24/7 which is UTC:

from zipline.utils.calendar_utils import get_calendar

run_algorithm(
...
    trading_calendar=get_calendar("24/7")
)
alex9434 commented 9 months ago

@phelps-sg Thank you so much for your fast reply. Really appreciated! Unfortunately I still get the same error message when I use the "24/7" calendar:

ValueError: Parameterstartreceived with timezone defined as 'UTC' although a Date must be timezone naive.

What am I missing? Below the complete program including your suggested changes. Did I misunderstood your suggestions? Alternatively, I am happy to use the XNYS calendar. How would I then have to modify the settings for start_date and end_date to use the XNYS calendar?

# This ensures that our graphs will be shown properly in the notebook.
%matplotlib inline

# Import Zipline functions that we need
from zipline import run_algorithm
from zipline.api import order_target_percent, symbol
from zipline.utils.calendar_utils import get_calendar

# Import date and time zone libraries
from datetime import datetime
import pytz

import pandas as pd

# Import visualization
import matplotlib.pyplot as plt

def initialize(context):
    # Which stock to trade
    context.stock = symbol('AAPL')

    # Moving average window
    context.index_average_window = 100

def handle_data(context, data):
    # Request history for the stock
    equities_hist = data.history(context.stock, 'close',
                                 context.index_average_window, '1d')

    # Check if price is above moving average
    if equities_hist[-1] > equities_hist.mean():
        stock_weight = 1.0
    else:
        stock_weight = 0.0

    # Place order
    order_target_percent(context.stock, stock_weight)

def analyze(context, perf):
    fig = plt.figure(figsize=(12, 8))

    # First chart
    ax = fig.add_subplot(311)
    ax.set_title('Strategy Results')
    ax.semilogy(perf['portfolio_value'], linestyle='-',
                label='Equity Curve', linewidth=3.0)
    ax.legend()
    ax.grid(False)

    # Second chart
    ax = fig.add_subplot(312)
    ax.plot(perf['gross_leverage'],
            label='Exposure', linestyle='-', linewidth=1.0)
    ax.legend()
    ax.grid(True)

    # Third chart
    ax = fig.add_subplot(313)
    ax.plot(perf['returns'], label='Returns', linestyle='-.', linewidth=1.0)
    ax.legend()
    ax.grid(True)

# Set start and end date
start_date = datetime(1996, 1, 1, tzinfo=pytz.UTC)
end_date = datetime(2018, 12, 31, tzinfo=pytz.UTC) 

# Fire off the backtest
results = run_algorithm(
    trading_calendar=get_calendar("24/7"),
    start=start_date,
    end=end_date,
    initialize=initialize,
    analyze=analyze,
    handle_data=handle_data,
    capital_base=10000,
    data_frequency = 'daily', bundle='quandl'
)
phelps-sg commented 9 months ago

Zipline is complaining because you are trying to override the time-zone of the trading_calendar. When you create start_date and end_date you need to ensure they do not have any time-zone info, e.g.:

start_date = pd.Timestamp("1996-01-01")
end_date = pd.Timestamp("2018-12-31")

These dates now have an implicit time-zone of UTC because the trading_calendar is "24/7".

alex9434 commented 9 months ago

@phelps-sg I did some more experiement. For trading_calendar=get_calendar("XNYS") I can specify the times as follow:

start_date = pd.Timestamp('2004-1-1')
end_date = pd.Timestamp('2017-12-31')

How would I specify start_date and end_date for trading_calendar=get_calendar("24/7")

Thank you so much for your help and pointing me in the right direction!

phelps-sg commented 9 months ago

How would I specify start_date and end_date for trading_calendar=get_calendar("24/7")

What happens when you use the 24/7 calendar with the non-tz start and end dates- are you getting a different error?

alex9434 commented 9 months ago

I get the following error: NotSessionError: Parametersessiontakes a session although received input that parsed to '1990-01-02 00:00:00' which is earlier than the first session of calendar '24/7' ('2003-10-10 00:00:00').

I have used the following time settings:

# Set start and end date
start_date = pd.Timestamp('2004-1-1')
end_date = pd.Timestamp('2017-12-31')

It seems like the dates get interpreted wrongly by the calendar.

phelps-sg commented 9 months ago

I think it is complaining that some of the data in the bundle is outside of the range of the calendar. So there are three date ranges involved here:

  1. the date range of the back-test
  2. the date range of the data-set in the bundle
  3. date range that the trading calendar is defined over

I believe zipline is complaining that the range of 2 is not inside of the range of 3. The calendar logic is provided by this library:

https://github.com/gerrymanoim/exchange_calendars/blob/master/exchange_calendars/exchange_calendar.py

If you look at the code you will notice that the default earliest trading day is 20 years before the current date, which is where the 2003-10-10 date is coming from.

To fix this I would suggest overriding the start date when creating the calendar, something like:

get_calendar("24/7", start=pd.Timestamp("1990-01-01"))

If this doesn't work and generates another error, please include the full-stack trace so we can find the relevant code that generates the error.

alex9434 commented 9 months ago

Unfortunately I get the same error message even if I use a much shorter time period.

...
# Set start and end date
start_date = pd.Timestamp('2005-1-1')
end_date = pd.Timestamp('2015-12-31')

# Fire off the backtest
results = run_algorithm(
    trading_calendar=get_calendar("24/7"),
...

gives the error message: NotSessionError: Parametersessiontakes a session although received input that parsed to '1990-01-02 00:00:00' which is earlier than the first session of calendar '24/7' ('2003-10-12 00:00:00').

If I add your additional suggestion

..
    trading_calendar=get_calendar("24/7", start=pd.Timestamp("2005-01-01")),
...

I still get NotSessionError: Parametersessiontakes a session although received input that parsed to '1990-01-02 00:00:00' which is earlier than the first session of calendar '24/7' ('2003-10-12 00:00:00').

Here is the full message:

---------------------------------------------------------------------------
NotSessionError                           Traceback (most recent call last)
Cell In[6], line 68
     65 end_date = pd.Timestamp('2015-12-31')
     67 # Fire off the backtest
---> 68 results = run_algorithm(
     69     trading_calendar=get_calendar("24/7",start=pd.Timestamp("2005-01-01")),
     70     start=start_date,
     71     end=end_date,
     72     initialize=initialize,
     73     analyze=analyze,
     74     handle_data=handle_data,
     75     capital_base=10000,
     76     data_frequency = 'daily', bundle='quandl'
     77 )

File /opt/conda3/envs/zipline/lib/python3.10/site-packages/zipline/utils/run_algo.py:397, in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, bundle, bundle_timestamp, trading_calendar, metrics_set, benchmark_returns, default_extension, extensions, strict_extensions, environ, custom_loader, blotter)
    393 load_extensions(default_extension, extensions, strict_extensions, environ)
    395 benchmark_spec = BenchmarkSpec.from_returns(benchmark_returns)
--> 397 return _run(
    398     handle_data=handle_data,
    399     initialize=initialize,
    400     before_trading_start=before_trading_start,
    401     analyze=analyze,
    402     algofile=None,
    403     algotext=None,
    404     defines=(),
    405     data_frequency=data_frequency,
    406     capital_base=capital_base,
    407     bundle=bundle,
    408     bundle_timestamp=bundle_timestamp,
    409     start=start,
    410     end=end,
    411     output=os.devnull,
    412     trading_calendar=trading_calendar,
    413     print_algo=False,
    414     metrics_set=metrics_set,
    415     local_namespace=False,
    416     environ=environ,
    417     blotter=blotter,
    418     custom_loader=custom_loader,
    419     benchmark_spec=benchmark_spec,
    420 )

File /opt/conda3/envs/zipline/lib/python3.10/site-packages/zipline/utils/run_algo.py:163, in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter, custom_loader, benchmark_spec)
    159         click.echo(algotext)
    161 first_trading_day = bundle_data.equity_minute_bar_reader.first_trading_day
--> 163 data = DataPortal(
    164     bundle_data.asset_finder,
    165     trading_calendar=trading_calendar,
    166     first_trading_day=first_trading_day,
    167     equity_minute_reader=bundle_data.equity_minute_bar_reader,
    168     equity_daily_reader=bundle_data.equity_daily_bar_reader,
    169     adjustment_reader=bundle_data.adjustment_reader,
    170     future_minute_reader=bundle_data.equity_minute_bar_reader,
    171     future_daily_reader=bundle_data.equity_daily_bar_reader,
    172 )
    174 pipeline_loader = USEquityPricingLoader.without_fx(
    175     bundle_data.equity_daily_bar_reader,
    176     bundle_data.adjustment_reader,
    177 )
    179 def choose_loader(column):

File /opt/conda3/envs/zipline/lib/python3.10/site-packages/zipline/data/data_portal.py:274, in DataPortal.__init__(self, asset_finder, trading_calendar, first_trading_day, equity_daily_reader, equity_minute_reader, future_daily_reader, future_minute_reader, adjustment_reader, last_available_session, last_available_minute, minute_history_prefetch_length, daily_history_prefetch_length)
    270 self._first_trading_day = first_trading_day
    272 # Get the first trading minute
    273 self._first_trading_minute = (
--> 274     self.trading_calendar.session_first_minute(self._first_trading_day)
    275     if self._first_trading_day is not None
    276     else (None, None)
    277 )
    279 # Store the locs of the first day and first minute
    280 self._first_trading_day_loc = (
    281     self.trading_calendar.sessions.get_loc(self._first_trading_day)
    282     if self._first_trading_day is not None
    283     else None
    284 )

File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/exchange_calendar.py:1065, in ExchangeCalendar.session_first_minute(self, session, _parse)
   1063 """Return first trading minute of a given session."""
   1064 nanos = self.first_minutes_nanos
-> 1065 return self._get_session_minute_from_nanos(session, nanos, _parse)

File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/exchange_calendar.py:1057, in ExchangeCalendar._get_session_minute_from_nanos(self, session, nanos, _parse)
   1054 def _get_session_minute_from_nanos(
   1055     self, session: Session, nanos: np.ndarray, _parse: bool
   1056 ) -> pd.Timestamp:
-> 1057     idx = self._get_session_idx(session, _parse=_parse)
   1058     return pd.Timestamp(nanos[idx], tz=UTC)

File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/exchange_calendar.py:973, in ExchangeCalendar._get_session_idx(self, session, _parse)
    971 def _get_session_idx(self, session: Date, _parse=True) -> int:
    972     """Index position of a session."""
--> 973     session_ = parse_session(self, session) if _parse else session
    974     if TYPE_CHECKING:
    975         assert isinstance(session_, pd.Timestamp)

File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/calendar_helpers.py:433, in parse_session(calendar, session, param_name)
    431 ts = parse_date(session, param_name, raise_oob=False)
    432 if calendar._date_oob(ts) or not calendar.is_session(ts, _parse=False):
--> 433     raise errors.NotSessionError(calendar, ts, param_name)
    434 return ts

NotSessionError: Parameter `session` takes a session although received input that parsed to '1990-01-02 00:00:00' which is earlier than the first session of calendar '24/7' ('2003-10-12 00:00:00').
phelps-sg commented 9 months ago

As discussed, there are three different periods: 1. The period of the back-test, 2. the period over which the data is defined, 3. the period over which the calendar is defined. If you change the period of the back-test that will not change the period over which the data defined. I believe the problem occurs because the period over which the data is defined is not within the period over which the calendar is defined.

You can see how the calendar is constructed by looking at src.zipline.utils.calendar_utils.get_calendar. The function is defined as follows:


@wrap_with_signature(inspect.signature(ec_get_calendar))
def get_calendar(*args, **kwargs):
    if args[0] in ["us_futures", "CMES", "XNYS", "NYSE"]:
        return ec_get_calendar(*args, side="right", start=pd.Timestamp("1990-01-01"))
    return ec_get_calendar(*args, side="right")

So zipline provides its own wrapper around the exchange calendars library, but unfortunately it looks like it doesn't pass in kwargs, so the start parameter is being ignored.

If you call exchange calendars directly you should be able to construct a calendar with a much earlier start date. It is important to make this earlier than the start of your trading session. If my hypothesis is correct then you need to make the start date earlier than the earliest data point in the ingested data.

Therefore try the following:

from exchange_calendars import get_calendar as ec_get_calendar

cal = ec_get_calendar("24/7", start=pd.Timestamp("1990-01-01"))
print(cal.schedule)

If the calendar is constructed correctly you should see that the schedule starts from the specified date, e.g.:

                                open break_start break_end                     close
1990-01-01 1990-01-01 00:00:00+00:00         NaT       NaT 1990-01-02 00:00:00+00:00
1990-01-02 1990-01-02 00:00:00+00:00         NaT       NaT 1990-01-03 00:00:00+00:00
1990-01-03 1990-01-03 00:00:00+00:00         NaT       NaT 1990-01-04 00:00:00+00:00
1990-01-04 1990-01-04 00:00:00+00:00         NaT       NaT 1990-01-05 00:00:00+00:00
1990-01-05 1990-01-05 00:00:00+00:00         NaT       NaT 1990-01-06 00:00:00+00:00
...                              ...         ...       ...                       ...
2024-10-09 2024-10-09 00:00:00+00:00         NaT       NaT 2024-10-10 00:00:00+00:00
2024-10-10 2024-10-10 00:00:00+00:00         NaT       NaT 2024-10-11 00:00:00+00:00
2024-10-11 2024-10-11 00:00:00+00:00         NaT       NaT 2024-10-12 00:00:00+00:00
2024-10-12 2024-10-12 00:00:00+00:00         NaT       NaT 2024-10-13 00:00:00+00:00
2024-10-13 2024-10-13 00:00:00+00:00         NaT       NaT 2024-10-14 00:00:00+00:00

If this works then in your backtest do something like:

from exchange_calendars import get_calendar as ec_get_calendar

cal = ec_get_calendar("24/7", start=pd.Timestamp("1990-01-01"))

run_algorithm(
...
  trading_calendar=cal
)

Do not change the start date in the code above to match the date of your back-test- it is important that the start date above is earlier than any date that occurs in the ingested data.

alex9434 commented 8 months ago

Apologies for the late reply, I was traveling. Unfortunately it still is not working and it is getting more mysterious to me.

Starting with the initial version:

from zipline.utils.calendar_utils import get_calendar

# Set start and end date
start_date = pd.Timestamp('2005-1-1')
end_date = pd.Timestamp('2015-12-31')

# Fire off the backtest
results = run_algorithm(
    trading_calendar=get_calendar("XNYS"),
    ...
)

I get a correct looking equity curve: XNYS

BTW: I get the same curve for the calendar "NYSE". What is the specific difference between these two calendars?

from exchange_calendars import get_calendar as ec_get_calendar

cal = ec_get_calendar("24/7", start=pd.Timestamp("1990-01-01"))

run_algorithm(
...
  trading_calendar=cal
)

I get the following result: 24-7

I also cannot mix and match the two versions. if I try:

from exchange_calendars import get_calendar as ec_get_calendar

cal = ec_get_calendar("XNYS", start=pd.Timestamp("1990-01-01"))

run_algorithm(
...
  trading_calendar=cal
)

I get the following error: AssertionError: All readers must share target trading_calendar. Reader=<zipline.data.bcolz_minute_bars.BcolzMinuteBarReader object at 0x7f36f89316f0> for type=<class 'zipline.assets._assets.Equity'> uses calendar=<exchange_calendars.exchange_calendar_xnys.XNYSExchangeCalendar object at 0x7f36f8762110> which does not match the desired shared calendar=<exchange_calendars.exchange_calendar_xnys.XNYSExchangeCalendar object at 0x7f36f82680d0>

Printing the calendar 24/7 gives the following result:

                                open break_start break_end  \
1990-01-01 1990-01-01 00:00:00+00:00         NaT       NaT   
1990-01-02 1990-01-02 00:00:00+00:00         NaT       NaT   
1990-01-03 1990-01-03 00:00:00+00:00         NaT       NaT   
1990-01-04 1990-01-04 00:00:00+00:00         NaT       NaT   
1990-01-05 1990-01-05 00:00:00+00:00         NaT       NaT   
...                              ...         ...       ...   
2024-10-14 2024-10-14 00:00:00+00:00         NaT       NaT   
2024-10-15 2024-10-15 00:00:00+00:00         NaT       NaT   
2024-10-16 2024-10-16 00:00:00+00:00         NaT       NaT   
2024-10-17 2024-10-17 00:00:00+00:00         NaT       NaT   
2024-10-18 2024-10-18 00:00:00+00:00         NaT       NaT   

                               close  
1990-01-01 1990-01-02 00:00:00+00:00  
1990-01-02 1990-01-03 00:00:00+00:00  
1990-01-03 1990-01-04 00:00:00+00:00  
1990-01-04 1990-01-05 00:00:00+00:00  
1990-01-05 1990-01-06 00:00:00+00:00  
...                              ...  
2024-10-14 2024-10-15 00:00:00+00:00  
2024-10-15 2024-10-16 00:00:00+00:00  
2024-10-16 2024-10-17 00:00:00+00:00  
2024-10-17 2024-10-18 00:00:00+00:00  
2024-10-18 2024-10-19 00:00:00+00:00  

I am happy that at least one version is working now but I am not sure I understand why and what is wrong with the other versions.

phelps-sg commented 8 months ago

BTW: I get the same curve for the calendar "NYSE". What is the specific difference between these two calendars?

There is no difference. NYSE is just an alias for XNYS. You can see this in the code for the exchange_calendars library at line 125 here: https://github.com/gerrymanoim/exchange_calendars/blob/master/exchange_calendars/calendar_utils.py

phelps-sg commented 8 months ago

nderstand why and what is wrong with the other versions.

I suspect that the issue is that at daily frequency zipline is trying to trade at 0:00 UTC each day (as specified by the trading calendar) but when it looks at the calendar for the asset it is seeing the exchange for the asset is closed at that time.

If your strategy is trading across different exchanges in different time-zones, then you may need to configure it to trade at minute frequency, and then code the strategy so that it only place trades when each exchange opens. I haven't done this myself though, so this is entirely speculative, and you would need to do further testing and experimentation to see whether this is the case.

Alternatively, if your strategy is always trading within the same time-zone and holidays, but you need to take into account that your local time is different from the exchange time, then you can simply manually adjust the start and end dates by +/- 1 day as required and then use a trading calendar with the same zone as the exchange.

neillbyrne commented 7 months ago

I get the following error: AssertionError: All readers must share target trading_calendar. Reader=<zipline.data.bcolz_minute_bars.BcolzMinuteBarReader object at 0x7f36f89316f0> for type=<class 'zipline.assets._assets.Equity'> uses calendar=<exchange_calendars.exchange_calendar_xnys.XNYSExchangeCalendar object at 0x7f36f8762110> which does not match the desired shared calendar=<exchange_calendars.exchange_calendar_xnys.XNYSExchangeCalendar object at 0x7f36f82680d0>

This may or may not help but I think the AssertionError is a known issue where the calendar used to create the bundle is not the exact same object as the calendar you create when simulating. See section entitled 'Patch to allow calendars other than US calendars for backtesting' from https://pypi.org/project/zipline-norgatedata/#patch-to-allow-calendars-other-than-us-calendars-for-backtesting for a fix