Open alex9434 opened 9 months ago
The start and end dates use the time-zone of the trading_calendar
that you pass in to run_algorithm()
. In your example you don't specify the trading_calendar
parameter so it defaults to the XNYS
calendar which is New York time, but you can pass in a different trading_calendar
e.g. XMAD
. To use 24/7
which is UTC:
from zipline.utils.calendar_utils import get_calendar
run_algorithm(
...
trading_calendar=get_calendar("24/7")
)
@phelps-sg Thank you so much for your fast reply. Really appreciated! Unfortunately I still get the same error message when I use the "24/7" calendar:
ValueError: Parameter
startreceived with timezone defined as 'UTC' although a Date must be timezone naive.
What am I missing? Below the complete program including your suggested changes. Did I misunderstood your suggestions? Alternatively, I am happy to use the XNYS calendar. How would I then have to modify the settings for start_date and end_date to use the XNYS calendar?
# This ensures that our graphs will be shown properly in the notebook.
%matplotlib inline
# Import Zipline functions that we need
from zipline import run_algorithm
from zipline.api import order_target_percent, symbol
from zipline.utils.calendar_utils import get_calendar
# Import date and time zone libraries
from datetime import datetime
import pytz
import pandas as pd
# Import visualization
import matplotlib.pyplot as plt
def initialize(context):
# Which stock to trade
context.stock = symbol('AAPL')
# Moving average window
context.index_average_window = 100
def handle_data(context, data):
# Request history for the stock
equities_hist = data.history(context.stock, 'close',
context.index_average_window, '1d')
# Check if price is above moving average
if equities_hist[-1] > equities_hist.mean():
stock_weight = 1.0
else:
stock_weight = 0.0
# Place order
order_target_percent(context.stock, stock_weight)
def analyze(context, perf):
fig = plt.figure(figsize=(12, 8))
# First chart
ax = fig.add_subplot(311)
ax.set_title('Strategy Results')
ax.semilogy(perf['portfolio_value'], linestyle='-',
label='Equity Curve', linewidth=3.0)
ax.legend()
ax.grid(False)
# Second chart
ax = fig.add_subplot(312)
ax.plot(perf['gross_leverage'],
label='Exposure', linestyle='-', linewidth=1.0)
ax.legend()
ax.grid(True)
# Third chart
ax = fig.add_subplot(313)
ax.plot(perf['returns'], label='Returns', linestyle='-.', linewidth=1.0)
ax.legend()
ax.grid(True)
# Set start and end date
start_date = datetime(1996, 1, 1, tzinfo=pytz.UTC)
end_date = datetime(2018, 12, 31, tzinfo=pytz.UTC)
# Fire off the backtest
results = run_algorithm(
trading_calendar=get_calendar("24/7"),
start=start_date,
end=end_date,
initialize=initialize,
analyze=analyze,
handle_data=handle_data,
capital_base=10000,
data_frequency = 'daily', bundle='quandl'
)
Zipline is complaining because you are trying to override the time-zone of the trading_calendar. When you create start_date
and end_date
you need to ensure they do not have any time-zone info, e.g.:
start_date = pd.Timestamp("1996-01-01")
end_date = pd.Timestamp("2018-12-31")
These dates now have an implicit time-zone of UTC because the trading_calendar is "24/7".
@phelps-sg I did some more experiement. For
trading_calendar=get_calendar("XNYS")
I can specify the times as follow:
start_date = pd.Timestamp('2004-1-1')
end_date = pd.Timestamp('2017-12-31')
How would I specify start_date and end_date for
trading_calendar=get_calendar("24/7")
Thank you so much for your help and pointing me in the right direction!
How would I specify start_date and end_date for trading_calendar=get_calendar("24/7")
What happens when you use the 24/7 calendar with the non-tz start and end dates- are you getting a different error?
I get the following error:
NotSessionError: Parameter
sessiontakes a session although received input that parsed to '1990-01-02 00:00:00' which is earlier than the first session of calendar '24/7' ('2003-10-10 00:00:00').
I have used the following time settings:
# Set start and end date
start_date = pd.Timestamp('2004-1-1')
end_date = pd.Timestamp('2017-12-31')
It seems like the dates get interpreted wrongly by the calendar.
I think it is complaining that some of the data in the bundle is outside of the range of the calendar. So there are three date ranges involved here:
I believe zipline is complaining that the range of 2 is not inside of the range of 3. The calendar logic is provided by this library:
If you look at the code you will notice that the default earliest trading day is 20 years before the current date, which is where the 2003-10-10 date is coming from.
To fix this I would suggest overriding the start date when creating the calendar, something like:
get_calendar("24/7", start=pd.Timestamp("1990-01-01"))
If this doesn't work and generates another error, please include the full-stack trace so we can find the relevant code that generates the error.
Unfortunately I get the same error message even if I use a much shorter time period.
...
# Set start and end date
start_date = pd.Timestamp('2005-1-1')
end_date = pd.Timestamp('2015-12-31')
# Fire off the backtest
results = run_algorithm(
trading_calendar=get_calendar("24/7"),
...
gives the error message:
NotSessionError: Parameter
sessiontakes a session although received input that parsed to '1990-01-02 00:00:00' which is earlier than the first session of calendar '24/7' ('2003-10-12 00:00:00').
If I add your additional suggestion
..
trading_calendar=get_calendar("24/7", start=pd.Timestamp("2005-01-01")),
...
I still get
NotSessionError: Parameter
sessiontakes a session although received input that parsed to '1990-01-02 00:00:00' which is earlier than the first session of calendar '24/7' ('2003-10-12 00:00:00').
Here is the full message:
---------------------------------------------------------------------------
NotSessionError Traceback (most recent call last)
Cell In[6], line 68
65 end_date = pd.Timestamp('2015-12-31')
67 # Fire off the backtest
---> 68 results = run_algorithm(
69 trading_calendar=get_calendar("24/7",start=pd.Timestamp("2005-01-01")),
70 start=start_date,
71 end=end_date,
72 initialize=initialize,
73 analyze=analyze,
74 handle_data=handle_data,
75 capital_base=10000,
76 data_frequency = 'daily', bundle='quandl'
77 )
File /opt/conda3/envs/zipline/lib/python3.10/site-packages/zipline/utils/run_algo.py:397, in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, bundle, bundle_timestamp, trading_calendar, metrics_set, benchmark_returns, default_extension, extensions, strict_extensions, environ, custom_loader, blotter)
393 load_extensions(default_extension, extensions, strict_extensions, environ)
395 benchmark_spec = BenchmarkSpec.from_returns(benchmark_returns)
--> 397 return _run(
398 handle_data=handle_data,
399 initialize=initialize,
400 before_trading_start=before_trading_start,
401 analyze=analyze,
402 algofile=None,
403 algotext=None,
404 defines=(),
405 data_frequency=data_frequency,
406 capital_base=capital_base,
407 bundle=bundle,
408 bundle_timestamp=bundle_timestamp,
409 start=start,
410 end=end,
411 output=os.devnull,
412 trading_calendar=trading_calendar,
413 print_algo=False,
414 metrics_set=metrics_set,
415 local_namespace=False,
416 environ=environ,
417 blotter=blotter,
418 custom_loader=custom_loader,
419 benchmark_spec=benchmark_spec,
420 )
File /opt/conda3/envs/zipline/lib/python3.10/site-packages/zipline/utils/run_algo.py:163, in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter, custom_loader, benchmark_spec)
159 click.echo(algotext)
161 first_trading_day = bundle_data.equity_minute_bar_reader.first_trading_day
--> 163 data = DataPortal(
164 bundle_data.asset_finder,
165 trading_calendar=trading_calendar,
166 first_trading_day=first_trading_day,
167 equity_minute_reader=bundle_data.equity_minute_bar_reader,
168 equity_daily_reader=bundle_data.equity_daily_bar_reader,
169 adjustment_reader=bundle_data.adjustment_reader,
170 future_minute_reader=bundle_data.equity_minute_bar_reader,
171 future_daily_reader=bundle_data.equity_daily_bar_reader,
172 )
174 pipeline_loader = USEquityPricingLoader.without_fx(
175 bundle_data.equity_daily_bar_reader,
176 bundle_data.adjustment_reader,
177 )
179 def choose_loader(column):
File /opt/conda3/envs/zipline/lib/python3.10/site-packages/zipline/data/data_portal.py:274, in DataPortal.__init__(self, asset_finder, trading_calendar, first_trading_day, equity_daily_reader, equity_minute_reader, future_daily_reader, future_minute_reader, adjustment_reader, last_available_session, last_available_minute, minute_history_prefetch_length, daily_history_prefetch_length)
270 self._first_trading_day = first_trading_day
272 # Get the first trading minute
273 self._first_trading_minute = (
--> 274 self.trading_calendar.session_first_minute(self._first_trading_day)
275 if self._first_trading_day is not None
276 else (None, None)
277 )
279 # Store the locs of the first day and first minute
280 self._first_trading_day_loc = (
281 self.trading_calendar.sessions.get_loc(self._first_trading_day)
282 if self._first_trading_day is not None
283 else None
284 )
File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/exchange_calendar.py:1065, in ExchangeCalendar.session_first_minute(self, session, _parse)
1063 """Return first trading minute of a given session."""
1064 nanos = self.first_minutes_nanos
-> 1065 return self._get_session_minute_from_nanos(session, nanos, _parse)
File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/exchange_calendar.py:1057, in ExchangeCalendar._get_session_minute_from_nanos(self, session, nanos, _parse)
1054 def _get_session_minute_from_nanos(
1055 self, session: Session, nanos: np.ndarray, _parse: bool
1056 ) -> pd.Timestamp:
-> 1057 idx = self._get_session_idx(session, _parse=_parse)
1058 return pd.Timestamp(nanos[idx], tz=UTC)
File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/exchange_calendar.py:973, in ExchangeCalendar._get_session_idx(self, session, _parse)
971 def _get_session_idx(self, session: Date, _parse=True) -> int:
972 """Index position of a session."""
--> 973 session_ = parse_session(self, session) if _parse else session
974 if TYPE_CHECKING:
975 assert isinstance(session_, pd.Timestamp)
File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/calendar_helpers.py:433, in parse_session(calendar, session, param_name)
431 ts = parse_date(session, param_name, raise_oob=False)
432 if calendar._date_oob(ts) or not calendar.is_session(ts, _parse=False):
--> 433 raise errors.NotSessionError(calendar, ts, param_name)
434 return ts
NotSessionError: Parameter `session` takes a session although received input that parsed to '1990-01-02 00:00:00' which is earlier than the first session of calendar '24/7' ('2003-10-12 00:00:00').
As discussed, there are three different periods: 1. The period of the back-test, 2. the period over which the data is defined, 3. the period over which the calendar is defined. If you change the period of the back-test that will not change the period over which the data defined. I believe the problem occurs because the period over which the data is defined is not within the period over which the calendar is defined.
You can see how the calendar is constructed by looking at src.zipline.utils.calendar_utils.get_calendar
. The function is defined as follows:
@wrap_with_signature(inspect.signature(ec_get_calendar))
def get_calendar(*args, **kwargs):
if args[0] in ["us_futures", "CMES", "XNYS", "NYSE"]:
return ec_get_calendar(*args, side="right", start=pd.Timestamp("1990-01-01"))
return ec_get_calendar(*args, side="right")
So zipline provides its own wrapper around the exchange calendars library, but unfortunately it looks like it doesn't pass in kwargs, so the start
parameter is being ignored.
If you call exchange calendars directly you should be able to construct a calendar with a much earlier start date. It is important to make this earlier than the start of your trading session. If my hypothesis is correct then you need to make the start date earlier than the earliest data point in the ingested data.
Therefore try the following:
from exchange_calendars import get_calendar as ec_get_calendar
cal = ec_get_calendar("24/7", start=pd.Timestamp("1990-01-01"))
print(cal.schedule)
If the calendar is constructed correctly you should see that the schedule starts from the specified date, e.g.:
open break_start break_end close
1990-01-01 1990-01-01 00:00:00+00:00 NaT NaT 1990-01-02 00:00:00+00:00
1990-01-02 1990-01-02 00:00:00+00:00 NaT NaT 1990-01-03 00:00:00+00:00
1990-01-03 1990-01-03 00:00:00+00:00 NaT NaT 1990-01-04 00:00:00+00:00
1990-01-04 1990-01-04 00:00:00+00:00 NaT NaT 1990-01-05 00:00:00+00:00
1990-01-05 1990-01-05 00:00:00+00:00 NaT NaT 1990-01-06 00:00:00+00:00
... ... ... ... ...
2024-10-09 2024-10-09 00:00:00+00:00 NaT NaT 2024-10-10 00:00:00+00:00
2024-10-10 2024-10-10 00:00:00+00:00 NaT NaT 2024-10-11 00:00:00+00:00
2024-10-11 2024-10-11 00:00:00+00:00 NaT NaT 2024-10-12 00:00:00+00:00
2024-10-12 2024-10-12 00:00:00+00:00 NaT NaT 2024-10-13 00:00:00+00:00
2024-10-13 2024-10-13 00:00:00+00:00 NaT NaT 2024-10-14 00:00:00+00:00
If this works then in your backtest do something like:
from exchange_calendars import get_calendar as ec_get_calendar
cal = ec_get_calendar("24/7", start=pd.Timestamp("1990-01-01"))
run_algorithm(
...
trading_calendar=cal
)
Do not change the start date in the code above to match the date of your back-test- it is important that the start date above is earlier than any date that occurs in the ingested data.
Apologies for the late reply, I was traveling. Unfortunately it still is not working and it is getting more mysterious to me.
Starting with the initial version:
from zipline.utils.calendar_utils import get_calendar
# Set start and end date
start_date = pd.Timestamp('2005-1-1')
end_date = pd.Timestamp('2015-12-31')
# Fire off the backtest
results = run_algorithm(
trading_calendar=get_calendar("XNYS"),
...
)
I get a correct looking equity curve:
BTW: I get the same curve for the calendar "NYSE". What is the specific difference between these two calendars?
from exchange_calendars import get_calendar as ec_get_calendar
cal = ec_get_calendar("24/7", start=pd.Timestamp("1990-01-01"))
run_algorithm(
...
trading_calendar=cal
)
I get the following result:
I also cannot mix and match the two versions. if I try:
from exchange_calendars import get_calendar as ec_get_calendar
cal = ec_get_calendar("XNYS", start=pd.Timestamp("1990-01-01"))
run_algorithm(
...
trading_calendar=cal
)
I get the following error:
AssertionError: All readers must share target trading_calendar. Reader=<zipline.data.bcolz_minute_bars.BcolzMinuteBarReader object at 0x7f36f89316f0> for type=<class 'zipline.assets._assets.Equity'> uses calendar=<exchange_calendars.exchange_calendar_xnys.XNYSExchangeCalendar object at 0x7f36f8762110> which does not match the desired shared calendar=<exchange_calendars.exchange_calendar_xnys.XNYSExchangeCalendar object at 0x7f36f82680d0>
Printing the calendar 24/7 gives the following result:
open break_start break_end \
1990-01-01 1990-01-01 00:00:00+00:00 NaT NaT
1990-01-02 1990-01-02 00:00:00+00:00 NaT NaT
1990-01-03 1990-01-03 00:00:00+00:00 NaT NaT
1990-01-04 1990-01-04 00:00:00+00:00 NaT NaT
1990-01-05 1990-01-05 00:00:00+00:00 NaT NaT
... ... ... ...
2024-10-14 2024-10-14 00:00:00+00:00 NaT NaT
2024-10-15 2024-10-15 00:00:00+00:00 NaT NaT
2024-10-16 2024-10-16 00:00:00+00:00 NaT NaT
2024-10-17 2024-10-17 00:00:00+00:00 NaT NaT
2024-10-18 2024-10-18 00:00:00+00:00 NaT NaT
close
1990-01-01 1990-01-02 00:00:00+00:00
1990-01-02 1990-01-03 00:00:00+00:00
1990-01-03 1990-01-04 00:00:00+00:00
1990-01-04 1990-01-05 00:00:00+00:00
1990-01-05 1990-01-06 00:00:00+00:00
... ...
2024-10-14 2024-10-15 00:00:00+00:00
2024-10-15 2024-10-16 00:00:00+00:00
2024-10-16 2024-10-17 00:00:00+00:00
2024-10-17 2024-10-18 00:00:00+00:00
2024-10-18 2024-10-19 00:00:00+00:00
I am happy that at least one version is working now but I am not sure I understand why and what is wrong with the other versions.
BTW: I get the same curve for the calendar "NYSE". What is the specific difference between these two calendars?
There is no difference. NYSE is just an alias for XNYS. You can see this in the code for the exchange_calendars library at line 125 here: https://github.com/gerrymanoim/exchange_calendars/blob/master/exchange_calendars/calendar_utils.py
nderstand why and what is wrong with the other versions.
I suspect that the issue is that at daily frequency zipline is trying to trade at 0:00 UTC each day (as specified by the trading calendar) but when it looks at the calendar for the asset it is seeing the exchange for the asset is closed at that time.
If your strategy is trading across different exchanges in different time-zones, then you may need to configure it to trade at minute frequency, and then code the strategy so that it only place trades when each exchange opens. I haven't done this myself though, so this is entirely speculative, and you would need to do further testing and experimentation to see whether this is the case.
Alternatively, if your strategy is always trading within the same time-zone and holidays, but you need to take into account that your local time is different from the exchange time, then you can simply manually adjust the start and end dates by +/- 1 day as required and then use a trading calendar with the same zone as the exchange.
I get the following error:
AssertionError: All readers must share target trading_calendar. Reader=<zipline.data.bcolz_minute_bars.BcolzMinuteBarReader object at 0x7f36f89316f0> for type=<class 'zipline.assets._assets.Equity'> uses calendar=<exchange_calendars.exchange_calendar_xnys.XNYSExchangeCalendar object at 0x7f36f8762110> which does not match the desired shared calendar=<exchange_calendars.exchange_calendar_xnys.XNYSExchangeCalendar object at 0x7f36f82680d0>
This may or may not help but I think the AssertionError is a known issue where the calendar used to create the bundle is not the exact same object as the calendar you create when simulating. See section entitled 'Patch to allow calendars other than US calendars for backtesting' from https://pypi.org/project/zipline-norgatedata/#patch-to-allow-calendars-other-than-us-calendars-for-backtesting for a fix
Dear Zipline Maintainers,
Before I tell you about my issue, let me describe my environment:
Environment
Description of Issue
I am trying to run a simple backtest based on the book "Trading Evolved" by Andreas Clenow. It is a slighty modified version of the BuyApple program from the tutorial:
What did you expect to happen? Backtesting results.
What happened instead? Error message
File /opt/conda3/envs/zipline/lib/python3.10/site-packages/zipline/utils/run_algo.py:397, in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, bundle, bundle_timestamp, trading_calendar, metrics_set, benchmark_returns, default_extension, extensions, strict_extensions, environ, custom_loader, blotter) 393 load_extensions(default_extension, extensions, strict_extensions, environ) 395 benchmark_spec = BenchmarkSpec.from_returns(benchmark_returns) --> 397 return _run( 398 handle_data=handle_data, 399 initialize=initialize, 400 before_trading_start=before_trading_start, 401 analyze=analyze, 402 algofile=None, 403 algotext=None, 404 defines=(), 405 data_frequency=data_frequency, 406 capital_base=capital_base, 407 bundle=bundle, 408 bundle_timestamp=bundle_timestamp, 409 start=start, 410 end=end, 411 output=os.devnull, 412 trading_calendar=trading_calendar, 413 print_algo=False, 414 metrics_set=metrics_set, 415 local_namespace=False, 416 environ=environ, 417 blotter=blotter, 418 custom_loader=custom_loader, 419 benchmark_spec=benchmark_spec, 420 )
File /opt/conda3/envs/zipline/lib/python3.10/site-packages/zipline/utils/run_algo.py:103, in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter, custom_loader, benchmark_spec) 100 trading_calendar = get_calendar("XNYS") 102 # date parameter validation --> 103 if trading_calendar.sessions_distance(start, end) < 1: 104 raise _RunAlgoError( 105 "There are no trading days between %s and %s" 106 % ( (...) 109 ), 110 ) 112 benchmark_sid, benchmark_returns = benchmark_spec.resolve( 113 asset_finder=bundle_data.asset_finder, 114 start_date=start, 115 end_date=end, 116 )
File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/exchange_calendar.py:2275, in ExchangeCalendar.sessions_distance(self, start, end, _parse) 2258 def sessions_distance(self, start: Date, end: Date, _parse: bool = True) -> int: 2259 """Return the number of sessions in a range. 2260 2261 Parameters (...) 2273
end
then return will be negated. 2274 """ -> 2275 start, end = self._parse_start_end_dates(start, end, _parse) 2276 negate = end < start 2277 if negate:File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/exchange_calendar.py:2172, in ExchangeCalendar._parse_start_end_dates(self, start, end, _parse) 2170 if not _parse: 2171 return start, end -> 2172 return parse_date(start, "start", self), parse_date(end, "end", self)
File /opt/conda3/envs/zipline/lib/python3.10/site-packages/exchange_calendars/calendar_helpers.py:378, in parse_date(date, param_name, calendar, raise_oob) 375 ts = parse_timestamp(date, param_name, raise_oob=False, side="left", utc=False) 377 if ts.tz is not None: --> 378 raise ValueError( 379 f"Parameter
{param_name}
received with timezone defined as '{ts.tz.zone}'" 380 f" although a Date must be timezone naive." 381 ) 383 if not ts == ts.normalize(): 384 raise ValueError( 385 f"Parameter{param_name}
parsed as '{ts}' although a Date must have" 386 f" a time component of 00:00." 387 )ValueError: Parameter
start
received with timezone defined as 'UTC' although a Date must be timezone naive.