search

quantopian / zipline

Zipline, a Pythonic Algorithmic Trading Library
https://www.zipline.io
Apache License 2.0
17.53k stars 4.71k forks source link

Time zone conversions, Delorean necessary? #148

Closed wesm closed 10 years ago

wesm commented 11 years ago

I've looked at the various places where time zone conversions are being done, e.g.:

def get_next_trading_dt(current, interval):
    naive = current.replace(tzinfo=None)
    delo = Delorean(naive, pytz.utc.zone)
    ex_tz = trading.environment.exchange_tz
    next_dt = delo.shift(ex_tz).datetime

    while True:
        next_dt = next_dt + interval
        next_delo = Delorean(next_dt.replace(tzinfo=None), ex_tz)
        next_utc = next_delo.shift(pytz.utc.zone).datetime
        if trading.environment.is_market_hours(next_utc):
            break

    return next_utc

I'm wondering whether you might be better served using the pandas.Timestamp object and its tz_convert method. Notably adding timedeltas to Timestamp shifts the underlying UTC timestamp and respects DST transitions, etc. It would be faster and simpler than this mucky business with Delorean, and you can get rid of the extra library dependency.

andrewkittredge commented 11 years ago

Perhaps related. I can't get the zipline.test_perf_tracking.TestPerformanceTracker to finish one way or the other. It seems like it gets into an infinite loop in the function Wes is complaining about.

I'm running the below on the bottom of TestPerformanceTracker

if name == 'main': suite = unittest.TestLoader().loadTestsFromTestCase(TestPerformanceTracker) unittest.TextTestRunner().run(suite)

ehebert commented 11 years ago

@wesm, we were unaware of that particular capability of Timestamp. Will try to drop it in. Thanks!

ehebert commented 11 years ago

@andrewkittredge does that test work for you when running nosetests --tests=tests.test_perf_tracking? On May 8, 2013 8:36 PM, "Andrew Kittredge" notifications@github.com wrote:

Perhaps related. I can't get the zipline.test_perf_tracking.TestPerformanceTracker to finish one way or the other. It seems like it gets into an infinite loop in the function Wes is complaining about.

I'm running the below on the bottom of TestPerformanceTracker

if name == 'main': suite = unittest.TestLoader().loadTestsFromTestCase(TestPerformanceTracker) unittest.TextTestRunner().run(suite)

— Reply to this email directly or view it on GitHubhttps://github.com/quantopian/zipline/issues/148#issuecomment-17642899 .

andrewkittredge commented 11 years ago

The tests fail in the same manner when I use nose. I've pasted the results below. I'm on a mac, and using the latest release of delorean and the master branch of zipline

This snippet below hangs which I believe narrows down the problem. trading.environment.is_market_hours never returns true because test_date is less than mkt_open, I think because the trade_date does not get shifted to UTC.

I'm playing with timezones, I'll see if can figure out a patch.

SNIPPET from zipline.utils.factory import get_next_trading_dt from zipline.finance.trading import TradingEnvironment from pytz import timezone from datetime import datetime, timedelta eastern_timezone = timezone('US/Eastern') with TradingEnvironment(): get_next_trading_dt(datetime(2013, 5, 19, tzinfo=eastern_timezone), timedelta(days=1))

NOSE RESULTS

nosetests --tests=tests.test_perf_tracking:TestPerformanceTracker Tests minute performance tracking. ... ERROR test_tracker_0_Dontdeleteanyevents (tests.test_perf_tracking.TestPerformanceTracker) ... ERROR test_tracker_1_Deletefirstdayofevents (tests.test_perf_tracking.TestPerformanceTracker) ... ERROR test_tracker_2_Deletefirsttwodaysofevents (tests.test_perf_tracking.TestPerformanceTracker) ... ERROR test_tracker_3_Deleteonedayofeventsfromthemiddle (tests.test_perf_tracking.TestPerformanceTracker) ... ERROR test_tracker_4_Deletetwoeventsfromthemiddle (tests.test_perf_tracking.TestPerformanceTracker) ... ERROR test_tracker_5_Deletelastdayofevents (tests.test_perf_tracking.TestPerformanceTracker) ... ERROR test_tracker_6_Deletelasttwodaysofevents (tests.test_perf_tracking.TestPerformanceTracker) ... ERROR test_tracker_7_Deleteallbutoneevent (tests.test_perf_tracking.TestPerformanceTracker) ... ERROR

ERROR: Tests minute performance tracking.

Traceback (most recent call last): File "/Users/akittredge/Documents/workspace/zipline/tests/test_perf_tracking.py", line 1036, in test_minute_tracker start_dt = trading.environment.exchange_dt_in_utc( AttributeError: 'NoneType' object has no attribute 'exchange_dt_in_utc'

ERROR: test_tracker_0_Dontdeleteanyevents (tests.test_perf_tracking.TestPerformanceTracker)

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose_parameterized-0.2-py2.7.egg/nose_parameterized/parameterized.py", line 99, in parameterized_expand_helper_helper return func(*(self + args)) File "/Users/akittredge/Documents/workspace/zipline/tests/test_perf_tracking.py", line 938, in test_tracker source_id="factory1" File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 145, in create_trade_history current = get_next_trading_dt(current, interval) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 129, in get_next_trading_dt next_delo = Delorean(next_dt.replace(tzinfo=None), ex_tz) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 187, in init self._dt = localize(datetime, timezone) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 131, in localize return tz.localize(dt) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib-1.1.1-py2.7-macosx-10.6-intel.egg/pytz/tzinfo.py", line 271, in localize loc_dt = dt + delta OverflowError: date value out of range

ERROR: test_tracker_1_Deletefirstdayofevents (tests.test_perf_tracking.TestPerformanceTracker)

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose_parameterized-0.2-py2.7.egg/nose_parameterized/parameterized.py", line 99, in parameterized_expand_helper_helper return func(*(self + args)) File "/Users/akittredge/Documents/workspace/zipline/tests/test_perf_tracking.py", line 938, in test_tracker source_id="factory1" File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 145, in create_trade_history current = get_next_trading_dt(current, interval) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 129, in get_next_trading_dt next_delo = Delorean(next_dt.replace(tzinfo=None), ex_tz) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 187, in init self._dt = localize(datetime, timezone) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 131, in localize return tz.localize(dt) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib-1.1.1-py2.7-macosx-10.6-intel.egg/pytz/tzinfo.py", line 271, in localize loc_dt = dt + delta OverflowError: date value out of range

ERROR: test_tracker_2_Deletefirsttwodaysofevents (tests.test_perf_tracking.TestPerformanceTracker)

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose_parameterized-0.2-py2.7.egg/nose_parameterized/parameterized.py", line 99, in parameterized_expand_helper_helper return func(*(self + args)) File "/Users/akittredge/Documents/workspace/zipline/tests/test_perf_tracking.py", line 938, in test_tracker source_id="factory1" File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 145, in create_trade_history current = get_next_trading_dt(current, interval) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 129, in get_next_trading_dt next_delo = Delorean(next_dt.replace(tzinfo=None), ex_tz) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 187, in init self._dt = localize(datetime, timezone) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 131, in localize return tz.localize(dt) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib-1.1.1-py2.7-macosx-10.6-intel.egg/pytz/tzinfo.py", line 271, in localize loc_dt = dt + delta OverflowError: date value out of range

ERROR: test_tracker_3_Deleteonedayofeventsfromthemiddle (tests.test_perf_tracking.TestPerformanceTracker)

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose_parameterized-0.2-py2.7.egg/nose_parameterized/parameterized.py", line 99, in parameterized_expand_helper_helper return func(*(self + args)) File "/Users/akittredge/Documents/workspace/zipline/tests/test_perf_tracking.py", line 938, in test_tracker source_id="factory1" File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 145, in create_trade_history current = get_next_trading_dt(current, interval) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 129, in get_next_trading_dt next_delo = Delorean(next_dt.replace(tzinfo=None), ex_tz) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 187, in init self._dt = localize(datetime, timezone) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 131, in localize return tz.localize(dt) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib-1.1.1-py2.7-macosx-10.6-intel.egg/pytz/tzinfo.py", line 271, in localize loc_dt = dt + delta OverflowError: date value out of range

ERROR: test_tracker_4_Deletetwoeventsfromthemiddle (tests.test_perf_tracking.TestPerformanceTracker)

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose_parameterized-0.2-py2.7.egg/nose_parameterized/parameterized.py", line 99, in parameterized_expand_helper_helper return func(*(self + args)) File "/Users/akittredge/Documents/workspace/zipline/tests/test_perf_tracking.py", line 938, in test_tracker source_id="factory1" File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 145, in create_trade_history current = get_next_trading_dt(current, interval) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 129, in get_next_trading_dt next_delo = Delorean(next_dt.replace(tzinfo=None), ex_tz) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 187, in init self._dt = localize(datetime, timezone) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 131, in localize return tz.localize(dt) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib-1.1.1-py2.7-macosx-10.6-intel.egg/pytz/tzinfo.py", line 271, in localize loc_dt = dt + delta OverflowError: date value out of range

ERROR: test_tracker_5_Deletelastdayofevents (tests.test_perf_tracking.TestPerformanceTracker)

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose_parameterized-0.2-py2.7.egg/nose_parameterized/parameterized.py", line 99, in parameterized_expand_helper_helper return func(*(self + args)) File "/Users/akittredge/Documents/workspace/zipline/tests/test_perf_tracking.py", line 938, in test_tracker source_id="factory1" File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 145, in create_trade_history current = get_next_trading_dt(current, interval) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 129, in get_next_trading_dt next_delo = Delorean(next_dt.replace(tzinfo=None), ex_tz) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 187, in init self._dt = localize(datetime, timezone) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 131, in localize return tz.localize(dt) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib-1.1.1-py2.7-macosx-10.6-intel.egg/pytz/tzinfo.py", line 271, in localize loc_dt = dt + delta OverflowError: date value out of range

ERROR: test_tracker_6_Deletelasttwodaysofevents (tests.test_perf_tracking.TestPerformanceTracker)

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose_parameterized-0.2-py2.7.egg/nose_parameterized/parameterized.py", line 99, in parameterized_expand_helper_helper return func(*(self + args)) File "/Users/akittredge/Documents/workspace/zipline/tests/test_perf_tracking.py", line 938, in test_tracker source_id="factory1" File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 145, in create_trade_history current = get_next_trading_dt(current, interval) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 129, in get_next_trading_dt next_delo = Delorean(next_dt.replace(tzinfo=None), ex_tz) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 187, in init self._dt = localize(datetime, timezone) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 131, in localize return tz.localize(dt) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib-1.1.1-py2.7-macosx-10.6-intel.egg/pytz/tzinfo.py", line 271, in localize loc_dt = dt + delta OverflowError: date value out of range

ERROR: test_tracker_7_Deleteallbutoneevent (tests.test_perf_tracking.TestPerformanceTracker)

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose_parameterized-0.2-py2.7.egg/nose_parameterized/parameterized.py", line 99, in parameterized_expand_helper_helper return func(*(self + args)) File "/Users/akittredge/Documents/workspace/zipline/tests/test_perf_tracking.py", line 938, in test_tracker source_id="factory1" File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 145, in create_trade_history current = get_next_trading_dt(current, interval) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.py", line 129, in get_next_trading_dt next_delo = Delorean(next_dt.replace(tzinfo=None), ex_tz) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 187, in init self._dt = localize(datetime, timezone) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delorean/dates.py", line 131, in localize return tz.localize(dt) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib-1.1.1-py2.7-macosx-10.6-intel.egg/pytz/tzinfo.py", line 271, in localize loc_dt = dt + delta OverflowError: date value out of range

Ran 9 tests in 1909.375s

FAILED (errors=9)

ehebert commented 10 years ago

As of 796b9fb67a43e98e167c59af976334d4e6bf2c1f, now using pd.Timestamp throughout.