search

quantopian / zipline

Zipline, a Pythonic Algorithmic Trading Library
https://www.zipline.io
Apache License 2.0
17.28k stars 4.67k forks source link

sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. #2152

Open hotea opened 6 years ago

hotea commented 6 years ago

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

Reproduction Steps

i use zipline api in flask web framework

from zipline.utils.calendars.calendar_utils import get_calendar
from zipline.algorithm import TradingAlgorithm
from zipline.data.bundles.core import load
from zipline.data.data_portal import DataPortal
import datetime
import pytz

bundle_data = load(name='quandl')
_, asset_db_path = re.split(r'sqlite:///', str(bundle_data.asset_finder.engine.url), maxsplit=1)
trading_calendar = get_calendar(calendar_name='NASDAQ')

trading_env = TradingEnvironment(
        asset_db_path=asset_db_path,
        trading_calendar=trading_calendar,
        )
data_portal = DataPortal(
        asset_finder=bundle_data.asset_finder,
        trading_calendar=trading_calendar,
        first_trading_day=bundle_data.equity_minute_bar_reader.first_trading_day,
        equity_minute_reader=bundle_data.equity_minute_bar_reader,
        equity_daily_reader=bundle_data.equity_daily_bar_reader,
        adjustment_reader=bundle_data.adjustment_reader,
        )
start = datetime.datatime(year=2017, month=1, day=1, hour=0, minute=0, second=0, microsecond=0, tzinfo=pytz.utc)
end = datetime.datatime(year=2017, month=10, day=1, hour=0, minute=0, second=0, microsecond=0, tzinfo=pytz.utc)
capital_base = 100000

sim_params = create_simulation_parameters(
        year=2017,
        start=start,
        end=end,
        capital_base=capital_base,
        num_days=None,
        data_frequency='daily',
        emission_rate='daily',
        trading_calendar=trading_calendar,
    )

algor_obj = TradingAlgorithm(script=codetext,
                                 sim_params=sim_params,
                                 env=trading_env,
                                 )
dataportal = get_dataportal(trading_calendar)
perf_manual = algor_obj.run(dataportal)

What steps have you taken to resolve this already?

I temporarily modified the function check_and_create_connection of the library file /VIRTUALENV_NAME/lib/python3.6/site-packages/zipline/utils/sqlite_utils.py.

add a perameter check_same_thread=False to sqlite3.connect() ...

Anything else?

Thanks in advance

hotea commented 6 years ago

and the script of the TradingAlgorithm is: https://github.com/quantopian/zipline/blob/master/zipline/examples/dual_ema_talib.py

llllllllll commented 6 years ago

Are you using threaded workers in your wsgi server? This issue is happening because you are creating the data portal in a different thread than you are using it.

One fix could be to store the data portal on a threading.local to ensure that it is only ever used in a single thread.

hotea commented 6 years ago

but is there any zipline api used to control the sqlite connection? I did't find one...