search

stefan-jansen / zipline-reloaded

Zipline, a Pythonic Algorithmic Trading Library
https://zipline.ml4trading.io
Apache License 2.0
1.12k stars 208 forks source link

sqlite3.IntegrityError: NOT NULL constraint failed: equity_symbol_mappings.symbol #182

Closed SuGuilin closed 1 year ago

SuGuilin commented 1 year ago

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

* Operating System: (Windows Version or `$ uname --all`) * Python Version: `$ python --version` * Python Bitness: `$ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))'` * How did you install Zipline: (`pip`, `conda`, or `other (please explain)`) * Python packages: `$ pip freeze` or `$ conda list`

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

Here is how you can reproduce this issue on your machine:

Reproduction Steps

I downloaded the historical data of 32 ETFs on Yahoo Finance and processed them in the form of date, open, high, low, close, volume, divided, and split. I would like to customize the bundle. The extension. py is as follows:

import pandas as pd

from zipline.data.bundles import register
from zipline.data.bundles.csvdir import csvdir_equities

start_session = pd.Timestamp('1990-01-01', tz='utc')#.tz_localize('US/Eastern')#
end_session = pd.Timestamp('2020-12-31', tz='utc')#.tz_localize('US/Eastern')#

# register the bundle
register(
    'ETF_2021_08_30_1990_to_2020_etf2005',  # name we select for the bundle
    csvdir_equities(
        # name of the directory as specified above (named after data frequency)
        ['daily'],
        # path to directory containing the
        r'D:\Code\Robo_Advisor\data',
    ),
    calendar_name='NYSE',  # New York Stock Exchange https://github.com/quantopian/trading_calendars
    start_session=start_session,
    end_session=end_session
)

When I run the command zipline ingest - b ETF 2021 08 30 1990 to 2020_ Etf2005, the following error occurred

(pytorch) PS D:\Code\Robo_Advisor> zipline ingest -b ETF_2021_08_30_1990_to_2020_etf2005
D:\Anaconda3\envs\pytorch\lib\site-packages\zipline\data\bcolz_minute_bars.py:424: FutureWarning: Indexing a timezone-naive DatetimeIndex with a t
imezone-aware datetime is deprecated and will raise KeyError in a future version.  Use a timezone-naive object instead.
  slicer = calendar.schedule.index.slice_indexer(
[2023-04-04T00:50:23+0800-INFO][zipline.data.bundles.core]
 Ingesting ETF_2021_08_30_1990_to_2020_etf2005
Loading custom pricing data:   [####################################]  100%
Merging daily equity files:  [####################################]
Traceback (most recent call last):
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\sqlalchemy\engine\base.py", line 1799, in _execute_context
    self.dialect.do_executemany(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\sqlalchemy\engine\default.py", line 729, in do_executemany
    cursor.executemany(statement, parameters)
sqlite3.IntegrityError: NOT NULL constraint failed: equity_symbol_mappings.symbol

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\Anaconda3\envs\pytorch\Scripts\zipline-script.py", line 10, in <module>
    sys.exit(main())
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\click\core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\zipline\__main__.py", line 397, in ingest
    bundles_module.ingest(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\zipline\data\bundles\core.py", line 445, in ingest
    bundle.ingest(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\zipline\data\bundles\csvdir.py", line 84, in ingest
    csvdir_bundle(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\zipline\data\bundles\csvdir.py", line 181, in csvdir_bundle
    asset_db_writer.write(equities=metadata)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\zipline\assets\asset_writer.py", line 812, in write
    self._real_write(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\zipline\assets\asset_writer.py", line 523, in _real_write
    self._write_assets(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\zipline\assets\asset_writer.py", line 851, in _write_assets
    self._write_df_to_table(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\zipline\assets\asset_writer.py", line 831, in _write_df_to_table
    df.to_sql(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\generic.py", line 2987, in to_sql
    return sql.to_sql(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\pandas\io\sql.py", line 695, in to_sql
    return pandas_sql.to_sql(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\pandas\io\sql.py", line 1738, in to_sql
    total_inserted = sql_engine.insert_records(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\pandas\io\sql.py", line 1335, in insert_records
    raise err
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\pandas\io\sql.py", line 1325, in insert_records
    return table.insert(chunksize=chunksize, method=method)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\pandas\io\sql.py", line 946, in insert
    num_inserted = exec_insert(conn, keys, chunk_iter)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\pandas\io\sql.py", line 853, in _execute_insert
    result = conn.execute(self.table.insert(), data)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\sqlalchemy\engine\base.py", line 1306, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\sqlalchemy\sql\elements.py", line 332, in _execute_on_connection
    return connection._execute_clauseelement(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\sqlalchemy\engine\base.py", line 1498, in _execute_clauseelement
    ret = self._execute_context(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\sqlalchemy\engine\base.py", line 1862, in _execute_context
    self._handle_dbapi_exception(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\sqlalchemy\engine\base.py", line 2043, in _handle_dbapi_exception
    util.raise_(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\sqlalchemy\util\compat.py", line 208, in raise_
    raise exception
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\sqlalchemy\engine\base.py", line 1799, in _execute_context
    self.dialect.do_executemany(
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\sqlalchemy\engine\default.py", line 729, in do_executemany
    cursor.executemany(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) NOT NULL constraint failed: equity_symbol_mappings.symbol
[SQL: INSERT INTO equity_symbol_mappings (id, company_symbol, end_date, symbol, share_class_symbol, start_date, sid) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ((0, '', 0, None, '', 0, 0), (1, '', 0, None, '', 0, 1), (2, '', 0, None, '', 0, 2), (3, '', 0, None, '', 0, 3), (4, '', 0, None, '',
 0, 4), (5, '', 0, None, '', 0, 5), (6, '', 0, None, '', 0, 6), (7, '', 0, None, '', 0, 7)  ... displaying 10 of 32 total bound parameter sets ...
  (30, '', 0, None, '', 0, 30), (31, '', 0, None, '', 0, 31))]
(Background on this error at: https://sqlalche.me/e/14/gkpj)

Sincerely, $ whoami

tan-yong-sheng commented 1 year ago

As temporary fix, run the command zipline ingest -b ETF_ 2021_ 08_ 30_ 1990_ to_ 2020_ Etf2005 --no-show-progress instead.

interzone2 commented 1 year ago

I have this exact issue. When I add the --no-show-progress flag I get:

zipline ingest -b custom-csvdir-bundle --no-show-progress
[2023-05-01T17:30:04+0800-INFO][zipline.data.bundles.core]
 Ingesting custom-csvdir-bundle
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/__attrs__ -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/high
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/high/__attrs__ -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/high
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/high/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/high/meta/storage -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/high/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/high/meta/sizes -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/high/meta
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/high/data
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/high/data/__0.blp -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/high/data
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/close
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/close/__attrs__ -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/close
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/close/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/close/meta/storage -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/close/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/close/meta/sizes -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/close/meta
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/close/data
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/close/data/__0.blp -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/close/data
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/low
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/low/__attrs__ -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/low
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/low/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/low/meta/storage -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/low/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/low/meta/sizes -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/low/meta
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/low/data
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/low/data/__0.blp -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/low/data
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/__rootdirs__ -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/id
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/id/__attrs__ -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/id
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/id/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/id/meta/storage -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/id/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/id/meta/sizes -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/id/meta
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/id/data
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/id/data/__0.blp -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/id/data
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/day
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/day/__attrs__ -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/day
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/day/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/day/meta/storage -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/day/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/day/meta/sizes -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/day/meta
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/day/data
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/day/data/__0.blp -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/day/data
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/volume
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/volume/__attrs__ -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/volume
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/volume/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/volume/meta/storage -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/volume/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/volume/meta/sizes -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/volume/meta
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/volume/data
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/volume/data/__0.blp -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/volume/data
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/open
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/open/__attrs__ -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/open
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/open/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/open/meta/storage -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/open/meta
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/open/meta/sizes -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/open/meta
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/open/data
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/open/data/__0.blp -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/daily_equities.bcolz/open/data
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/adjustments.sqlite -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112
[2023-05-01T17:30:04+0800-INFO][root]
 creating /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/minute_equities.bcolz
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/minute_equities.bcolz/metadata.json -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112/minute_equities.bcolz
[2023-05-01T17:30:04+0800-INFO][root]
 copying /var/folders/0t/wzdbfj4x15vd_nw6bg3wvnrm0000gn/T/tmpd88a5q7z/custom-csvdir-bundle/2023-05-01T09;30;04.360112/assets-7.sqlite -> /Users/amh17/.zipline/data/custom-csvdir-bundle/2023-05-01T09;30;04.360112

Now I have the bundle folder:

Screenshot 2023-05-01 at 5 44 17 pm

when trying to run the tutorial.ipynb file line:

!zipline run -f /Users/amh17/Documents/PyQuant/zipline-reloaded/src/zipline/examples/buyapple.py --start 2019-2-1 --end 2022-2-1 -o buynvda_out.pickle --no-ben

I just get:

[2023-05-01T18:02:11+0800-WARNING][ZiplineLog]
 Cannot place order for NVDA, as it has de-listed. Any existing positions for this asset will be liquidated on 2018-03-28 00:00:00.
[2023-05-01T18:02:11+0800-WARNING][ZiplineLog]
 Cannot place order for NVDA, as it has de-listed. Any existing positions for this asset will be liquidated on 2018-03-28 00:00:00.
[2023-05-01T18:02:11+0800-WARNING][ZiplineLog]
 Cannot place order for NVDA, as it has de-listed. Any existing positions for this asset will be liquidated on 2018-03-28 00:00:00.
[2023-05-01T18:02:11+0800-WARNING][ZiplineLog]
 Cannot place order for NVDA, as it has de-listed. Any existing positions for this asset will be liquidated on 2018-03-28 00:00:00.
[2023-05-01T18:02:11+0800-WARNING][ZiplineLog]
 Cannot place order for NVDA, as it has de-listed. Any existing positions for this asset will be liquidated on 2018-03-28 00:00:00.
[2023-05-01T18:02:11+0800-WARNING][ZiplineLog]
 Cannot place order for NVDA, as it has de-listed. Any existing positions for this asset will be liquidated on 2018-03-28 00:00:00.
[2023-05-01T18:02:11+0800-WARNING][ZiplineLog]
 Cannot place order for NVDA, as it has de-listed. Any existing positions for this asset will be liquidated on 2018-03-28 00:00:00.
[2023-05-01T18:02:11+0800-WARNING][ZiplineLog]
 Cannot place order for NVDA, as it has de-listed. Any existing positions for this asset will be liquidated on 2018-03-28 00:00:00.
[2023-05-01T18:02:11+0800-WARNING][ZiplineLog]
 Cannot place order for NVDA, as it has de-listed. Any existing positions for this asset will be liquidated on 2018-03-28 00:00:00.
[2023-05-01T18:02:11+0800-WARNING][ZiplineLog]

Thereafter any of the functions of the tutorial.ipynb file offer null results.

2019-02-01 14:31:00+00:00 | 2019-02-01 21:00:00+00:00 | 0 | [] | 0.0 | 0 | [] | 0.0 | 0.0 | 0.0 | ... | None | 0.0 | 0.0 | NaN | 0.0 | 0.0 | 0.0 | 1 | NaN | 2019-02
2019-02-04 14:31:00+00:00 | 2019-02-04 21:00:00+00:00 | 0 | [] | 0.0 | 0 | [] | 0.0 | 0.0 | 0.0 | ... | None | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 2 | 0.0 | 2019-02
2019-02-05 14:31:00+00:00 | 2019-02-05 21:00:00+00:00 | 0 | [] | 0.0 | 0 | [] | 0.0 | 0.0 | 0.0 | ... | None | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 3 | 0.0 | 2019-02
2019-02-06 14:31:00+00:00 | 2019-02-06 21:00:00+00:00 | 0 | [] | 0.0 | 0 | [] | 0.0 | 0.0 | 0.0 | ... | None | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 4 | 0.0 | 2019-02
2019-02-07 14:31:00+00:00 | 2019-02-07 21:00:00+00:00 | 0 | [] | 0.0 | 0 | [] | 0.0 | 0.0 | 0.0 | ... | None

Is there another method to ingesting custom csv data, registering it and then performing calculations?

Thanks!

praxarma commented 1 year ago

As a temporary fix, I did the following. It works for me as of July 2, 2023.

  1. The conda version as well as the pip version both fails with this error. So I built zipline-reloaded from source (steps given here: https://zipline.ml4trading.io/development-guidelines.html ) .
  2. Changed the ~/.zipline/extension.py slightly to remove timestamp localization (otherwise it complains about not being able to compare tz_naive with tz_aware timestamps). Also, I ensured that the parameter start_session is the earliest of the start dates in the csv files. The end_session was set at None so that it reads till the end of the csv files. If it complains about missing dates in some csv files, these need to be forward filled. My exension.py looks like this:
import pandas as pd

from zipline.data.bundles import register
from zipline.data.bundles.csvdir import csvdir_equities

start_session = pd.Timestamp('2003-07-07')
end_session = None

register(
    'custom-csvdir-bundle',
    csvdir_equities(
        ['daily'],
        '/path/to/csv/files/not-including-daily-folder/',
    ),
    calendar_name='NYSE', # US equities
    start_session=start_session,
    end_session=end_session
)
stefan-jansen commented 1 year ago

Thanks for sharing your workaround!